Laravel Pennant Experiments

What it solves

Pennant decides who gets a feature, but an A/B test also needs to know who actually saw each variant, who converted, and whether the difference is more than noise.

Pennant records none of that, and it has no way to say when there is enough data to read a result.

An unofficial, independent package for laravel/pennant (not made or endorsed by Laravel). Each experiment is a Pennant feature whose value is the variant name. It counts only scopes that were shown a variant, records conversion goals from PHP or JavaScript, and has a plain Blade results page that holds back verdicts until there is enough data and the planned sample is reached.

• Experiments are Pennant features: two or more variants with whole-number weights, the first one is the control
• Sticky assignment for users, teams, any scope and guests (a random id in a first-party, http-only cookie)
• Counts only exposed scopes, once each: expose() or @variant records, variant() does not
• Conversion goals from PHP or from JavaScript endpoints with CSRF and a rate limit, once per scope and goal
• Two-proportion z-test, Bayesian probability to beat control, Wilson intervals, Bonferroni correction for several challengers and minimum-data and planned-sample rules
• End and roll out the winner from the results page or artisan; optional Experiments tab in laravel-pennant-dashboard 1.1+

Install

Requires PHP 8.3+ and Laravel 12 and 13.

composer require amjadiqbal/laravel-pennant-experiments

Usage

use AmjadIqbal\PennantExperiments\Facades\Experiments;

// In a service provider's boot():
Experiments::define('checkout-button', ['control' => 50, 'green' => 50], [
    'goals' => ['purchase'],
    'minimum_detectable_effect' => 0.10,   // plan for a +10% relative lift
]);

// Where the variant is shown (records the exposure):
$variant = Experiments::expose('checkout-button');

// When the goal is reached:
Experiments::convert('checkout-button', 'purchase');

Setup

This is an independent package, not made or endorsed by Laravel. It requires laravel/pennant 1.18+ and works with any Pennant driver; with the database driver assignments are also stored. Its migration adds three tables: pennant_experiments (run state and winner), pennant_experiment_exposures and pennant_experiment_conversions. Define experiments in a service provider, then open /pennant/experiments; outside the local environment it returns 403 until you define who may see it.

1. 1 composer require amjadiqbal/laravel-pennant-experiments
2. 2 php artisan migrate
3. 3 Experiments::define(...) in a service provider's boot().
4. 4 Define the viewPennantExperiments gate (and optionally managePennantExperiments).
5. 5 Open /pennant/experiments, or run php artisan pennant-experiments:results.

How variants are assigned

Experiments::define() registers a Pennant feature of the same name whose resolver picks a variant by weight from sha256 of the experiment name (or a salt option) and Pennant's serialized scope, so the same scope gets the same variant on any driver while the weights stay the same. With the database driver Pennant stores the first answer, so changing weights later only changes the split for scopes not assigned yet; with the array driver the hash alone decides, so new weights move existing scopes too. Either way, changing weights mid-test means the groups are no longer comparable. Without a scope, the participant is your resolveScopeUsing() resolver, else the signed-in user, else a guest id: a random UUID in the pennant_experiments_visitor cookie (one year, http-only, SameSite=Lax). Pennant's own Feature::value() uses Pennant's default scope, which is null for guests, so use Experiments::variant() or expose() for guests.

@variant('checkout-button', 'green')
    <button class="btn-green">Buy now</button>
@elsevariant('checkout-button', 'blue')
    <button class="btn-blue">Buy now</button>
@else
    <button class="btn">Buy now</button>
@endvariant

Experiments::expose('team-pricing', $team);   // any scope
Experiments::resolveScopeUsing(fn () => auth()->user()?->currentTeam);

Exposures and conversions

expose() and the @variant directive return the variant and record an exposure; variant() only returns it. Each scope has one exposure row per experiment, holding the variant it was first recorded with. A conversion counts once per scope and goal, only for a scope that was exposed, and is credited to that exposed variant; unknown goals are refused. Ineligible scopes (an eligible closure that returns false) get the control and are not counted. While the experiment is ended nothing is recorded. From the browser, @experimentsScript adds PennantExperiments.expose() and convert() and a data-experiment-convert click handler. The endpoints under /_experiments run in the web middleware group (so Laravel's CSRF check applies) with throttle:120,1 by default, and the participant always comes from the session or the visitor cookie, never from the request body.

Experiments::convert('checkout-button', 'purchase');          // the current participant
Experiments::convert('checkout-button', 'purchase', $user);   // a given scope, e.g. from a job

<button data-experiment-convert="checkout-button" data-experiment-goal="purchase">Buy</button>

How the results are worked out

• Rates are unique converting scopes divided by exposed scopes, per goal and variant (binary goals only, no revenue metrics).
• Each challenger is compared with the control by a two-sided, two-proportion z-test with the pooled standard error; the difference interval uses the unpooled (Wald) standard error.
• With more than one challenger, alpha (0.05 by default) is divided by the number of challengers (Bonferroni). The difference and Wilson rate intervals on the page use the same adjusted level, which is why a three-variant test shows 97.5% intervals.
• Probability to beat control uses Beta(1 + conversions, 1 + non-conversions) posteriors from a flat prior and Evan Miller's closed-form sum, so there is no random sampling; past about 20,000 conversions in the challenger it uses a normal approximation. It says how likely the challenger is better, not by how much.
• Too early: shown until every variant has 100 exposures and 10 conversions for that goal (min_exposures, min_conversions).
• Planned sample per variant: sample_size, or computed from minimum_detectable_effect at the control's observed rate with the adjusted alpha and 80% power (Fleiss, no continuity correction). Because it follows the observed rate, it can move while the test runs. Until every variant reaches it, the verdict is Keep running whatever the p-value.
• Without a planned sample, a significant result is labelled Better or Worse (no planned sample) as a provisional reading. Repeated looks still raise the false-positive rate, so set the plan before you start.

Ending and running again

Pick a winner on the results page and press End and roll out, or run pennant-experiments:end with --winner. The experiment is marked ended, Pennant's activateForEveryone() rewrites the stored values for that feature to the winner, the resolver answers the winner for scopes not seen before, and no more exposures or conversions are recorded. Run again marks it running and purges the feature's stored Pennant values, so scopes are assigned afresh; earlier exposures and conversions are kept and still count, and a scope already exposed keeps its first recorded variant. When you are done, replace the experiment in code with the winning behaviour and remove the definition; experiments with data but no definition are listed on the index page.

php artisan pennant-experiments:results                   # every experiment
php artisan pennant-experiments:results checkout-button   # one, per goal
php artisan pennant-experiments:end checkout-button --winner=green

Who can see it, and the dashboard tab

viewPennantExperiments decides who can open the results pages; without your own definition it allows the local environment only. managePennantExperiments (ending and running again) defaults to whoever may view, so define it too if those should differ. PENNANT_EXPERIMENTS_READ_ONLY=true blocks ending and restarting for everyone, and PENNANT_EXPERIMENTS_ENABLED=false removes the results routes. If laravel-pennant-dashboard (see /laravel/packages/laravel-pennant-dashboard) is installed, the results live under its path and the package adds an Experiments entry to its pennant-dashboard.tabs config, which version 1.1 and later shows in the header; with an older dashboard the pages still link back to it. Neither package requires the other, and dashboard_tab => false turns the tab off.

use Illuminate\Support\Facades\Gate;

Gate::define('viewPennantExperiments', fn ($user = null) => $user?->is_admin);
Gate::define('managePennantExperiments', fn ($user = null) => $user?->is_admin);   // end / run again

Things to know

• When a guest signs in, the Login listener moves their exposures and conversions to the user and stores their variant for the user, for each experiment the user had not been exposed to yet. A guest who later signs in on another device stays a separate participant there.
• Bot exclusion applies to guests: a visitor whose user agent matches bot_pattern, or is empty, gets the control and is not counted. Signed-in users and scopes you pass yourself are not checked.
• If a feature value is set outside the experiment (for example turned on for everyone in a dashboard), it is not a variant, so the scope is shown the control.
• The visitor cookie holds only a random id. No IP addresses or user agents are stored, and the package makes no network calls. There is no Do Not Track handling.
• Tables and connection are configurable; set run_migrations to false after publishing the migrations.

Support

Found a bug or want a feature? Open an issue on GitHub. For help adding it to your own project, use the quote form above.