Mutually exclusive experiments and holdouts

Keep two experiments from reaching the same user. One allocation feature puts every user in a cohort, and each experiment requires its own.

When experiments must not overlap#

Use this when two experiments could affect each other's results, such as ctaButton and hero from Experiments, which both change the landing page's hero. The allocation feature chooses one cohort. The experiments read that same result through conditions. For priority rather than fixed cohorts, see Make one feature give way to another.

Start from the default scaffold with a userId attribute and no configured environments.

Create an allocation feature#

features/experimentAllocation.yml

description: Shared experiment allocation
bucketBy: userId
variations:
  - value: first
    weight: 40
  - value: second
    weight: 40
  - value: holdout
    weight: 20
rules:
  - key: everyone
    conditions: '*'
    percentage: 100

Keep this feature at 100%. Represent users outside the experiments with an explicit holdout variation, not a partial allocation rollout.

Require a cohort in each experiment#

features/firstExperiment.yml

description: First experiment
bucketBy: userId
conditions:
  - feature: experimentAllocation
    variation: first
rules:
  - key: everyone
    conditions: '*'
    percentage: 100

features/secondExperiment.yml

description: Second experiment
bucketBy: userId
conditions:
  - feature: experimentAllocation
    variation: second
rules:
  - key: everyone
    conditions: '*'
    percentage: 100

Evaluate each experiment normally with isEnabled() and, if it defines variations, getVariation(). You do not need to evaluate the allocation feature first.

How users are assigned#

Allocation variation	First experiment	Second experiment	Approximate audience
`first`	Enabled	Disabled	40%
`second`	Disabled	Enabled	40%
`holdout`	Disabled	Disabled	20%

Use the same stable bucketBy identity for the allocation and experiments. With bucketBy: { or: [userId, deviceId] }, signing in can change the identity and move a user to another cohort. Evaluate against the same context, datafile and sticky state: changing these between reads can change allocation.

Rollouts and variations inside a cohort#

An experiment still applies its own rules after its requirement passes. A 25% rollout inside a 40% cohort reaches about 10% of all users. The experiment's own variations then divide those users, with its own bucketing.

Holdouts and reserved cohorts#

Choose cohort weights before starting. Changing weights can reassign users. To add experiments later without resizing existing cohorts, create reserved variations such as reservedA up front, then make a new experiment require that value.

A holdout receives neither experiment. Reusing a completed experiment's cohort carries those users' previous exposure into the next experiment. It is not a fresh population.

Give QA access#

Force the allocation feature's variation to put a QA user into one cohort. Add this field to experimentAllocation, keeping its normal variations and rules:

Excerpt: allocation QA override

features/experimentAllocation.yml

force:
  - conditions:
      - attribute: userId
        operator: equals
        value: qa-user
    enabled: true
    variation: first

Force entries and sticky values on the experiments themselves can bypass their requirements and defeat exclusion. Apply cohort overrides to the allocation feature instead.

Test it#

Use fixed bucket values for boundary examples and an allocation sticky value to check dependency behaviour:

tests/features/firstExperiment.spec.yml

feature: firstExperiment
assertions:
  - at: 20
    expectedToBeEnabled: true
  - at: 60
    expectedToBeEnabled: false
  - at: 90
    expectedToBeEnabled: false
  - at: 20
    stickyFeatures:
      experimentAllocation:
        enabled: true
        variation: second
    expectedToBeEnabled: false

Add a scenario to check exclusion across cohort boundaries:

tests/scenarios/exclusiveExperiments.spec.yml

matrix:
  at: [5, 35, 45, 75, 85, 99]
at: ${{ at }}
checks:
  - not:
      - feature: firstExperiment
        enabled: true
      - feature: secondExperiment
        enabled: true

This not means the two experiments must not both be enabled. Also test many user IDs without at, asserting zero overlap and approximate cohort sizes. Fixed bucket tests alone do not verify real hashing.

Track exposures#

Record the experiment's own variation when the user sees it, not the allocation feature's variation. The Google Analytics module skips eligibility reads using dependencyPurpose: 'condition'. Do the same in custom tracking modules. Do not skip consumed 'composition' reads merely because they have dependencyOf.

Targets, environments and sets#

Targets include the allocation feature as a required dependency even when they select only the experiments. The allocation needs rules in every configured environment where experiments run. Missing environment rules disable it and therefore its experiments.

Do not hide prerequisites

expose: false removes a prerequisite from generated datafiles, including targeted builds. Lint does not warn about this relationship. A missing allocation feature makes its variation requirements fail, leaving the experiments off. In the priority pattern, a missing feature satisfies enabled: false, potentially enabling the dependent feature for everyone. Keep prerequisites exposed and inspect every generated target.

Promotions bring the allocation feature along as a dependency. Set promotable: false on it if an existing destination set must keep its own cohort weights. A missing destination feature is still created. Different allocations across sets can assign the same user differently.

Experiments, for experiment variations
Managing dependencies, for priority exclusion
Conditions, for the complete syntax