Experiments

An experiment splits users between versions of something, shows each user theirs, records who saw what, and compares outcomes in your analytics. This page runs two experiments on a landing page, an A/B test and a multivariate test.

Variations covers the definitions in depth, with the pricing experiment.

A/B or multivariate#

An A/B test compares versions of one thing, such as a button's text. A multivariate test compares combinations of several things, such as a headline and a button together, and needs more users to tell the combinations apart. With three or more versions, an A/B test is sometimes called A/B/n.

Plan the test#

Before it starts, write down:

• the outcome you expect to move, such as sign ups
• how many visitors each variation needs, which sets how long the test runs
• what you will do with each possible result

Deciding these afterwards makes any difference easy to find and hard to trust. Do not stop the first time a variation pulls ahead either, since an early lead can be chance.

Run an A/B test#

Compare the sign up button's text, Sign up against Get started, for every visitor:

description: Sign up button text
bucketBy: deviceId
variations:
  - value: control
    description: Sign up
    weight: 50
  - value: treatment
    description: Get started
    weight: 50
rules:
  - key: everyone
    conditions: '*'
    percentage: 100

Every visitor gets one of the two variations, half each. The illustration calls control A and treatment B, as Variations does.

Visitors to a landing page are usually not signed in, so it buckets by deviceId, a random ID the application stores on the device. The same device then always sees the same text, as Bucketing explains. If the same test should follow users who sign in later, bucket by { or: [userId, deviceId] }, and expect some to change variation when their ID switches from one to the other.

To run it for some visitors only, give the rule a segment instead of '*'.

Show each variation#

Read the variation where the button renders:

const context = {
  deviceId: storedDeviceId,
}
const text =
  f.getVariation('ctaButton', context) === 'treatment'
    ? 'Get started'
    : 'Sign up'

Test combinations#

To test the headline and the button text together, put both in one feature as variables, and give each variation its own values:

description: Landing page hero
bucketBy: deviceId
variablesSchema:
  headline:
    type: string
    defaultValue: Welcome
  ctaButtonText:
    type: string
    defaultValue: Sign up
variations:
  - value: control
    weight: 25
  - value: treatment1
    weight: 25
    variables:
      ctaButtonText: Get started
  - value: treatment2
    weight: 25
    variables:
      headline: Hello there
  - value: treatment3
    weight: 25
    variables:
      headline: Hello there
      ctaButtonText: Get started
rules:
  - key: everyone
    conditions: '*'
    percentage: 100

Each variation sets only what differs from the defaults. The application reads the values, without knowing which combination it has:

const headline = f.getVariable('hero', 'headline', context)
const buttonText = f.getVariable('hero', 'ctaButtonText', context)

The copy now lives in the feature, so whoever owns it can change the words in a pull request, without an application release. Change them between tests, not during one, or the results mix two versions of the copy.

Preview each combination#

Before the test starts, let the people who wrote the copy check every combination in production. A force entry gives a test device one variation, whatever its bucket:

# ...
force:
  - conditions:
      - attribute: deviceId
        operator: equals
        value: qa-device-1
    variation: treatment2

qa-device-1 sees Hello there with Sign up. Change the variation to preview the next one. The activation module below skips forced results, so previews do not count in the results. Testing in production shows how to keep test devices in a segment.

Record who saw what#

Featurevisor decides who gets which variation. Your analytics compares that with what users did next, so it needs an event each time a user is shown a variation. The Google Analytics guide has a module that sends one, once per user and feature. Add it to your instance, bucketing by the same deviceId:

import { createActivationModule } from './activation-tracking.js'
f.addModule(
  createActivationModule({
    subjectId: storedDeviceId,
    identityAttribute: 'deviceId',
    send: (payload) => analytics.enqueue(payload),
  }),
)
const context = {
  deviceId: storedDeviceId,
}
// Read the variation when the hero is shown, not while prefetching it
const variation = f.getVariation('hero', context)

Reading variables alone records nothing, so read the variation at the moment the hero is shown. Reset the module when the user or session changes, as the guide describes.

End the test#

When your analytics shows a winner, send everyone to it, then remove the experiment's code and flag. Variations shows how. Changing the weights reassigns users, which is what you want once the test is over.

Run several at once#

When experiments run side by side, a user can land in more than one, and their results can affect each other. Mutually exclusive experiments keeps them apart without logic in your application.

Related#

• Variations
• Google Analytics
• Mutually exclusive experiments