Kill switches

When something breaks in production, a kill switch turns it off for everyone without a release. Decide before the incident which switch covers what, who can flip it and how fast the change reaches users.

How it works#

• Every feature is already a switch. A force entry turns it off for everyone, whatever its rules say.
• A boolean global variable switches off behaviour that spans several features, such as the whole checkout while a payment provider is down.
• Flipping either is a pull request like any other change. Your pipeline publishes the datafile, and applications pick it up on their next refresh.

Switch one feature off#

Add a force entry that matches everyone to checkout:

# ...
force:
  - conditions: '*'
    enabled: false

Force entries are checked before the rules, so every user gets false, Dutch users included, with the reason forced. The rules stay as they were. Removing the entry later gives the new checkout back to exactly the users who had it, since their buckets never changed.

Switch off a family#

A feature that requires another goes off with it. chatFileUpload requires liveChat, as Managing dependencies shows, so forcing liveChat off switches off both in one change.

Switch off a whole flow#

Some incidents are bigger than one feature. Keep a boolean global variable for them, off by default:

description: Maintenance kill switch
type: boolean
defaultValue: false

Features that should stop during maintenance depend on it through their conditions. checkout stays on only while the switch is false:

# ...
conditions:
  - variable: maintenanceMode
    operator: equals
    value: false

Settings can follow it too, such as supportEmail sending everyone to a status page address. The application can also read the switch directly, to show a maintenance page:

const context = {
  userId: '8f2a',
  country: 'nl',
}
if (f.getVariableBoolean('maintenanceMode', context)) {
  // show the maintenance page
}

To flip it for everyone, set defaultValue: true in a pull request. To switch off one market only, such as during a local payment outage, add an override instead:

# ...
overrides:
  - key: netherlands
    conditions: netherlands
    value: true

Dutch users get maintenance, and everyone else carries on.

Comparing with false makes checkout fail closed: a datafile without the switch keeps checkout off rather than on. Lint stops anyone archiving the switch while a feature still depends on it.

Make it fast#

A switch is only as fast as three waits added together: the pipeline, the cache in front of the datafiles and the application's next fetch. How long a change takes adds them up. For a switch that must land within a minute, keep the cache short and refresh when told, such as on a message after each deployment.

Mobile apps usually fetch when they open or return to the foreground, so a phone that stays in the background picks the change up later, as React Native explains.

Decide who can flip it#

• Name who may approve a switch out of hours, and make sure branch protection lets them merge without waiting for the usual reviewers. Establishing ownership shows how owners and required reviews work.
• Rehearse it. Flip the switch in staging, watch the change arrive and time it.
• Write down which switch covers what, next to your incident runbook, so nobody searches the repository during an outage.

Test it#

A spec keeps a switch from being left on by accident:

variable: maintenanceMode
assertions:
  - description: Off unless someone switches it on
    context:
      country: nl
    expectedValue: false

To check what a switch turns off, set it with stickyVariables in a feature spec, as Testing lists, and expect the feature to be disabled.

Related#

• Flags, for the force entry
• Configuration rollback, for undoing a change
• Deployment, for the timing