Test feature flags before deployment

A change to a definition can change what users see, with no application code changing at all. Test it next to the definition, and let CI stop a publish when a test fails.

The change under review#

A pull request gives users of the new checkout their own support address. It adds the newCheckout override to supportEmail from Variables:

description: Support email shown in checkout
type: string
defaultValue: [email protected]
overrides:
  - key: priority
    conditions: payingAccounts
    value: [email protected]
  - key: netherlands
    conditions: netherlands
    value: support-[email protected]
  - key: newCheckout
    conditions:
      - feature: checkout
    value: checkout-[email protected]

description: Support email shown in checkout
type: string
defaultValue: [email protected]
overrides:
  - key: priority
    conditions: payingAccounts
    value: [email protected]
  - key: netherlands
    conditions: netherlands
    value: support-[email protected]
  - key: newCheckout
    conditions: feature('checkout')
    value: checkout-[email protected]

Four kinds of user now get four different addresses, and the first override that matches wins. A test for each keeps a later change from moving anyone by accident.

Test every path#

Give each override, and the default, an assertion:

variable: supportEmail
assertions:
  - description: Paying accounts get priority support
    context:
      country: nl
      plan: paid
    expectedValue: [email protected]
  - description: Dutch users get the Dutch team
    context:
      country: nl
    expectedValue: support-[email protected]
  - description: Users of the new checkout get its team
    at: 10
    context:
      country: de
    expectedValue: checkout-[email protected]
  - description: Everyone else gets the default
    at: 50
    context:
      country: de
    expectedValue: [email protected]

A global variable is not bucketed itself, so at sets the bucket for the features it depends on. At 10, a German user is inside checkout's 25%, and at 50 they are outside. Testing covers the rest of the syntax.

Test the relationship#

The new override ties supportEmail to checkout. A scenario checks both together, so a later change to either shows up as a failure:

description: The new checkout comes with its own support team
at: 10
context:
  country: de
checks:
  - feature: checkout
    enabled: true
  - variable: supportEmail
    operator: equals
    value: checkout-[email protected]

If someone narrows checkout to the Netherlands later, this scenario fails, and the review sees that support routing for German users changes too.

Run them before publishing#

Lint, test and build in every pull request, and again before publishing:

npx featurevisor lint
npx featurevisor test
npx featurevisor build

A failing test exits with a nonzero code, which stops the CI job before anything is published. GitHub Actions has both workflows, and Deployment explains the order.

What these tests do not cover#

They check how definitions evaluate. They do not check that your application renders the new checkout, shows the address or fetches the datafile. Keep those in your application's own tests. If applications load filtered datafiles, add a target to the assertions that depend on what a target includes.

Related#

• Testing
• Conditions
• Configuration rollback