Feature Flags Are Production State

Aug 26

Feature flags decouple deployment from release. They also create a second production configuration system capable of changing behavior without a code review, build, or deployment.

Treating flags as temporary booleans hides their real role: they are versioned production state with owners, permissions, failure modes, and retirement obligations.

Classify the flag before creating it

Different flags require different controls:

TypePurposeExpected lifetime
releaseprogressive rolloutdays or weeks
experimentmeasure variantsbounded by analysis
operationaldegrade or reroutelong-lived
permissioncontractual entitlementproduct lifetime
emergencydisable dangerous pathlong-lived, rarely changed

Do not use a release flag as an entitlement system. Do not use an experiment flag as a security boundary. Classification defines who may change it, its default, required telemetry, and removal date.

Store metadata with the flag:

key: checkout.new-tax-engine
type: release
owner: team-commerce
created: 2026-08-26
expires: 2026-09-30
safe_default: false
rollback_signal: tax_error_rate
ticket: COM-1842

A flag without an owner and expiry is deferred code complexity.

Make evaluation typed and deterministic

OpenFeature defines typed evaluation methods with a caller-supplied default. The type and default are part of the application contract.

const enabled = client.getBooleanValue(
  'checkout.new-tax-engine',
  false,
  { targetingKey: accountId, region },
)

Choose defaults by failure analysis:

  • new optional UI: default off;
  • safety rate limit: last known value or conservative limit;
  • dangerous integration: default disabled;
  • purchased entitlement: use durable entitlement authority, not an unavailable experiment provider.

Evaluate once per operation and pass the decision through the call path. Re-evaluating at several layers can produce mixed behavior if configuration changes mid-request.

Fractional rollout must be sticky. Hash a stable, non-sensitive targeting key with the flag key so the same subject receives the same variant. Random evaluation per request destroys user experience and makes results uninterpretable.

Minimize targeting data

Evaluation context may include users, services, regions, or hosts. OpenFeature defines a targeting key and custom fields; it does not require sending an entire customer record.

Send only attributes needed by rules. Avoid email, names, access tokens, and free-form profiles. Document where context is evaluated and retained. Client-side evaluation can expose flag rules and targeting data to the browser; use server-side evaluation for sensitive policy.

Tenancy is non-negotiable. If a rule targets account_id, obtain it from authenticated context rather than a request parameter the caller can forge.

Make rollout a sequence of evidence

A safe rollout is not “10%, then 50%, then 100%.” Each step has entry and rollback conditions.

internal → 1% → 5% → 25% → 50% → 100% → remove flag

At each stage compare candidate and control across:

  • business success;
  • errors and saturation;
  • p95 and p99 latency;
  • database and dependency load;
  • cost per successful outcome;
  • segment-specific harm.

Percentage alone can hide concentration. One percent of traffic may contain no enterprise tenant or large account. Include named cohorts for risky boundaries.

Record the evaluated variant in traces and business events. OpenFeature’s observability guidance maps evaluation details such as flag key, variant, and reason into telemetry. Do not attach high-cardinality or sensitive evaluation context indiscriminately.

Design for provider failure

Flag evaluation may use local snapshots, streaming updates, or synchronous remote calls. A remote network call in every customer request creates a new critical dependency.

Prefer locally evaluated, versioned snapshots for latency-sensitive decisions. Keep the last known-good snapshot, define its validity window, and expose its age.

flag provider unavailable
  → use cached snapshot if valid
  → otherwise use declared safe default
  → emit degraded evaluation telemetry

Operational kill switches need a tested path during provider impairment. If the same control plane is down, can on-call still disable the feature? Consider a small independent emergency override with narrow scope and audited access.

Prevent combinatorial states

Five interacting booleans create up to 32 combinations. Most will never be tested.

Reduce interaction by:

  • grouping mutually exclusive behavior into a typed variant;
  • forbidding one flag from changing the meaning of another;
  • testing supported combinations explicitly;
  • keeping evaluation near one architectural boundary;
  • retiring release flags quickly.
bad: use_new_api + use_new_schema + use_new_cache
better: checkout_architecture = legacy | shadow | v2

Structured variants express the state machine more honestly.

Remove the flag completely

At 100%, the work is not done. Delete:

  • the old branch;
  • flag evaluation calls;
  • control-plane configuration;
  • obsolete tests and metrics;
  • fallback schemas and dependencies;
  • dashboards that no longer mean anything.

Automate expiry reporting. Block creation without an owner. Track median flag age by type, and place removal in the original delivery plan rather than a future cleanup backlog.

Before deleting, verify there are no evaluations for the old variant across scheduled jobs, mobile versions, or dormant tenants. Telemetry plus a consumer inventory is stronger than code search alone.

CTO review

  1. What class of flag is this, and when does it expire?
  2. Is the default safe during provider failure?
  3. Is targeting deterministic, authenticated, and privacy-minimized?
  4. Which evidence advances or reverses each rollout stage?
  5. Can combinations produce untested system states?
  6. Does the data plane require a synchronous control-plane call?
  7. Is the kill switch available during a control-plane outage?
  8. Who removes both branches and the flag?

Flags make release safer only when their own lifecycle is engineered. Otherwise they move deployment risk into an invisible, mutable control plane.

References

>