Control Planes Must Fail Quietly

Aug 26

By

distributed systemscontrol planesreliabilityplatform engineering

A control plane decides what the system should become. A data plane serves the request in front of the customer. When those responsibilities share the same synchronous failure path, an administrative outage becomes a product outage.

The governing invariant is simple:

Existing, authorized traffic should continue safely for a bounded period when the control plane is unavailable.

That is not always possible—credential revocation and risk controls may require fail-closed behavior—but it should be a conscious exception.

Separate desired state from serving state

Kubernetes provides a useful model. Its API server exposes desired state; controllers repeatedly compare desired and observed state and act to reduce the difference. Worker nodes run the workloads. The architecture is not valuable because every product needs Kubernetes. It is valuable because reconciliation tolerates delay and retry.

operator → control API → desired state store
                         ↓
                    reconciler
                         ↓
                  serving snapshot → data plane

The data plane should consume a validated, versioned snapshot—not query a mutable management database during every request.

For a feature-routing system, the snapshot might be:

{
  "version": 1842,
  "generated_at": "2026-08-26T03:20:00Z",
  "routes": [{"tenant":"acme","upstream":"cluster-b"}],
  "valid_until": "2026-08-26T04:20:00Z"
}

The version allows monotonic application. The validity window bounds staleness. The data plane can keep serving version 1842 while the control API is temporarily unavailable.

Make reconciliation idempotent

A controller will retry after timeouts, crashes, and leadership changes. Therefore the operation must converge when executed repeatedly.

Bad:

on event: increment desired replica count

Better:

observe replicas = 3
desired replicas = 5
create replicas until observed = 5

Events can wake a reconciler, but current state should decide the action. Persist an operation identity for external effects and use compare-and-set or resource versions so two reconcilers cannot overwrite each other blindly.

Model the loop explicitly:

read desired → read observed → compute delta
→ apply bounded action → record result → requeue

One loop should make limited progress. A controller that attempts to repair ten thousand resources in one transaction creates long locks and large failure domains.

Define stale-state policy per decision

Not all configuration has the same safety profile.

StateDuring control-plane outage
route tableserve last valid snapshot
price catalogueserve briefly, then stop checkout
revoked credentialfail closed after short TTL
UI experimentuse deterministic default
rate limitretain last limit locally

“Fail open” and “fail closed” are incomplete without a time horizon. Decide the maximum stale age, default behavior, and customer-visible degradation for every control object.

Keep an emergency path small. If disabling a dangerous integration requires the same broken deployment system that introduced it, the control plane cannot control the incident.

Protect the data plane from control churn

Administrative activity can be bursty: a bulk import, policy rollout, or controller bug may touch every tenant. Bound its effect with:

  • rate-limited reconciliation;
  • per-tenant work queues;
  • jittered retries;
  • generation numbers that collapse obsolete updates;
  • priority for safety changes;
  • circuit breakers around external dependencies.

Never let reconcilers share an unbounded resource pool with customer traffic. Separate connection pools, worker queues, quotas, and ideally compute capacity. Otherwise a repair storm consumes the system it is trying to repair.

Publish snapshots atomically

Data-plane readers must not observe half a configuration. Build the next snapshot, validate it, then switch a pointer atomically:

write config/version-1843
validate schema + invariants
compare-and-set active: 1842 → 1843
notify readers

Readers retain the last known-good version if the new one fails verification. Record rejection reason and control-plane version in telemetry.

Compatibility matters. During rollout, old data planes may read new snapshots. Version the schema, support an overlap window, and test rollback. A new control plane that emits state the old data plane cannot parse removes your rollback path.

Observe convergence, not only availability

A healthy API server does not prove that desired state reached production. Measure:

  • desired-to-observed convergence time;
  • age and version of serving snapshots;
  • reconciliation attempts and terminal failures;
  • queue age by tenant and priority;
  • data planes on unsupported versions;
  • rejected snapshots;
  • control-plane dependency saturation.

Alert on customer-impacting drift: “4% of tenants are more than two versions behind” is more useful than “controller CPU is 80%.”

CTO review

  1. Which customer paths synchronously depend on the control plane?
  2. How long may each type of state remain stale?
  3. Are reconciler actions idempotent and bounded?
  4. Can control churn exhaust data-plane resources?
  5. Is serving configuration published atomically and versioned?
  6. Can old data planes read state produced during a new rollout?
  7. What metric proves desired state converged?
  8. Is there an independent emergency-disable path?

A good control plane changes production deliberately. A great one can be broken for an hour without making the customer discover it first.

References

>