Kubernetes Availability Is Placement Plus Disruption Policy

Aug 30

Three replicas on one node are one failure domain wearing three Pod names. Kubernetes availability comes from where replicas can run, what may evict them, whether replacements can fit, and when traffic considers them ready.

The invariant is:

After any planned single-domain disruption, enough ready capacity remains to serve the declared load.

Replica count is only an input to that statement.

Placement encodes the failure model

Topology spread constraints let the scheduler distribute matching Pods across node, zone, or another labeled domain. A strict zone constraint might be:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: checkout

maxSkew: 1 limits imbalance. DoNotSchedule preserves the placement invariant by refusing an unsafe placement. ScheduleAnyway prioritizes progress and treats spread as a preference. Neither is universally correct: strict policy can leave Pods pending when one zone lacks capacity, while soft policy may silently concentrate the service.

The selector must match the workload’s own Pod labels. Otherwise the scheduler can create “ghost” placements that do not count themselves in the calculation.

A PDB governs only voluntary eviction

A PodDisruptionBudget limits how many selected Pods may be unavailable during voluntary disruptions that use the Eviction API, such as a respectful node drain.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout

It does not create replicas, add capacity, repair a bad readiness probe, or protect against every deletion path. Kubernetes documentation explicitly notes that direct deletion of Pods or Deployments can bypass PDBs. Involuntary failures such as hardware loss are not prevented either.

A PDB is therefore a maintenance admission policy—not an availability guarantee.

Capacity closes the loop

Strict spreading can deadlock a rollout if the cluster has no spare slot in the required zone. Autoscaling may not recognize a topology domain whose node group is scaled to zero unless the autoscaler understands the full domain set.

Before enabling strict policy, test:

  1. one zone unavailable;
  2. one node draining during a deployment;
  3. a replica already unready when maintenance begins;
  4. the largest expected surge plus one failed domain;
  5. a scale-from-zero node group.

Reserve enough headroom for replacement Pods. A design requiring all healthy nodes to stay at 90% utilization cannot honestly claim single-node fault tolerance.

Readiness is part of the budget

The PDB counts health using Pod readiness. A probe that turns green before caches, connections, or migrations are ready allows the platform to evict another replica too early. A probe that depends on every downstream service can make the whole fleet unready during a dependency incident.

Readiness should answer one narrow question: can this instance safely accept its class of traffic now?

Review the complete policy

BoundaryQuestion
replicashow many ready instances does peak load require?
placementwhich node, zone, or rack losses must be tolerated?
disruptionhow much planned unavailability may proceed?
capacitywhere can replacements actually schedule?
readinesswhen is a replacement truly serving?
rolloutcan old and new versions coexist within the same budget?

Observe replicas per topology domain, unschedulable Pod reasons, PDB-blocked evictions, rollout duration, readiness latency, and available capacity after a simulated domain loss.

The CTO decision

Declare the failure domain before choosing replica count. Use strict spread where concentration violates the service objective, soft spread where degraded placement is preferable to no placement, and PDBs to encode the amount of voluntary disruption the application can absorb. Then prove that capacity and readiness make those declarations true.

Availability is not “replicas: 3.” It is a tested relationship between placement, health, disruption, and spare capacity.

References

>