Kubernetes Termination Is a Distributed Protocol

Handling SIGTERM is necessary for graceful shutdown. It is not sufficient.

A terminating Kubernetes Pod participates in several concurrent systems: the API server marks it for deletion, EndpointSlices change readiness state, kubelet may execute a preStop hook, the runtime signals PID 1, proxies and load balancers converge, clients reuse connections, and the application drains work. These events are coordinated, not instantaneous.

The shutdown invariant is:

Stop accepting new work before the grace budget expires, finish or safely transfer accepted work, then terminate.

There is no universal ordering #

Kubernetes begins local shutdown while the control plane updates service endpoints. A terminating endpoint remains represented but has ready: false; systems aware of the newer conditions can inspect whether it is still serving. External load balancers and client-side caches may observe that change later.

That creates an overlap:

endpoint withdrawal ──────────────► converges
application signal ──► draining ──► exits
existing connections ─────────────► may still send requests

If the process exits immediately on SIGTERM, late traffic fails. If it keeps reporting ready and accepts new work forever, the grace period ends with SIGKILL.

Budget the whole path #

The termination grace countdown includes the preStop hook. A 30-second grace period with a 20-second sleep leaves roughly 10 seconds for the application to drain. Kubernetes may grant a small one-off extension when a hook is still running, but that is not a capacity plan.

Use an explicit budget:

propagation allowance + maximum request duration
+ worker drain time + safety margin
< terminationGracePeriodSeconds

The numbers must come from production behavior. A service with 200 ms requests and a stream processor with ten-minute leases need different protocols.

Application shutdown sequence #

process.on('SIGTERM', async () => {
  readiness.set(false)
  server.close()                 // stop new connections
  await workers.stopClaiming()   // stop new background work
  await Promise.race([
    Promise.all([server.drained(), workers.drained()]),
    deadline(20_000),
  ])
  await telemetry.flush()
  process.exit(0)
})

This sketch still needs bounded operations and cancellation. Shutdown code that waits indefinitely converts a graceful deployment into forced termination.

For queue workers, distinguish “claimed” from “completed.” A lease or visibility timeout must allow another worker to recover abandoned work. For HTTP writes, idempotency protects the caller that received a disconnect after the server committed.

Failure policy #

Verify with disruption #

Deployment success is not proof. Run repeated rolling updates under load. Track response-code spikes, terminated requests, queue lease recovery, shutdown duration, and forced kills. Test persistent connections and slow requests. Confirm PID 1 receives signals in the actual container image.

Avoid a blind preStop: sleep. A small propagation delay can be pragmatic, but it should accompany application draining and be justified by measured convergence.

Deployment is the acceptance test #

A readiness probe that turns false is useful only if every routing layer responds in time. Verify the complete path: Kubernetes Service, ingress, cloud load balancer, service mesh, and any client-side discovery cache. Track requests that arrive after the application entered draining state. That counter turns a timing assumption into evidence.

Keep shutdown dependencies minimal. If draining requires a database that is already unavailable, use bounded cleanup and rely on leases or idempotency for recovery. Termination must remain safe during the same partial failures that often trigger rescheduling.

Conclusion #

Graceful termination is a protocol across routing, process lifecycle, workload ownership, and time. Kubernetes supplies signals and state transitions; the application supplies the business meaning.

Design shutdown with the same care as startup. A service that cannot leave safely is not highly available—it is merely easy to start.

Further reading: Kubernetes Pod lifecycle, container lifecycle hooks, termination-flow tutorial, and Kubernetes availability budgets.