Many services expose one /health endpoint and wire it into startup, readiness, and liveness probes. That is convenient configuration and weak failure design. The three probes cause different state transitions, so they should answer different questions.
The invariant is:
A probe should report only the condition for the recovery action Kubernetes will take when that probe fails.
Liveness failure restarts a container. Readiness failure removes a Pod from Service traffic. Startup failure prevents the other probes from running and eventually restarts a process that cannot initialize. Treating them as synonyms converts a dependency incident into self-inflicted churn.
Begin with actions, not endpoints #
| Probe | Question | Failure action |
|---|---|---|
| startup | Has this process completed initialization? | restart after startup budget expires |
| readiness | Can this instance accept its intended traffic now? | remove it from matching Service endpoints |
| liveness | Is this process incapable of recovering without restart? | restart the container |
A database outage may make an API temporarily unable to serve requests, so readiness could fail. It does not usually mean restarting every API process will repair the database. If liveness checks the database, the outage can trigger a fleet-wide restart storm, cold caches, reconnection spikes, and more load on the failing dependency.
Keep liveness local and conservative #
Liveness should detect states where process replacement is the correct treatment: a deadlocked event loop, a permanently stopped worker, or a corrupted internal state the application knows it cannot repair.
It should be cheap, bounded, and independent of shared remote dependencies:
app.get('/live', (_req, res) => {
const eventLoopProgressing = heartbeat.ageMs() < 5_000
const workerProgressing = worker.lastProgressAgeMs() < 30_000
if (!eventLoopProgressing || !workerProgressing)
return res.status(503).end()
return res.status(204).end()
})Even this logic needs workload context. A worker with no jobs is not stuck. Progress signals must distinguish “nothing to do” from “unable to do it.”
Readiness is an admission decision #
Readiness decides whether a particular instance should receive more work. It may include strictly required local and remote conditions, but avoid a naïve fan-out to every dependency on every probe interval.
If each of 500 Pods probes five downstream services every second, the health system creates 2,500 requests per second before customer traffic arrives. A slow dependency then occupies probe sockets and CPU across the fleet.
Maintain bounded, asynchronous dependency state instead:
let databaseReady = false
setInterval(async () => {
databaseReady = await checkDatabaseWithDeadline(200)
}, 2_000)
app.get('/ready', (_req, res) => {
if (draining || !configurationLoaded || !databaseReady)
return res.status(503).end()
return res.status(204).end()
})The endpoint remains fast while the background check has its own deadline. Decide how stale the cached readiness signal may be, and expose its age for diagnosis.
Not every dependency belongs in readiness. If recommendations can fall back to popular items, the recommendation service should not remove itself because its ranking store is down. Readiness should represent minimum viable service, not perfect service.
Startup needs a measured budget #
Startup probes protect slow initialization from premature liveness restarts. Their effective budget is approximately:
failureThreshold × periodSecondsSet it from observed cold-start distributions plus operational margin, not from one developer laptop. Include image startup, secrets delivery, migrations or cache hydration, and runtime compilation where applicable.
Do not make every replica run schema migrations before becoming ready. Separate one-time control-plane work from replica startup when possible. Otherwise a slow lock can prevent an entire deployment from starting.
Thresholds are temporal policy #
The default one-second timeoutSeconds can be too aggressive for loaded nodes; an extremely long timeout hides failure. failureThreshold filters brief noise, while successThreshold on readiness prevents a flapping instance from re-entering too quickly.
Model the timeline:
detection time ~= periodSeconds × failureThreshold + probe timeout effects
re-entry time ~= periodSeconds × successThresholdThat timeline should fit the service SLO and the capacity reserve. If readiness removes Pods faster than autoscaling or recovery can replace capacity, one transient slowdown can cascade into overload on the survivors.
Probe behavior belongs in load tests #
Test probes under conditions that matter:
- CPU throttling and event-loop delay;
- exhausted database pools;
- DNS and dependency latency;
- rolling deployments at minimum replica count;
- node drain and graceful termination;
- cold starts after a regional scale-up;
- partial dependency failure with fallback paths active.
Track probe latency and outcomes, Pod readiness transitions, restarts by reason, time to ready, endpoint churn, traffic error rate, and downstream connection attempts. Correlate them on one incident timeline.
Common mistakes #
- Using the same handler for all three probes.
- Calling every dependency from liveness.
- Returning a large diagnostic payload; probes need a status, not a dashboard.
- Hiding overload behind a readiness failure without retaining capacity headroom.
- Using a TCP-open check for a process whose worker loop can be deadlocked.
- Treating readiness removal as instantaneous connection draining.
- Forgetting that a probe itself consumes CPU, sockets, processes, and logs.
The CTO decision #
Health checks are executable recovery policy. For each failure signal, state who is unhealthy, what automatic action follows, how quickly it should happen, and whether that action improves the situation under a shared dependency outage.
Good probes make the smallest safe state transition. They stop traffic before restarting, distinguish startup from steady state, and never multiply a remote incident into fleet-wide self-destruction.