Most timeout configurations are numbers copied from another service. 30s feels conservative, 5s feels responsive, and nobody can explain what either number protects.
That is not configuration. It is an undocumented availability policy.
A deadline decides how long a caller will reserve memory, a connection, a worker, and user patience for an operation. A retry decides how much additional load the system may create when it is already unhealthy. Together they define whether partial slowness remains local or becomes a fleet-wide outage.
The architecture question is therefore not “what timeout should this HTTP client use?” It is: how will one end-to-end latency budget be spent, propagated, and enforced across every dependency?
Start with the user-visible deadline #
Assume an API must complete within 800 ms at the 99th percentile. Its path is:
client -> gateway -> orders -> inventory -> database
\-> pricingGiving every hop an 800 ms timeout does not preserve an 800 ms experience. It permits sequential work to consume multiples of the budget and leaves no time to return a useful failure.
Instead, carry an absolute deadline or remaining duration:
request budget 800 ms
gateway admission + routing 40 ms
orders local work 80 ms
inventory including database 260 ms
pricing 180 ms
response serialization + safety margin 90 ms
unallocated contingency 150 msThese are not universal numbers. The important design is explicit ownership. Every layer knows its maximum spend and stops work that can no longer affect the response.
gRPC recommends setting deadlines because, by default, a client may otherwise wait indefinitely. It also supports deadline propagation, converting the deadline into a timeout for the next hop while accounting for elapsed time. That is the correct mental model even outside gRPC: downstream work inherits a shrinking budget; it does not receive a fresh lease on the user’s patience.
Cancellation is part of correctness #
A caller timing out is not enough. If the downstream service continues querying, rendering, or calling another vendor, the user has left but the cost remains.
Every blocking boundary should answer three questions:
- Can the operation observe cancellation?
- Does cancellation release the scarce resource promptly?
- If the work cannot be cancelled, is it bounded and isolated?
In application code, pass cancellation context rather than inventing a new timeout at each function:
func Quote(ctx context.Context, orderID string) (Quote, error) {
ctx, cancel := context.WithTimeout(ctx, 180*time.Millisecond)
defer cancel()
return pricing.GetQuote(ctx, orderID)
}The local cap protects the caller, while the parent context ensures an earlier end-to-end cancellation wins.
For a database, configure both client cancellation and a server-side statement limit. A dead client socket is not an operational policy. For background work, use leases so abandoned work becomes recoverable rather than immortal.
Retries spend a second budget #
Retries are selfishly rational and globally dangerous. One caller sees a transient failure and tries again. Thousands of callers see the same failure and multiply load at the moment the dependency has the least capacity.
With five layers and three attempts per layer, a single original request can theoretically produce 3^5 = 243 calls at the bottom if every layer retries independently. Real systems have branching and early successes, but the lesson holds: retry placement is an architectural decision.
Choose one retry owner for a call path. Usually it is the layer that:
- knows whether the operation is idempotent;
- has enough remaining deadline;
- understands the user-visible outcome;
- can observe the full attempt history.
A retry policy needs more than maxAttempts:
retry:
retryable: [UNAVAILABLE, RESOURCE_EXHAUSTED]
maxAttempts: 3
perAttemptTimeout: 120ms
backoff: exponential
jitter: full
budget: 10% of baseline trafficThe retry budget is the critical line. It limits retry traffic relative to healthy request volume. When the budget is exhausted, fail fast instead of converting a dependency incident into an overload incident.
Only retry a safe semantic operation #
“POST is not idempotent” is too crude, and “our handler is idempotent” is usually too optimistic. Retry safety belongs to the business effect.
Creating a payment can be retried only if every attempt carries the same idempotency key and the receiver persists the resulting outcome:
Idempotency-Key: checkout_92f1_payment_v1The key must identify the logical operation, not the network attempt. If a timeout leaves the outcome ambiguous, query by that key before issuing a new effect.
Reads are not automatically harmless either. A heavy analytical query retried after a client timeout may double the exact load that caused the timeout. Safety includes capacity, not just data mutation.
Backoff without jitter synchronizes failure #
Exponential backoff spaces attempts, but identical clients still wake together. Jitter randomizes the delay so recovery traffic arrives as a slope rather than a wall. AWS’s Builders’ Library describes this as a core technique for avoiding correlated retry storms.
Use server hints where available (Retry-After, explicit overload metadata), but cap them by the remaining deadline. A request with 90 ms left cannot honor a 2-second retry recommendation.
remaining = deadline - now
delay = min(full_jitter(base * 2^attempt), server_hint, remaining - execution_margin)If the remaining time cannot fund both delay and a meaningful attempt, do not retry.
Measure deadline economics #
Average latency will hide the failure mode. Operate the policy with:
- end-to-end deadline-exceeded rate;
- remaining budget at each service entry and exit;
- attempts per logical operation;
- retry success rate by attempt number;
- retry volume as a percentage of baseline traffic;
- cancelled work that continued executing;
- dependency latency distributions by outcome;
- load-shed responses versus accidental timeouts.
A high retry-success count is not automatically good. It may reveal a dependency that is unreliable enough to require constant hidden duplication.
Common mistakes #
One timeout everywhere. Different work has different value and cost; the only shared value should be the inherited deadline.
Retrying at every layer. This creates multiplicative traffic and destroys causal evidence.
Timing out without cancelling. The caller leaves while resource consumption continues.
Retrying non-idempotent effects. An ambiguous failure becomes a duplicate business action.
Using circuit breakers without admission control. A breaker can reduce calls to one dependency, but it does not ensure the rest of the service can survive the queued demand.
Treating p99 as a timeout. A timeout must include the end-to-end objective, downstream cost, false-timeout tolerance, and recovery policy—not merely yesterday’s percentile.
CTO review questions #
- What user-visible deadline is the system protecting?
- How is remaining time propagated across protocols and queues?
- Which layer owns retries, and what prevents amplification?
- Which business operations have durable idempotency keys?
- What happens to work after its caller disappears?
- Can we distinguish healthy retries from retry-driven overload?
Timeouts and retries are not resilience decorations. They are a distributed resource-allocation protocol. Design them with the same care as a database schema, because during an incident they decide who waits, who retries, and whether the system gets a chance to recover.