gRPC gives services a typed RPC model and commonly carries it over HTTP/2. That transport provides multiplexed streams, framing, and flow control. None of those features decides whether retrying a payment is safe, when a request is no longer useful, or how an overloaded dependency sheds work.
Transport capability and application reliability are different layers.
What HTTP/2 actually provides #
HTTP/2 associates concurrent request/response exchanges with streams on one connection. It applies flow control at both stream and connection level. Receivers advertise credit with WINDOW_UPDATE; senders must remain inside both windows.
This creates useful efficiency, but also shared fate. If connection-level flow-control credit is exhausted, all data-bearing streams on that connection are constrained. A connection reset can affect many in-flight RPCs. Multiplexing removes HTTP/1.1 request queuing on separate application exchanges, but TCP loss and shared connection state still matter.
Four policies the transport cannot choose #
Deadline #
A caller should declare when the result stops being valuable. Without a deadline, work can accumulate across hops after the user or upstream request has gone away.
ctx, cancel := context.WithTimeout(parent, 250*time.Millisecond)
defer cancel()
reply, err := client.Lookup(ctx, req)Each service must propagate the remaining budget, reserve time for its own cleanup, and stop downstream work on cancellation.
Retry #
A retry is a new attempt and additional load. It needs an error policy, bounded attempt count, backoff with jitter, and sufficient deadline budget. Connection failure does not prove the server did not commit an operation.
Idempotency #
For writes, carry an operation identity and persist its outcome at the same authority that applies the side effect. Transport stream IDs are connection-scoped mechanics, not business idempotency keys.
Admission control #
HTTP/2 flow control protects buffers between adjacent endpoints. It does not know a request’s database cost, tenant quota, model-token demand, or deadline. Concurrency limits and load shedding still belong at the application boundary.
Failure policy #
| Condition | Default response |
|---|---|
| caller cancelled | stop work immediately |
| deadline expired | fail; do not begin another attempt |
| explicit retryable unavailable | retry only within budget |
| validation failure | do not retry |
| ambiguous write outcome | resolve by idempotency key |
| local concurrency full | reject early with useful status |
| connection draining | shift new calls; let bounded calls finish |
Observe both layers #
RPC metrics should include method, final status, attempt count, remaining deadline at entry, handler concurrency, and queue time. Transport metrics should include active streams, connection churn, flow-control stalls, bytes, and reset causes. Correlating them prevents a transport symptom from becoming a fictional application diagnosis.
Trade-offs #
Long-lived multiplexed connections reduce handshake and socket overhead, while increasing correlated impact when one connection degrades. More connections can isolate traffic classes but use more sockets and memory. Aggressive retries improve recovery from brief faults but amplify overload. Short deadlines bound waste but fail requests that might have completed.
The architecture is complete only when the protobuf contract, transport behavior, and failure policy agree. HTTP/2 carries the call. It does not decide whether the call remains correct under uncertainty.