Rate Limiting Is Admission Control

Sep 3

By

backend engineeringreliabilityapi designdistributed systemsplatform engineering

Rate limiting is usually presented as a counter: allow 100 requests per minute. Production systems need a harder question—which work should enter when demand exceeds the capacity to finish it usefully?

The invariant is:

Reject work before it consumes the scarce resource, using an identity and budget aligned with the business promise.

A gateway request counter cannot protect a database if one accepted export query consumes 10,000 times more work than one cached read. Requests, concurrency, bytes, tokens, rows scanned, and downstream calls are different currencies.

Put the limit near the bottleneck

Use layered admission:

edge: abusive IP and gross tenant rate
API: authenticated plan and operation cost
worker: queue depth and deadline
database: pool/concurrency budget
AI gateway: token and accelerator budget

The outer layer sheds cheaply; the inner layer protects the real resource. Do not let each layer retry the next one independently or rejection becomes amplification.

Identity determines fairness

Per-IP limits punish offices behind NAT and are weak against distributed clients. Per-user limits can let one organization consume the fleet through many users. Per-tenant limits need endpoint or cost weighting. Most platforms need a hierarchy with a global safety ceiling plus tenant and principal budgets.

Reserve capacity for recovery and control operations. If bulk exports consume every worker, cancellation, status checks, and incident tools must still run.

Rate and concurrency solve different failures

A token bucket controls arrivals over time and permits bounded bursts. A concurrency limit controls in-flight work. When dependency latency rises, the same arrival rate creates more concurrency:

in_flight ~= arrival_rate × service_time

That is why a fixed requests-per-second limit can fail during a slowdown. Adaptive concurrency or queue-deadline admission protects finite sockets, threads, database connections, and memory.

Rejection is an API contract

RFC 6585 defines 429 Too Many Requests; Retry-After tells a cooperative client when to try again. Return a stable machine-readable error, the budget scope, and a retry time when known. Add jitter client-side and preserve the original operation deadline.

HTTP/1.1 429 Too Many Requests
Retry-After: 7
Content-Type: application/problem+json

{"type":"rate-limit","scope":"tenant","retryable":true}

Do not promise a precise reset if distributed counters provide only approximate agreement. A conservative hint is better than false certainty.

Distribution has a consistency price

Local buckets are fast and available but allow a tenant to multiply its rate across replicas. A global service enforces fairness more closely but adds latency and another dependency. Hybrid designs allocate leases of budget to each region, accepting bounded overshoot in exchange for local decisions.

Write the maximum overshoot:

overshoot <= regions × local lease size

If that bound can exhaust the protected resource, reduce leases or add a hard global circuit breaker.

Measure accepted and rejected work by scope, limiter latency, concurrency, queue age, completion before deadline, distributed overshoot, and customer success—not only 429 count. Load-test skewed tenants and slow dependencies.

The CTO decision

Define capacity in the units that drive cost and failure. Allocate it by customer promise, protect control traffic, combine arrival and concurrency limits, and make rejection actionable.

A rate limiter is successful when admitted work completes predictably—not when a counter is perfectly accurate.

References

>