LLM Serving Is KV-Cache Capacity Planning

Aug 30

By

ai infrastructurellm inferencecapacity planninggpu

Teams often size an inference fleet by model weights and GPU count. That misses the resource whose usage changes with every request: the key-value cache. Long prompts, long generations, and concurrent sequences compete for KV-cache blocks; when the budget is exhausted, requests queue, preempt, recompute, or fail.

The governing invariant is:

Admit work only when its estimated KV-cache demand and deadline fit the capacity reserved for its service class.

Why the cache dominates concurrency

During autoregressive decoding, a transformer reuses keys and values from prior tokens. The cache grows with active sequence length, layer count, head geometry, precision, and concurrent sequences. Model weights are mostly fixed; KV state is workload-shaped.

A rough planning model is:

KV bytes per token ≈ 2 × layers × KV heads × head dimension × bytes per element
request KV bytes ≈ KV bytes per token × (prompt tokens + generated tokens)

The estimate is deliberately conservative. Parallelism layout, quantization, allocator block size, and model architecture change the real number. Measure the engine rather than treating the equation as billing truth.

PagedAttention fixes allocation, not economics

The vLLM paper identifies fragmentation and redundant cache duplication as major barriers to batching. PagedAttention divides KV state into blocks and maps logical sequence blocks to non-contiguous physical blocks, borrowing the core idea of virtual memory. That reduces waste and enables sharing such as copy-on-write for related sequences.

This improves usable capacity. It does not make capacity infinite. An unbounded 128k-token request can still displace many short interactive requests.

Continuous batching changes the queue

Static batching waits for a group and runs it together, leaving capacity idle as sequences finish at different times. Continuous batching can admit new sequences between decode iterations. Throughput improves, but the scheduler is now a production control plane: its policy determines time to first token, inter-token latency, starvation, and fairness.

Separate at least two classes:

  • interactive traffic, optimized for time to first token and bounded generation;
  • batch traffic, optimized for throughput and willing to wait.

Do not allow offline summarization to occupy every cache block while customer requests queue.

Admission must happen before allocation

Validate and tokenize early enough to estimate cost. Then gate on:

const estimatedTokens = promptTokens + maxOutputTokens
const fitsRequest = estimatedTokens <= policy.maxSequenceTokens
const fitsPool = estimatedTokens <= scheduler.availableTokenSlots()

if (!fitsRequest) throw new RequestTooLargeError()
if (!fitsPool) return queueOrReject(policy.overloadMode)

Reserve maxOutputTokens; do not assume the model will stop early. Put tenant quotas and deadlines above the engine. When overload occurs, reject before spending expensive prefill compute if the request cannot complete usefully.

Watch prefill and decode separately

Prefill processes the prompt and is compute-heavy. Decode generates tokens iteratively and is often memory-bandwidth constrained. One aggregate “request latency” hides which stage is saturated.

Track:

  • time to first token;
  • inter-token latency and generation throughput;
  • queued, running, and preempted sequences;
  • KV-cache utilization and allocation failures;
  • prompt and generated-token distributions;
  • latency and rejection by service class;
  • useful tokens per accelerator-second.

If preemption rises, reducing max_num_seqs or maximum batched tokens may improve tail latency even when headline throughput falls. The correct setting depends on the SLO, not a benchmark maximum.

Common mistakes

  1. Advertising maximum context length as the default entitlement.
  2. Routing only by model name while ignoring current cache pressure.
  3. Mixing batch and interactive work in one unreserved pool.
  4. Retrying rejected generations without preserving the caller deadline.
  5. Autoscaling on GPU utilization alone; a saturated cache can coexist with misleading compute metrics.

The CTO decision

Treat context length and output tokens as scarce capacity, not UI parameters. Establish per-class limits, queue bounds, rejection behavior, and cost attribution before increasing fleet size. PagedAttention makes allocation more efficient; admission control decides whether the product remains predictable.

References

>