Static batching waits for a group of requests, pads work to compatible shapes, and completes the group together. Continuous batching revisits the active set at generation steps: finished requests leave and waiting requests can enter.
That improves accelerator utilization. It also turns the inference runtime into a scheduler deciding whose tokens are computed, whose prompt is admitted, and who waits behind a large prefill.
Continuous batching is therefore a product policy expressed through GPU scheduling.
The two workloads are different #
Prefill processes the input prompt and is highly parallel, but a long prompt can consume substantial compute and allocate a large KV cache. Decode generates one or a small number of tokens per active sequence repeatedly and is often constrained by memory movement.
request -> tokenize -> admission -> prefill -> repeated decode -> finish
| |
+-- KV cache -+If the scheduler admits only by request count, one 100,000-token prompt can be treated like one 500-token prompt even though their resource demands differ dramatically.
Use token-aware admission. Bound total scheduled tokens, active sequences, per-request context, and reserved KV-cache headroom. Keep a queue deadline so requests that can no longer meet their SLO are rejected rather than computed uselessly.
Define fairness explicitly #
First-in-first-out is simple but can let large prefills block interactive requests. Always preferring short jobs reduces mean latency while starving long-context work. Tenant-blind scheduling lets one customer occupy the batch.
A practical policy separates traffic classes:
| Class | Objective | Guardrail |
|---|---|---|
| interactive | low time to first token | small prompt cap, reserved capacity |
| standard generation | balanced latency and throughput | weighted fair share |
| batch/offline | tokens per second | preemptible, no interactive reservation |
| administrative | predictable control access | isolated small pool |
Weighted queues should charge by estimated tokens or GPU time, not request count. Enforce tenant concurrency and token-rate budgets before work enters the runtime.
Cancellation is capacity recovery #
Clients disconnect, abandon streams, or reach deadlines. Propagate cancellation into the engine, remove waiting requests immediately, and reclaim finished or cancelled sequence state safely. A proxy noticing disconnect while the model continues generating is a capacity leak.
Cancellation must not corrupt a shared batch. The scheduler removes one sequence at a defined step boundary while preserving the state of others.
PagedAttention changes memory management, not limits #
The vLLM paper introduces PagedAttention to manage KV-cache memory in blocks, reducing fragmentation and enabling sharing. Better allocation increases feasible batch size; it does not make KV cache unlimited. Admission still needs a budget for model weights, runtime workspace, cache blocks, and safety headroom.
When pressure rises, decide deliberately whether to queue, reject, preempt, swap, or route elsewhere. Unplanned preemption storms can improve nominal admission while destroying tail latency.
Measure the scheduler, not only the GPU #
Track:
- queue time and time to first token by class and tenant;
- inter-token latency and end-to-end latency;
- prompt and generated tokens;
- running, waiting, and preempted requests;
- KV-cache utilization and prefix-cache hit rate;
- cancellation-to-reclamation delay;
- deadline misses and rejection reasons;
- tokens per second per accelerator.
GPU utilization at 100% is not success if interactive requests wait behind offline jobs. Throughput and latency must be evaluated together under the production prompt-length distribution.
Failure policy #
Reject requests exceeding hard context or output limits before allocation. Shed low-priority work when the queue cannot meet its deadline. Route only when the destination has compatible model, adapter, cache, and policy. Preserve request identity across routing so retries do not duplicate metered work or tool side effects.
Trade-offs #
Larger batches improve throughput until memory pressure, queueing, or decode interference harms tails. Chunked prefill can interleave long prompts with decode but adds scheduler complexity. Reserved capacity protects interactive latency while leaving hardware idle during quiet periods. Preemption raises utilization but spends recomputation.
The correct configuration is not the largest batch that fits. It is the scheduling policy that meets differentiated service promises at an acceptable cost.