LLM KV-Cache Admission Budget

Context and output tokens are capacity reservations. Admission must happen before expensive prefill work begins.

AI infrastructure · TypeScript

KV-Cache Admission Budget

Tokens are capacity reservations; reject work that cannot finish inside its cache and deadline budget.

type Request = { promptTokens: number; maxOutputTokens: number; tier: 'interactive' | 'batch' }

async function admit(request: Request) {
  const requested = request.promptTokens + request.maxOutputTokens
  const limit = policy[request.tier].maxSequenceTokens
  if (requested > limit) throw new RequestTooLargeError(limit)

  const lease = await tokenBudget.tryReserve(request.tier, requested)
  if (!lease) throw new OverloadedError({ retryable: request.tier === 'batch' })
  try {
    const result = await inference.generate(request)
    await lease.settle(result.promptTokens + result.outputTokens)
    return result
  } finally { await lease.release() }
}

Invariant: No request begins prefill unless its maximum token reservation fits both its service-class limit and currently available capacity.

Use when: An inference gateway must protect interactive latency from oversized or concurrent LLM requests.

Why this boundary matters

Model weights are fixed, but KV state grows with active tokens. Unbounded contexts can evict many short, latency-sensitive requests.

Failure policy

Boundary	Action
Request exceeds per-request tokens	Reject before tokenization or prefill cost grows
Pool has a reservation	Admit and settle actual use on completion
Interactive pool full	Queue within the caller deadline or reject
Batch work consumes reserve	Preempt or defer batch work according to policy
Reservation expires	Release capacity idempotently

Trade-offs

Conservative reservations protect latency but leave some accelerator capacity idle when generations end early. Aggressive overcommit improves average utilization while increasing preemption and tail latency.

Decision rule: Reserve the declared worst-case token demand for latency-sensitive traffic and optimize overcommit only with measured workload distributions.

Further reference

Browse all engineering snippets · Read the architecture guide