Prompt Caching Is a Context-Layout Problem

Prompt caching is often presented as a switch that makes repeated LLM calls cheaper. In production it is a context-layout problem.

Caches benefit from shared, stable prefixes. Agent prompts often do the opposite: inject a timestamp, request ID, user state, retrieved documents, and tool output near the beginning. One changing token can move the reusable boundary and reduce the value of the cache.

The operating invariant is:

Put stable, broadly reusable instructions before volatile, narrowly scoped state—without weakening isolation or correctness.

Context has a memory hierarchy #

An agent request mixes several classes of information:

stable:   system policy, tool schemas, output contract
versioned: product rules, workflow definition, tenant policy
session:  compact conversation state, user preferences
volatile: current request, retrieval results, tool responses, timestamp

Arrange them in roughly that order. Stable prefixes maximize reuse; volatile suffixes preserve request specificity. This is similar to laying out data for locality: the model can accept the same semantic content in many orders, but the infrastructure cost differs.

Do not hide semantic changes to protect cache hits. A cached old policy is cheap and wrong.

Version the prefix deliberately #

Treat the reusable prefix as a deployable artifact:

const promptVersion = 'sales-agent/decision-policy/v7'

const input = [
  stableSystemPolicy,
  stableToolSchemas,
  `Policy-Version: ${promptVersion}`,
  tenantPolicy,
  sessionSummary,
  retrievedEvidence,
  userRequest,
]

The version should change when behaviorally relevant content changes. Record it with model, tool versions, retrieval snapshot, cache usage, latency, token counts, and final policy outcome. That makes a regression traceable.

Cache keys are isolation boundaries #

Provider features differ, but the architecture question is universal: which requests are allowed to share cached computation?

Never optimize cross-tenant reuse by placing private tenant data in a supposedly shared prefix. Separate public product instructions from tenant policy and user data. If an API exposes a prompt cache key, derive it from a bounded workload identity—not raw personal data—and understand the provider’s retention and routing behavior.

Extended caching can have data-retention implications. Privacy and compliance policy must be reviewed before latency wins are counted.

Measure outcome, not hit rate #

A high hit rate is not the goal. Track:

• cached versus uncached input tokens;
• time to first token and end-to-end latency;
• cost per completed business operation;
• correctness and policy-violation rate by prompt version;
• cache reuse by tenant and workload class;
• miss reasons: version churn, volatile prefixes, model routing, truncation.

Conversation truncation can also destroy prefix reuse by dropping messages from the beginning. Compact old state into a versioned summary before the window is full, and test whether the summary preserves decisions and unresolved obligations.

Failure boundaries #

Prompt caching must be an optimization, not durable memory. If the cache disappears, the request should remain correct—only slower or more expensive. If business correctness depends on remembered facts, store them in an authoritative system and reconstruct context.

Do not log complete prompts merely to debug caching. Capture structured fingerprints, versions, token ranges, and approved redacted excerpts.

Roll out layout changes like code #

Changing prompt order can affect both cache behavior and model behavior, even when the words are unchanged. Evaluate the new layout against a frozen task set, then canary it by prompt version. Compare task success, tool-call correctness, latency distribution, cached-token ratio, and cost per successful outcome.

Route fallback models through explicit compatibility tests. Tokenization, cache thresholds, tool-schema handling, and context limits vary. A layout optimized for one provider should degrade safely on another instead of silently dropping policy or evidence to fit a smaller window.

Conclusion #

Efficient AI systems do not merely choose a model; they arrange context. Stable prefixes, explicit versions, narrow sharing scopes, and observable invalidation convert prompt caching from a vendor feature into an engineering discipline.

The cache should accelerate a correct request. It must never become the place where policy, tenant state, or product truth secretly lives.

Further reading: OpenAI prompt caching, API reference, model context specifications, LLM KV-cache capacity planning, and AI inference routing.