AI Inference Routing Is Capacity Control

Aug 29

By

ai infrastructureai agentsreliabilitycloud economics

Many AI systems begin with one model behind one API call. Production introduces conflicting objectives: low latency, bounded spend, regional availability, privacy, tool support, context size, and quality that varies by task.

A router that merely sends requests to the cheapest available model will eventually convert a capacity incident into a correctness incident.

The governing invariant is:

A request may be routed only to a model configuration proven acceptable for that task and risk class.

Availability does not mean “some model returned text.” It means the system produced an outcome inside the product’s quality and safety boundary.

Classify the task before selecting the model

Route on a small, deterministic task contract:

{
  "task": "support_reply_draft",
  "risk": "reviewed",
  "latency_slo_ms": 2500,
  "max_cost_usd": 0.03,
  "required": ["json_schema"],
  "data_region": "in",
  "evaluation_set": "support-reply-v7"
}

Do not let the model self-declare its risk or budget. Those are product decisions derived from the user action, data class, and consequence of error.

Build a capability registry containing provider, model version, supported features, context limit, regional policy, observed latency, price, and evaluation status. Configuration changes should be versioned and auditable.

Use a constrained candidate set

Routing is two steps:

  1. Eligibility: remove models that fail capability, privacy, risk, or evaluation requirements.
  2. Optimization: among eligible candidates, choose for latency, cost, and capacity.
candidates
→ policy filter
→ evaluation threshold
→ capacity admission
→ cost/latency choice
→ execution

This order matters. Cost must never make an unapproved model eligible.

Evaluation gates should be task-specific. A model that performs well at extraction may be unacceptable for SQL generation. Store the dataset version, scoring method, sample size, and approval time. Re-evaluate when prompts, tools, model versions, or decoding settings change.

Reserve capacity before execution

Tokens, concurrency, and spend are finite shared resources. Estimate an upper bound, then atomically reserve it before calling the provider.

tenant budget + task budget + provider concurrency + deadline
                           ↓
                     admit or reject

Settle actual usage afterward and release the difference. Without reservation, one burst can admit thousands of individually valid requests that collectively exceed budget or provider concurrency.

Apply backpressure at the product boundary. Queue only work whose value survives delay. Interactive requests should degrade or reject within a deadline rather than wait behind an unbounded batch.

Design semantic fallback

Fallback is safe only when the substitute preserves required capability and quality.

Primary failurePossible response
provider timeouttry one eligible alternate inside remaining deadline
rate limithonor retry signal or route to reserved alternate capacity
schema failureone constrained repair attempt, then deterministic rejection
policy filter failuredo not retry with a weaker policy
no evaluated model availableuse non-AI workflow, cached result, review queue, or fail
ambiguous tool effectreconcile by operation ID before any retry

Do not silently fall from a high-capability model to a cheap model for a high-risk action. “No answer” can be safer than a plausible wrong one.

Observe decisions, not private prompts

Record stable routing evidence:

  • task and risk class;
  • router policy version;
  • selected provider and model version;
  • candidate rejection reasons;
  • estimated and actual tokens and cost;
  • queue, provider, and total latency;
  • fallback path;
  • schema, policy, and evaluation outcome.

Use OpenTelemetry’s generative-AI semantic conventions where they fit, but avoid capturing sensitive prompts by default. High-cardinality request detail belongs in controlled traces or audit storage, not metric labels.

Roll out routing changes like code

A router policy can alter quality, cost, and customer behavior instantly. Treat it as a production release:

  1. replay against a frozen evaluation set;
  2. shadow on real traffic where permitted;
  3. canary by task and tenant cohort;
  4. compare quality, latency, fallback, and cost;
  5. promote gradually with an automatic rollback condition.

Keep the last known-good policy locally available. The data path should not require a healthy configuration control plane for every inference request.

Conclusion

An inference router is not a model switch. It is an admission controller and policy enforcement point for correctness, capacity, risk, and economics.

Filter by capability and evidence first. Reserve scarce resources. Fall back semantically. Observe routing decisions. Release changes gradually. That is how multi-model infrastructure creates resilience instead of hiding nondeterminism behind another layer.

References

>