AI Systems Engineering

AI systems engineering is the work required to turn probabilistic model behaviour into a dependable product. The model matters, but it is only one component. Production quality emerges from the contracts around it: evidence, data freshness, evaluation, serving, tool authority, observability, cost controls, and recovery.

This page is the map I use when designing those systems. Each section links to deeper implementation notes and failure analyses.

The short definition #

AI systems engineering is the design and operation of the complete production system around a model: data, retrieval, evaluation, serving, tools, security, observability, cost, and failure recovery.

A useful AI feature is not simply a prompt that worked in a demonstration. It is a versioned system that can explain which model, instructions, evidence, tools, policies, and fallbacks produced an outcome—and can stop safely when those dependencies become unreliable.

Start with a bounded outcome #

“Build an AI assistant” is not an engineering boundary. “Draft a support reply from authorised customer records without sending it” is bounded enough to test. The outcome identifies the user, permitted evidence, acceptable error, prohibited actions, latency budget, and accountable owner.

Write the production contract before choosing a model:

• what decision or artifact the system produces;
• what evidence it may use and how fresh that evidence must be;
• which mistakes are reversible;
• which actions require deterministic validation or human approval;
• what happens when the model, retrieval layer, or dependency is unavailable.

This keeps model capability from silently becoming product authority. The deeper control-plane design is in A Bigger Prompt Will Not Make Your Agent Safe.

Treat evidence as a data system #

Retrieval-augmented generation is not “put documents into a vector database.” It is a data pipeline with ingestion, parsing, chunking, metadata, indexing, ranking, authorization, freshness, and deletion semantics.

Evaluate retrieval separately from generation. If the necessary evidence never reaches the context, a better prompt cannot recover it reliably. Measure whether the right source appeared, whether the right passage ranked highly enough, and whether filters removed relevant evidence. Then measure whether the answer stayed faithful to what was retrieved.

Important operational questions include:

1. Which source is authoritative when two documents disagree?
2. How quickly do corrections and revocations reach the index?
3. Can one tenant retrieve another tenant’s material?
4. Which query classes suffer after an embedding or chunking change?
5. Can an answer cite the exact evidence used?

Read RAG Quality Starts With Retrieval Evidence, Filtered Vector Search Is a Recall Budget, and Embedding Upgrades Are Data Migrations for the implementation consequences.

Make evaluation release engineering #

An evaluation score is useful only when it controls a production decision. Keep a versioned evaluation set for critical behaviours and known failures. Record the model, prompt, tool schemas, retrieval index, policy version, and evaluator with every result. Otherwise, a changed score cannot be reproduced or explained.

A production gate should combine several kinds of evidence:

Do not reduce these signals to one attractive average. A release can improve general quality while regressing a rare, high-consequence workflow. AI Evals Are Release Engineering describes the gating and rollback model.

Put authority outside the model #

Structured output proves shape, not truth or permission. A valid refund object may still refer to the wrong customer, use stale evidence, or exceed the caller’s authority. Treat model output as a proposal. Deterministic application code must establish identity, authorization, current state, business invariants, idempotency, and approval before any material side effect.

For agent systems, make budgets explicit: allowed tools, maximum steps, wall-clock deadline, spend ceiling, data scope, and termination conditions. Log proposed and executed actions separately. Require human approval where an error is costly or difficult to reverse.

The relevant boundary is developed in Structured AI Output Is Not an Authorized Command and AI Agent Memory Is Governed State.

Engineer the serving path #

LLM serving is a capacity and scheduling problem. Prompt length, output length, batching, KV-cache occupancy, model size, accelerator memory, and tenant fairness interact. Requests with the same token count may have different latency because prefill and decode stress different resources.

Measure admission decisions, time to first token, inter-token latency, end-to-end latency, queue time, cache occupancy, tokens per second, rejection rate, and cost per successful task. Autoscaling cannot rescue requests whose deadlines expired while capacity was starting. Admission control, bounded queues, graceful degradation, and smaller fallback models are part of the serving contract.

See LLM Serving Is KV-Cache Capacity Planning, Continuous Batching Is a Scheduling Policy, and AI Inference Routing Is Capacity Control.

Observe behaviour, not only infrastructure #

CPU, memory, and HTTP status codes cannot tell you whether an AI system is useful. Trace the semantic path: request class, model and prompt version, retrieved evidence identifiers, tool proposals, policy decisions, fallbacks, user correction, and final outcome. Avoid recording sensitive prompts by default; retain structured evidence and carefully governed samples.

Useful service-level indicators include successful-task latency, grounded-answer rate, tool success by action, policy-denial rate, fallback quality, user correction rate, and cost per accepted outcome. Segment them by version. Aggregate metrics can hide a regression affecting one tenant, language, workflow, or document type.

Design degradation before failure #

Define what the product does when retrieval is stale, a model provider is slow, a tool is unavailable, or the evaluation gate detects regression. Safe responses may include using a smaller model, returning evidence without synthesis, producing a draft without execution, routing to a human, or declining the operation.

Retries require particular care. A slow dependency can cause timeouts; timeouts cause retries; retries increase queue pressure; pressure increases latency; latency creates more timeouts. The correct boundary is a shared deadline and retry budget, not an arbitrary number of attempts. Model this feedback loop in CTO Workbench V2 and study it in Deadline Budgets and Retry Amplification.

A practical architecture review #

Before production, ask:

• Is the user outcome bounded and measurable?
• Can every answer or action be tied to permitted, current evidence?
• Are model output and execution separated by deterministic checks?
• Do evaluations control releases and preserve version provenance?
• Are latency, capacity, and cost measured per successful task?
• Can the system degrade without inventing certainty or widening authority?
• Can an operator stop, replay, investigate, and roll back the complete system?

If these questions have precise answers, the architecture is becoming operable. If they are answered with prompt wording alone, the system is still a prototype.

Frequently asked questions #

What is AI systems engineering? #

AI systems engineering designs and operates the full production system around a model: data, retrieval, evaluation, serving, tools, security, observability, cost, and recovery.

How is it different from model development? #

Model development improves model capability. Systems engineering makes that capability dependable inside a product through evidence contracts, evaluation gates, authority boundaries, operational controls, and safe degradation.

What should a production AI architecture measure? #

Measure task quality, retrieval quality, grounding, latency, cost, tool outcomes, policy violations, fallback rates, and user-impacting failures. Segment every signal by model, prompt, data, and policy version.