An AI feature is not a prompt plus a model endpoint. It is a probabilistic release whose behavior depends on model version, instructions, tools, retrieval, data, sampling, policy, and the surrounding application.
If any of those can change without a regression gate, the organization does not have an AI release process. It has production sampling.
Define the system under test #
Pin a release candidate as a complete configuration:
release: support-agent-2026-08-25.3
model: pinned-provider-version
prompt_sha: 4e91c2a
tool_schema_sha: b8031f7
retrieval_index: help-center-2026-08-24
policy_version: support-v7
temperature: 0.1“We tested the model” is incomplete when production uses different tools, documents, or policies. Store this manifest with every evaluation result and production trace. Without provenance, a better or worse score cannot be explained or reproduced.
Evaluate business behavior #
Generic benchmarks help compare broad model capabilities. They do not prove that your refund agent applies your policy, cites the right account record, or refuses to expose another tenant’s data.
Build an evaluation set from the real task distribution:
- normal high-volume requests;
- high-value business paths;
- ambiguous and underspecified inputs;
- known production incidents;
- adversarial security cases;
- long-tail languages and formats;
- tool and dependency failures;
- explicit “must refuse” cases.
Each case needs a reason for existing and an owner. A thousand uncurated examples can provide less signal than one hundred cases mapped to failure modes.
Use layered graders #
No single score is trustworthy enough for release control.
Deterministic checks validate properties with exact answers: JSON schema, required citations, allowed tool names, arithmetic, SQL syntax, tenant IDs, or absence of secrets.
def grade_tool_call(output, account_id):
call = output["tool_calls"][0]
return (
call["name"] in {"lookup_order", "create_refund_request"}
and call["arguments"]["account_id"] == account_id
)Reference checks compare extracted facts, selected actions, or expected outcomes.
Model graders help assess open-ended qualities such as relevance or completeness. Calibrate them against human judgments, randomize candidate order when comparing outputs, and retain disagreement samples.
Human review remains necessary for policy, brand, and novel high-impact failures.
The release gate should expose a scorecard, not collapse every concern into one average:
| Dimension | Gate |
|---|---|
| tenant isolation | 100% |
| unsafe action prevention | 100% |
| tool selection | ≥ 98% |
| answer correctness | ≥ 95% |
| p95 latency | ≤ 4 s |
| cost per resolved case | within budget |
A gain in writing style must never hide a regression in tenant isolation.
Test trajectories, not only final text #
Agent systems can reach a plausible answer through an unsafe path. Capture the trajectory:
user input
→ retrieved documents
→ model decision
→ tool request
→ tool response
→ final answerGrade whether retrieval crossed tenant boundaries, whether the chosen tool was allowed, whether arguments were validated, whether retries duplicated a side effect, and whether the final statement is supported by observed tool output.
Tool authorization belongs in deterministic application code. An evaluation verifies that the agent behaves correctly; it must not be the only barrier preventing a model from deleting data.
Prevent evaluation leakage #
Keep at least three datasets:
- Development set: visible to people tuning the system.
- Release set: stable regression suite with controlled access.
- Holdout set: refreshed and hidden from routine optimization.
If every failed release case becomes a prompt example, the suite eventually measures memorization. Rotate samples from production distributions and create transformed variants that preserve the failure mechanism.
Version datasets. Never overwrite a case after changing its expected result; record the policy version that changed the expectation.
Join offline and online evidence #
Offline evals are repeatable and safe, but production traffic changes. Online signals reveal new inputs and real dependency behavior.
Instrument model, retrieval, and tool spans using a consistent trace ID. OpenTelemetry’s generative-AI semantic conventions provide a common vocabulary, though the conventions continue to evolve; isolate vendor-specific attributes behind your telemetry boundary.
Collect:
- task success or escalation;
- corrected answers and user feedback;
- tool errors and rejected arguments;
- groundedness and citation failures;
- token usage, latency, and cost;
- policy refusals by category;
- release configuration and trace provenance.
Do not log sensitive prompts by default. Apply redaction, access control, retention, and sampling before payload capture.
Convert production failures into reviewed eval cases. That closes the loop:
incident → minimized case → regression test
→ candidate fix → canary → production evidenceRoll out like infrastructure #
Use shadow traffic where privacy and cost allow, then a bounded canary. Compare the candidate against the current release on the same task distribution. Set automatic rollback triggers for safety violations, tool-error spikes, latency, and unit cost.
Keep the previous complete release manifest deployable. Rolling back only the prompt while leaving a new tool schema or retrieval index active is not a rollback.
For high-impact actions, separate recommendation from execution. Require deterministic policy checks, idempotency keys, audit trails, and human approval above a risk threshold.
CTO review #
- Is the complete AI configuration versioned and reproducible?
- Which business failures does each eval set represent?
- Which dimensions are hard gates rather than averages?
- Are agent trajectories and tool effects evaluated?
- How are graders calibrated against human judgment?
- How do production incidents become permanent regression cases?
- Can the complete previous release be restored quickly?
- What action remains impossible regardless of model output?
Evals are not a demo score. They are the test, provenance, rollout, and rollback system for probabilistic software.