Teams often ask whether a model is “good enough for production.” That question is too broad to answer and too vague to govern a release.
A model is not deployed in isolation. It is placed inside a use case, given data, connected to tools, shown to users, and allowed to influence decisions. Production readiness belongs to that entire system.
NIST’s AI Risk Management Framework organises work around four continuing functions: govern, map, measure, and manage. The useful engineering translation is a set of release gates tied to a named use case and an explicit risk tolerance.
Gate 1: define the decision boundary #
Write down what the AI component does and, equally important, what it cannot do.
input: customer email and permitted account context
output: proposed reply and cited account facts
authority: draft only
human decision: approve, edit, or discard
prohibited: send message, change price, alter account state“Sales assistant” is not a boundary. “Draft a response using these authorised records without sending it” is testable.
Classify consequences. A bad internal summary is different from an incorrect medical recommendation or an agent that can execute a refund. As consequence rises, require stronger evidence, narrower authority, deterministic validation, and human approval.
Gate 2: build an evaluation contract #
An evaluation set should represent the distribution the product expects, including edge cases and adversarial inputs. Separate dimensions instead of compressing them into one score:
- task correctness;
- factual support and citation validity;
- policy compliance;
- refusal behaviour;
- tool-selection correctness;
- sensitive-data handling;
- latency and cost;
- consistency across relevant languages or customer segments.
Define blocking thresholds before running the release candidate. Averages can hide catastrophic classes, so add zero-tolerance or near-zero-tolerance gates for severe outcomes.
release_gate:
task_success: ">= 0.90"
unsupported_account_claim: "<= 0.005"
unauthorised_tool_attempt: "0"
p95_latency_ms: "<= 2500"
cost_per_completed_task_usd: "<= 0.04"The example values are illustrative; production thresholds must follow the use case and evidence. Version the dataset, rubric, evaluator, prompt, model, retrieval configuration, and tool schema so results are reproducible.
Gate 3: constrain authority in code #
Prompts are instructions to a probabilistic component, not access control. Enforce permissions outside the model.
Every tool call should pass through a policy layer that validates identity, tenant, requested action, resource scope, and budget. Use narrow tools such as draft_follow_up(account_id) instead of generic database or HTTP access.
For consequential writes:
- produce a typed proposal;
- validate it deterministically;
- show material effects to an authorised human;
- record approval with the proposal hash;
- execute idempotently;
- preserve an audit event.
Authority should be revocable without redeploying the model. Feature flags or policy configuration can disable one tool, tenant, or workflow while preserving safe read-only capability.
Gate 4: design degradation before launch #
Providers throttle, models change, retrieval becomes stale, and latency spikes. Define the product behaviour for each dependency failure:
| Failure | Safe behaviour |
|---|---|
| model timeout | deterministic template or manual workflow |
| retrieval unavailable | do not make account-specific claims |
| evaluator/policy unavailable | block consequential action |
| primary model unavailable | approved fallback only if it passes the same gates |
| cost budget exhausted | queue, reduce optional context, or disable non-critical generation |
A fallback model is not safe because it returns text. It must satisfy the relevant evaluation contract and tool semantics.
Gate 5: make production observable #
Capture enough evidence to reconstruct a decision without indiscriminately logging sensitive prompts. Useful fields include:
- use-case and policy version;
- model and configuration;
- retrieval document identifiers and freshness;
- requested and permitted tools;
- validation and approval outcomes;
- latency, token usage, and cost;
- user correction, rejection, or escalation;
- final business outcome where measurable.
Apply retention, redaction, tenant isolation, and access controls to telemetry. An audit trail that creates a new data leak is not a control.
Monitor both system and product drift. Latency and errors detect infrastructure regressions; correction rate, groundedness, task completion, and policy violations detect behavioural regressions. Sample production cases into a reviewed evaluation pipeline rather than treating the launch dataset as permanent truth.
Gate 6: assign release accountability #
The release record should name:
- product owner for the intended outcome;
- engineering owner for the system and rollback;
- risk owner for prohibited outcomes;
- dataset/evaluation owner;
- incident escalation path;
- expiry or review date for the approval.
High-risk exceptions need written reasoning, compensating controls, and an expiration. “Leadership accepted the risk” without a named risk, duration, or evidence is not governance.
Common mistakes #
One benchmark score. It hides severe failure classes and rarely matches the production workflow.
Prompt-only safety. The same model interpreting the request cannot be the only authority deciding whether that request is permitted.
Unversioned retrieval. A model release can appear unchanged while its knowledge source silently changes.
Invisible fallback. Switching provider or model without recording it breaks incident analysis and user expectations.
No kill path. If an unsafe tool cannot be disabled quickly and independently, the control plane is incomplete.
The production standard #
Do not ask whether the organisation trusts the model. Ask whether this version of the system, for this bounded use case, has passed named tests; whether its authority is constrained; whether failure becomes safe degradation; and whether the team can observe, stop, and investigate it.
That is a release decision a CTO can defend.