Valid JSON Is Not Permission to Execute

An AI system returns a perfectly shaped refund proposal. Every field has the right type. The customer identifier exists. The amount is positive. The proposal can still be unauthorized, based on an outdated balance, or justified by evidence that says the opposite.

Structured output reduces one category of integration failure. It does not turn model output into trusted business state.

Model output is a proposal until deterministic systems establish evidence, authority, freshness, and execution safety.

That boundary should exist even when the model has performed well on every demonstration.

Validate shape without overstating the guarantee #

A standard JSON Schema can define a bounded proposal:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["orderId", "amountMinor", "evidenceIds"],
  "properties": {
    "orderId": { "type": "string", "minLength": 1, "maxLength": 100 },
    "amountMinor": { "type": "integer", "minimum": 1 },
    "evidenceIds": {
      "type": "array",
      "minItems": 1,
      "maxItems": 10,
      "uniqueItems": true,
      "items": { "type": "string", "minLength": 1, "maxLength": 100 }
    }
  }
}

Required fields and additional-property restrictions have distinct jobs, as the JSON Schema object reference explains. Provider-supported schema subsets differ; this is a standard-schema illustration, not a promise that every generation API accepts it unchanged.

Validate again server-side. Bound total response bytes before parsing, preserve a refusal or incomplete-response path, and reject malformed output rather than silently manufacturing missing fields.

Keep annotations out of the security model #

A description saying “only authorized refunds” helps communicate intent. It does not enforce authorization. JSON Schema’s annotation documentation distinguishes descriptive metadata from validation.

Even format requires care: the type reference notes that format is an annotation by default. A plausible identifier or date is not proof that the referenced object exists or is usable.

Business constraints belong in code and authoritative data checks, not prose attached to schema properties.

Build a proposal-to-command boundary #

model proposal
 -> parse and shape validation
 -> authenticated principal and tenant binding
 -> evidence lookup and policy checks
 -> fresh state/version check
 -> durable command with idempotency
 -> execution and reconciliation

The server derives tenant and principal from the authenticated request. It does not let the model supply a privileged identity.

For each evidence ID, verify that it was available to this request, belongs to the correct tenant, and refers to a known source revision. These checks establish provenance, not that the evidence semantically supports the proposal. High-risk decisions may need explicit deterministic rules or human review in addition.

Close the time-of-check gap #

Suppose the proposal is valid against order version 17. Another process issues a partial refund before execution. Reusing the earlier eligibility decision can over-refund.

Bind approval to an operation identity, relevant inputs, policy version, and state version. At execution, recheck the mutable constraints transactionally or reserve the eligible amount through an owned workflow. If the version changes, reevaluate; do not ask the model to reinterpret a concurrency failure as permission.

Money should use a currency-aware representation and authoritative limits. An integer field alone does not establish currency scale, maximum refundable amount, or settlement eligibility.

Different failures need different actions #

Repeatedly prompting the model until it returns an allowed answer can turn a policy gate into a negotiation. Limit retries and preserve the original failure reason in the audit record.

Evaluate execution safety separately #

Track schema validity, evidence provenance, unsupported claims, policy violations, and actual business outcomes as separate measurements. A higher valid-JSON rate can coexist with worse decisions.

Build adversarial fixtures: cross-tenant identifiers, fabricated evidence IDs, previously refunded orders, conflicting sources, excessive amounts, duplicate command delivery, and a timeout after the provider accepted the refund. Use known expected decisions so failures are actionable.

Log proposal IDs, policy decisions, source revisions, execution versions, and provider references. Avoid placing raw customer documents or full prompts in unrestricted observability systems.

The CTO decision #

Keep the generative component where ambiguity is useful: extraction, explanation, and proposing options. Put irreversible authority behind narrow commands with deterministic eligibility checks, bounded approval scope, and durable operation identity.

The goal is not to make every response executable. It is to make inappropriate execution impossible at a boundary the model cannot talk its way around.

Further reading #

• AI agents need a control plane
• Production AI needs release gates
• Durable workflows still require idempotency