Evolve APIs Without Flag Days

Aug 25

By

api designdistributed systemsplatform engineeringsystem design

An API change is not deployed when the producer ships. It is deployed when every relevant producer, consumer, stored message, retry, cache, replay, and rollback path can coexist with it.

That coexistence period is the compatibility envelope. CTO-level platform design makes the envelope explicit rather than hoping teams deploy in the correct order.

Model versions that can meet

During a rolling migration, at least four interactions are possible:

old client → old server
old client → new server
new client → old server
new client → new server

Queues and stored payloads extend the matrix: a new consumer may read a message written months ago, while an old consumer may receive a freshly produced event after rollback.

Write the required compatibility matrix before changing a schema. If “new client → old server” is unsupported, state how deployment and rollback prevent that pairing. Compatibility is an operational promise, not a serialization feature.

Expand, migrate, contract

Safe evolution usually has three phases.

Expand: add the new representation while preserving the old contract.

{
  "customer_name": "A. Basak",
  "customer": { "display_name": "A. Basak" }
}

Migrate: update readers, backfill stored data, observe adoption, and move writers.

Contract: remove the legacy field only after evidence shows that nothing depends on it.

The hard part is not adding a field. It is proving deletion is safe. Assign every migration an owner, deadline, adoption metric, and removal condition. Otherwise compatibility layers accumulate into permanent complexity.

Make readers tolerant but semantics strict

Readers should generally tolerate additive fields. They should not silently reinterpret meaning.

Changing timeout_seconds from a request timeout to an end-to-end deadline keeps the JSON type but breaks behavior. Renaming amount to amount_cents without a dual-read period may be syntactically obvious and operationally unsafe.

For each field, document:

  • unit and allowed range;
  • absence versus explicit zero or empty;
  • defaulting owner;
  • whether unknown enum values are accepted;
  • security and tenancy scope;
  • lifetime and retention.

Schema compatibility cannot protect an undocumented semantic contract.

Protobuf safety is more than “it parses”

Protocol Buffers supports additive binary evolution: old code can ignore unknown fields and new code can read messages without newly added fields. But its official guidance includes constraints that matter in production:

  • never change an existing field number;
  • reserve deleted field numbers and names;
  • do not reuse tags;
  • treat type changes marked “compatible” as rollout-sensitive because values can be lossy;
  • be careful with oneof changes and unknown enum behavior;
  • distinguish binary wire behavior from ProtoJSON behavior.
message Account {
  reserved 3;
  reserved "legacy_tier";

  string id = 1;
  string display_name = 2;
  optional string billing_region = 4;
}

Add compatibility checks to CI, but do not confuse them with migration approval. A wire-safe change can still violate authorization or business semantics.

Version behavior, not every URL

Global /v2 endpoints are useful when the resource model genuinely changes. They are expensive when used for every additive field: clients split, documentation duplicates, and old versions never disappear.

Choose the smallest mechanism matching the change:

  • additive optional field: evolve in place;
  • behavior negotiated by capability: explicit header or field;
  • incompatible resource model: new version or resource;
  • one consumer’s special need: avoid contaminating the shared contract; consider a dedicated boundary.

HTTP semantics matter. A PUT remains idempotent; a retry should not create another resource because a version changed. Status codes, cache validators, and content negotiation are part of the contract described by HTTP, not decoration around the JSON.

Events make old contracts immortal

An event in a durable log can outlive every currently deployed service. Before changing an event schema, test:

new reader × oldest retained event
old reader × new event
replay × current side-effect policy
rollback × messages emitted during new release

Prefer facts with stable meaning. If OrderConfirmed later needs a tax jurisdiction, add it without redefining what “confirmed” meant historically. When meaning truly changes, create a new event type and make the transition explicit.

Keep golden payloads from real historical versions. Decode and re-encode them in CI. Generated examples are less likely to contain the odd omissions and enum values that break production replays.

Observe the migration

Instrument contract usage:

  • requests by client identity and version;
  • reads and writes of legacy fields;
  • unknown enum or field incidents;
  • fallback/default path usage;
  • decode failures by schema version;
  • remaining stored rows needing backfill;
  • oldest message schema in retention.

A removal gate might be:

legacy writes = 0 for 14 days
legacy reads = 0 for 14 days
backfill remaining = 0
rollback no longer requires old field
named owners approve consumer inventory

Logs sampled at one percent may miss the monthly billing job. Combine telemetry with ownership and code search.

Design rollback before rollout

A deployment is not safely reversible if the new writer emits data the old reader cannot interpret. Sequence changes so rollback remains possible:

  1. Deploy readers that understand old and new forms.
  2. Confirm fleet convergence.
  3. Enable new writes behind a controlled flag.
  4. Observe both paths.
  5. Stop old writes.
  6. Remove old reads only after the rollback window closes.

For destructive database changes, application rollback and schema rollback are separate decisions. Often the safest rollback is forward-fixing the application while leaving expanded storage intact.

CTO review

  1. Which old and new versions can interact during rollout and rollback?
  2. Are syntax and semantics both compatible?
  3. How are durable messages and stored payloads tested?
  4. What telemetry proves old behavior is unused?
  5. Who owns lagging consumers outside the team?
  6. When does the compatibility layer expire?
  7. Can the old binary read data produced by the new writer?

The mature alternative to a flag day is not permanent backward compatibility. It is controlled coexistence followed by evidence-based deletion.

References

>