The migration was additive. A new column, a backfill, and a release that started reading it. Every forward step passed in staging.
Production exposed a behavior we had not modeled. The obvious response was to roll the application back. The previous version could still run, but it no longer understood the partially migrated state.
We had tested whether the new code worked with the old schema. We had not tested whether the old code worked after the new code wrote data.
The missing compatibility direction #
Safe rollout requires more than backward-compatible DDL. It requires a compatibility window across readers, writers, and stored representations.
old reader + old data
new reader + old data
new writer + mixed readers
old reader + data written by new writerThe final case was our gap.
The immediate choice #
We stopped expansion rather than reversing blindly. The new write path was disabled, affected rows were identified, and reads temporarily fell back to the old representation. Recovery became a data decision, not a deployment button.
The revised migration had four explicit phases:
- expand the schema;
- deploy code that can read both forms while writing the old form;
- begin dual writes and verify equivalence;
- switch reads, then contract only after the rollback window closes.
Each phase had a measurable completion condition and a named reversal path.
What I kept #
Rollback is not “deploy the previous commit.” It is a promise that the previous behavior can interpret the state created since it left production.
That promise expires. Once an irreversible backfill or destructive contraction begins, recovery may require a forward fix instead. The runbook should say when that boundary is crossed.
The migration did not fail because the SQL was unsafe. It failed because compatibility had been modeled in only one direction.
All Field Notes · Zero-downtime migrations are compatibility problems · Evolve APIs without flag days