Embedding Upgrades Are Data Migrations

Sep 5

By

ai systemsembeddingsvector searchdata engineering

Changing an embedding model changes the coordinate system in which similarity is computed. Even when old and new models emit vectors with the same dimension, distances across model versions have no promised meaning.

An embedding upgrade is therefore not a configuration flip. It is a data migration with an online serving path, a backfill, a quality gate, and a rollback plan.

Version the representation

Store enough identity to reproduce and interpret every vector:

CREATE TABLE document_embedding (
  document_id     bigint NOT NULL,
  model_id        text NOT NULL,
  model_revision  text NOT NULL,
  chunker_version text NOT NULL,
  content_hash    text NOT NULL,
  embedding       vector NOT NULL,
  created_at      timestamptz NOT NULL,
  PRIMARY KEY (document_id, model_id, model_revision, chunker_version)
);

The chunker belongs in the version because changing chunk boundaries changes what each vector represents. A content hash makes backfill idempotent and prevents recomputing unchanged material.

Do not overwrite the old vector in place. The existing index is your rollback path and your experimental control.

Separate migration phases

1. Offline evaluation

Build a query set from real traffic, support failures, and important edge cases. Label relevant results. Compare recall, ranking metrics, latency, and downstream answer quality by cohort—not only as one average.

2. Dual write

New or changed documents receive both representation versions. Observe generation failures and queue lag. The old read path remains authoritative.

3. Backfill

Scan by stable cursor, claim bounded batches, rate-limit model calls, and record progress durably. Backfill must yield to live indexing traffic.

eligible = source updated AND target version absent
claim batch -> embed -> validate dimension -> persist -> checkpoint

Retries should use the same (document, model, revision, chunker) key so they converge rather than duplicate work.

4. Shadow retrieval

Run the new retriever without serving its result. Log candidate overlap, latency, empty-result rate, and disagreements on a sampled fraction of traffic. Redact or hash sensitive query material.

5. Gradual cutover

Route a small cohort to the new index. Hold a rollback switch at the retrieval-router boundary. Increase exposure only when technical and product metrics remain inside agreed thresholds.

6. Retire deliberately

After the rollback window, remove old vectors and indexes through a retention job. Confirm that no consumer still requests the old version before reclaiming storage.

Failure boundaries

FailureSafe behavior
model API unavailablepreserve old reads; retry bounded backfill
dimension mismatchreject write before index insertion
partial backfillroute by complete version, not whichever row exists
new model quality regressionflip retrieval router back
source changes during backfillcompare content hash and requeue
index build failskeep dual-written source rows for rebuild

What to measure

Track coverage by version, oldest unembedded update, backfill throughput, embedding cost, index size, query latency, recall@k, reranker outcomes, grounded-answer rate, and user-task success. An upgrade that improves a benchmark while worsening filtered retrieval or multilingual queries is not complete.

Trade-offs

Dual storage temporarily increases cost. Shadow traffic consumes retrieval capacity. A fast in-place rewrite is cheaper in the short term but removes comparison and rollback. Versioning adds columns and routing logic, while turning an opaque model change into an observable, reversible operation.

Embedding systems are data systems. Give their migrations the same discipline as a database schema change.

Further reading

>