Change Data Capture is usually introduced as integration plumbing: read database changes, publish events, keep search or analytics up to date.
That framing hides the harder obligation. Once downstream systems depend on the stream, CDC becomes a recovery contract between mutable database state and every derived copy. The architecture must answer not only “can we stream today?” but also:
Can we rebuild the truth after offsets are lost, schemas change, a slot falls behind, or a consumer has been wrong for three months?
Name the source of truth #
CDC emits a history of database changes. It does not automatically create a domain event model.
row change: orders.status PENDING -> PAID
domain fact: PaymentCapturedThese can coincide, but they have different contracts. A row-change stream reflects storage layout, transactions, and connector behavior. A domain event reflects business meaning and an intentionally versioned API.
Use CDC directly when consumers need a faithful projection of database state and can tolerate storage-shaped contracts. Use an outbox when the producer must choose stable business facts. Many systems combine them: write a domain event to an outbox table, then use CDC as the reliable transport.
Do not let downstream teams infer critical business meaning from undocumented column transitions.
A snapshot and a stream must meet cleanly #
A new consumer needs existing state plus future changes. The difficult boundary is the moment between the snapshot and live log.
An unsafe sequence is:
SELECT all rows
then begin reading changesWrites committed between those operations can disappear. Starting the stream first and then snapshotting can produce duplicates or updates before creates.
Production CDC connectors coordinate a consistent snapshot with a log position. Debezium’s PostgreSQL connector documents snapshot modes and uses PostgreSQL logical decoding to continue from a recorded position after the initial snapshot.
The consumer still needs idempotent application because restarts, retries, and re-snapshots can repeat records.
For a projection, prefer an upsert keyed by stable source identity:
INSERT INTO customer_projection(customer_id, source_version, payload)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id) DO UPDATE
SET source_version = EXCLUDED.source_version,
payload = EXCLUDED.payload
WHERE customer_projection.source_version < EXCLUDED.source_version;The version guard prevents an older replay from overwriting newer state.
Offsets are business continuity state #
The connector offset, PostgreSQL replication slot, publication configuration, and downstream consumer offsets together define recoverability. Treat them like production data.
For PostgreSQL logical decoding, a replication slot retains required WAL until the consumer advances. If consumption stops, retained WAL can fill disk. If the slot is dropped or allowed to fall behind available history, continuity is lost and the consumer may require a new snapshot.
Operate at least:
- current and confirmed log position;
- retained WAL bytes per slot;
- connector heartbeat freshness;
- transaction and event lag;
- offset commit failures;
- slot existence after failover;
- time and capacity required for re-snapshot.
“Connector is running” is not evidence that the stream is current or recoverable.
Define Recovery Point Objective (RPO) and Recovery Time Objective (RTO) for every derived system. A recommendation index may accept hours of lag; fraud decisions may not.
Ordering exists only within a declared scope #
Database commits provide an order in the log, but a distributed pipeline can repartition, parallelize, and retry records. A consumer should not assume global business order unless the architecture preserves it end to end.
Choose an ordering key—often aggregate or primary key—and include a monotonic source position or version. Then handle:
- duplicate version: ignore;
- older version: reject as stale;
- next version: apply;
- gap: defer, reconcile, or rebuild the key.
Cross-table transactions need special care. A consumer that observes line items before the order header may temporarily violate an invariant even though the database transaction was atomic. Use transaction metadata, an outbox event, or a projection design that tolerates convergence.
Total ordering across every tenant and entity is expensive and rarely the business requirement. Preserve only the order that protects an invariant.
Schema evolution is a distributed deployment #
A column rename is not one database migration once CDC exists. It is a contract change across connectors, schemas, brokers, consumers, replay archives, and projections.
Use expand-and-contract:
- add the new field;
- emit both representations where needed;
- deploy tolerant consumers;
- backfill and verify;
- stop old reads;
- remove the old field after the replay window.
Avoid reusing a field name with different meaning. Old records remain in logs and backups. A consumer rebuilding from history must interpret a schema version, not guess from the current database definition.
Database schema changes can also alter connector output unexpectedly: defaults, data types, replica identity, primary keys, and table inclusion all matter. Test the actual serialized records in a staging stream.
Deletes need an explicit semantic #
A source-row deletion may produce a delete event and, depending on the pipeline, a tombstone. Consumers must decide whether deletion means:
- remove the projection;
- retain a redacted audit record;
- mark inactive;
- trigger a business workflow;
- or ignore a storage-level cleanup.
For privacy deletion, removing a source row does not automatically delete data from compacted topics, warehouses, object storage, search indexes, caches, and backups. Track deletion as a governed workflow with evidence from every materialized copy.
Rebuild is a product feature #
Every important consumer should have a documented rebuild mode:
freeze or version target
-> establish snapshot/log boundary
-> load snapshot
-> replay changes
-> validate counts and invariants
-> atomically switch readers
-> retain rollback targetDo not rebuild directly into the live index if partial state would be visible. Build a new generation and switch an alias after validation.
Validation needs more than row counts:
- sums or balances for financial domains;
- per-tenant counts;
- min/max source positions;
- sampled content hashes;
- missing and duplicate keys;
- business invariants;
- lag to current source position.
Run rebuild drills before an incident. The first full replay will expose hidden assumptions about retention, throughput, schema compatibility, and external rate limits.
Prevent a consumer bug from becoming historical truth #
Derived stores should retain provenance: source table or event type, key, source position, schema version, and projection code version. Without this, you cannot identify which records were produced by a faulty consumer release.
Canary new projection code into a shadow target. Compare outputs and invariants before promotion. When logic changes, decide whether old records need reprocessing or only future records use the new rule.
CDC makes changes easy to distribute. It also makes mistakes easy to distribute.
CTO review questions #
- Is the stream a storage contract or a domain-event contract?
- How do snapshot and live changes meet without loss?
- Which offsets and slots must survive disaster recovery?
- What is the ordering scope and how are gaps detected?
- Can old records be interpreted after schema evolution?
- How does a delete propagate to every copy?
- How long does a full rebuild take, and when was it last tested?
- Can we identify data produced by a faulty consumer version?
CDC is valuable because it turns committed changes into reusable infrastructure. It becomes trustworthy only when snapshots, offsets, schemas, deletes, and rebuilds are designed as one recovery system.