A worker acquires a lease, pauses, and resumes after the lease expires. A second worker has already acquired ownership. The lock service may be behaving perfectly while both processes believe they are allowed to write.
Expiry revokes permission in the coordination service. It does not stop a CPU, recall a network packet, or cancel a request already buffered by a proxy.
The resource receiving a write must reject obsolete ownership; the worker’s belief about its lease is insufficient.
This is the distinction between controlling who should work and controlling whose effects are accepted.
Walk through the race #
worker A acquires epoch 41
A pauses before sending its write
lease expires
worker B acquires epoch 42
B writes with epoch 42
A resumes and writes with epoch 41If storage ignores ownership, A can overwrite B. Extending the lease reduces the frequency of the race but does not prove safety under long pauses or delayed messages.
Hazelcast’s FencedLock documentation illustrates the stale-client problem and the use of increasing fencing tokens. The essential enforcement happens at the protected resource, not in the lock acquisition response alone.
Define the storage contract #
A fencing token is an ordered ownership epoch issued by a coordinator with the required consistency guarantees. The resource remembers the greatest accepted epoch and rejects smaller ones.
Do not substitute a random lock-owner UUID. Random identity can support safe lock release, but it does not establish whether one owner is newer than another. Likewise, timestamps from unsynchronized clients are not a reliable epoch authority.
For a row that accepts one result per ownership epoch, an illustrative conditional update is:
UPDATE report_result
SET payload = $3::jsonb,
last_fence = $2::bigint
WHERE report_id = $1
AND last_fence < $2::bigint
RETURNING report_id;The application must treat zero returned rows as rejection, not success. This assumes last_fence is non-null, all writes use this path, and the supplied fence comes from a trusted coordinator. Access control must prevent arbitrary clients from submitting enormous epochs.
The comparison and mutation must be atomic. A separate SELECT followed by an unconditional UPDATE recreates the race.
Fencing has a precise boundary #
The example rejects epoch 41 after storage accepts epoch 42. It does not automatically reject A merely because B acquired a lease somewhere else: storage must learn the newer epoch.
If the requirement is “no old writes after new ownership is acquired,” ownership activation must include advancing the fence at the resource before the new owner starts work, or ownership must be checked atomically with the write. State that stronger contract explicitly.
Strictly increasing checks also mean repeated writes using the same epoch are rejected. That suits one-result-per-epoch operations. Multi-write owners need a separate operation sequence, transaction model, or idempotency ledger. Do not change the comparison to allow equal epochs without defining replay behavior.
A lock service cannot fence every API #
A payment provider or email gateway may not understand your token. Passing an extra header achieves nothing unless the receiver enforces it.
For external effects, use the provider’s supported idempotency key where available, record operation identity durably, and reconcile ambiguous outcomes. If the provider offers neither fencing nor deduplication, acknowledge that duplicate effects remain possible and design compensation or human review.
That limitation is architectural. It cannot be repaired by selecting a more fashionable lock algorithm.
Test the stale process, not just lock expiry #
A useful failure test pauses A after ownership acquisition, lets B acquire and activate a newer epoch, commits B’s result, then resumes A. Storage must reject A and preserve B’s value.
Also test delayed delivery, coordinator failover, duplicate tokens, token reuse after backup restoration, and a write path that accidentally bypasses the guarded update. Monitor rejected epochs, lease renewal failures, ownership transitions, and operation reconciliation.
A restore that resets the epoch allocator while storage retains high epochs can stop all progress. A restore that resets storage’s fence while old clients remain alive can re-admit obsolete writes. Recovery procedures must preserve the ordering contract across both systems.
Trade-offs and ownership #
Fencing adds state and coupling to the resource API. For occasional duplicate cache rebuilds, that may be unnecessary. For mutable authoritative data, avoiding this coupling often means accepting a correctness hole.
The Redis locking guidance is useful for understanding lease assumptions and safe release. Treat those assumptions as something to audit, not a replacement for downstream enforcement.
The CTO decision #
Classify each lock as efficiency-only or correctness-critical. For correctness-critical work, locate the final write boundary, prove obsolete owners are rejected there, and specify what happens when the receiver cannot enforce that proof.