Strongly Consistent Object Storage Still Needs Concurrency Control

Aug 31

By

cloud infrastructureobject storageconcurrency controlsystem design

Amazon S3 provides strong read-after-write consistency for object PUT and DELETE operations. After a successful write, a later read sees the change. That guarantee removes a large class of stale-read workarounds. It does not prevent two writers from overwriting one another.

The invariant is:

A mutable object changes only if the writer’s precondition still matches the version it read; immutable content is never replaced under the same identity.

Strong consistency is not compare-and-set

Two workers can read ETag v1, independently create updates, and both perform unconditional PUTs. Each write is strongly consistent. The later write still destroys the earlier result.

worker A: GET v1 ── compute A ── PUT A
worker B: GET v1 ───── compute B ───── PUT B
final value: B; A is lost

Use a conditional request. If-Match permits the write only when the current ETag equals the version the writer observed. If-None-Match: * creates only when no current object exists.

await s3.send(new PutObjectCommand({
  Bucket: bucket,
  Key: key,
  Body: payload,
  IfMatch: previousETag,
}))

A precondition failure is a concurrency conflict. Reload, recompute the decision, and retry inside a budget. Do not repeat the same stale body automatically.

ETag is an entity version, not universally a content hash

An ETag changes with the object and is useful for conditional requests. Do not assume it is always the MD5 of payload bytes. Multipart uploads and encryption modes can produce different semantics. Use explicit checksum fields when integrity verification requires a known algorithm.

Separating concerns avoids fragile designs:

  • ETag for object version preconditions;
  • checksum for transfer/content integrity;
  • business version for domain ordering;
  • object version ID for recovery when bucket versioning is enabled.

Immutable keys simplify the system

For artifacts, exports, images, model outputs, and deployment bundles, prefer content-addressed or generation-addressed keys:

reports/tenant-42/2026-08-31/run-8f31/result.parquet
releases/sha256-6c2.../bundle.tar.zst

Write the immutable object first, verify it, then update a small pointer or metadata record conditionally. Readers either see the old pointer or the new pointer—never half an upload.

PUT immutable generation
HEAD + verify checksum and metadata
conditional PUT manifest If-Match: prior-manifest-etag

This is the object-storage form of publish-then-switch.

Listing is not a workflow database

Strong listing consistency means a successful write is reflected in subsequent listings. A prefix listing still does not encode job ownership, dependency completion, authorization, retries, or a multi-object transaction.

If a workflow needs “all 20 parts exist and belong to generation 7,” publish a manifest naming those exact objects and checksums. Consumers read the manifest as the commit record. Or keep transactional workflow state in a database and treat objects as immutable payloads.

Ambiguous writes still exist

The client can lose its connection after S3 accepts a write but before the response arrives. A retry to an immutable key with the same checksum is naturally safe. A retry to a mutable key needs a precondition and reconciliation through HEAD or version history.

Do not generate a new object key for every transport retry unless duplicates are intentionally tolerated; otherwise ambiguity becomes orphaned storage and inconsistent metadata.

Deletion and retention are policy boundaries

Bucket versioning can preserve overwritten and deleted versions, improving recovery while increasing storage and lifecycle complexity. Object Lock can enforce retention, but governance and compliance modes change who can remove data. Lifecycle rules are asynchronous policy execution, not immediate proof of deletion.

Track incomplete multipart uploads, noncurrent versions, delete markers, failed conditional writes, orphan objects not referenced by manifests, checksum failures, and lifecycle backlog.

The CTO decision

Use object storage for immutable, large payloads and publish their availability through a small transactional or conditional metadata boundary. Use conditional writes for mutable pointers. Specify checksum, version, retention, and ambiguity behavior separately.

Strong consistency tells you which write is visible. Concurrency control decides whether that write was allowed to replace what came before.

References

>