S3 Multipart Uploads Need Garbage Collection

A multipart upload is not an object under construction. It is a durable upload session containing separately stored parts until the client explicitly completes or aborts it.

If the client disappears after sending parts, those bytes remain chargeable. Ordinary object expiration rules do not remove incomplete multipart uploads. The storage system needs an explicit garbage-collection policy.

Model the upload as a state machine #

initiated -> uploading -> completing -> complete
                 |             |
                 +-----------> aborting -> aborted

Persist the provider upload ID, destination key, expected content identity, initiator, creation time, and application status. The upload ID is authority to add parts to one session; it should not be reconstructed from an object key.

A minimal control table might look like:

CREATE TABLE object_upload (
  operation_id uuid PRIMARY KEY,
  bucket text NOT NULL,
  object_key text NOT NULL,
  provider_upload_id text UNIQUE NOT NULL,
  expected_size bigint,
  content_sha256 text,
  status text NOT NULL,
  created_at timestamptz NOT NULL,
  completed_at timestamptz
);

operation_id is the business idempotency key. Retrying initiation with the same operation should return the same active session or the already completed outcome, rather than create another set of orphaned parts.

Completion is the commit boundary #

Uploading the final part does not publish the object. The client completes the upload by supplying the part numbers and returned identifiers in order. A timeout during completion creates an ambiguous outcome: the server may have committed even though the response did not arrive.

Resolve ambiguity by reading authoritative object or upload state. Do not blindly initiate a replacement upload. Completion retries must use the stored session and exact part manifest.

Checksums prove transfer integrity; they do not prove that the object corresponds to the intended business operation. Bind bucket, key, expected content hash, tenant, and operation identity at the application boundary.

Add a lifecycle safety net #

AWS provides AbortIncompleteMultipartUpload as a lifecycle action. A rule can make sessions older than a chosen number of days eligible for abort and deletion of their stored parts.

<Rule>
  <ID>abort-incomplete-uploads</ID>
  <Status>Enabled</Status>
  <Filter><Prefix>uploads/</Prefix></Filter>
  <AbortIncompleteMultipartUpload>
    <DaysAfterInitiation>7</DaysAfterInitiation>
  </AbortIncompleteMultipartUpload>
</Rule>

The lifecycle rule is a backstop, not the normal control path. Applications should abort known cancellations promptly. The lifecycle window must exceed legitimate slow uploads, paused mobile transfers, and recovery time for downstream incidents.

Operational contract #

Monitor incomplete multipart bytes, upload count, oldest active session, completion latency, abort failures, and sessions per tenant. Alert on rate of growth, not only a large absolute total.

Security boundaries #

Presigned part URLs should be short-lived and scoped to one upload. Enforce allowed object prefixes and content limits before initiation. Do not let a client choose arbitrary buckets or overwrite another tenant’s keys. Treat object metadata as untrusted input when it later becomes an HTTP header or download filename.

Trade-offs #

Larger parts reduce request count but increase the cost of retrying one failed part. More parallelism shortens ideal upload time while increasing client memory, network bursts, and server request rate. Longer lifecycle windows protect slow uploads but retain garbage longer. Shorter windows save cost but can terminate valid resumable work.

Multipart upload is a distributed transaction without an automatic rollback. Give it durable state, a reconciliation loop, and garbage collection.

Further reading #

• Amazon S3 multipart upload overview
• Abort incomplete uploads with lifecycle rules
• Strongly consistent object storage still needs concurrency control