A Webhook Endpoint Is a Durable Inbox

Sep 5

By

backend engineeringwebhooksreliabilitysecuritydistributed systems

A webhook is often implemented as a controller that parses JSON, runs business logic, and returns 200. That works until the sender retries, events arrive out of order, the handler times out after committing, or a framework mutates the request body before signature verification.

The robust model is different:

A webhook endpoint authenticates and durably accepts a message. Processing happens after acceptance.

This separates a short network acknowledgement from a potentially long, failure-prone business workflow.

Define the acknowledgement boundary

The receiver should perform only the work required to decide whether it can safely take responsibility:

  1. read the exact raw request bytes;
  2. identify the provider and endpoint configuration;
  3. verify the signature and timestamp according to that provider’s scheme;
  4. validate basic size and event-envelope limits;
  5. insert the delivery into a durable inbox with a unique key;
  6. return success only after the insert commits.
provider -> authenticate -> durable inbox -> 2xx
                                  |
                                  v
                         async processor -> domain effects

Returning success before durable storage risks losing the event. Returning success only after all business work completes couples provider retry behavior to internal dependency latency.

Stripe recommends returning a successful response before complex logic and handling events asynchronously. GitHub expects a 2xx promptly and likewise recommends queued processing.

Verify the bytes you received

Signature schemes authenticate the payload as delivered, not a parsed-and-reserialized equivalent. Stripe explicitly warns that whitespace changes, key reordering, encoding changes, or body parsing can break verification.

Capture raw bytes before generic JSON middleware. Associate secrets with a specific provider, environment, and endpoint. During secret rotation, accept the documented overlap set rather than trying every secret ever issued.

A valid signature proves possession of the configured secret for those bytes. It does not prove that processing the same valid delivery twice is safe.

Deduplicate at two levels

Providers may retry the same delivery. Store the provider’s delivery or event identifier behind a uniqueness constraint.

CREATE TABLE webhook_inbox (
  provider text NOT NULL,
  delivery_id text NOT NULL,
  event_type text NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  payload jsonb NOT NULL,
  status text NOT NULL DEFAULT 'pending',
  PRIMARY KEY (provider, delivery_id)
);

This makes repeated receipt idempotent. But Stripe notes that separate Event objects can sometimes represent the same underlying object and event type. Business effects therefore need their own idempotency boundary: invoice transition, subscription version, or (object_id, event_type, semantic_version).

Transport deduplication answers “have I accepted this message?” Domain idempotency answers “has this effect already happened?”

Assume unordered delivery

Stripe does not guarantee event order, and GitHub documents that deliveries may arrive in a different order from the underlying events. An event handler must not infer current truth solely from arrival order.

Prefer one of these policies:

  • retrieve current authoritative state from the provider;
  • apply a provider-supplied monotonic version when available;
  • use a domain state machine that rejects invalid backward transitions;
  • record ambiguous events for reconciliation.

Timestamps are weak ordering tokens when clocks, generation, and redelivery semantics differ. “Latest message received” is not necessarily “latest state.”

Separate replay defense from retry support

Stripe signs a timestamp and recommends a recency tolerance to limit captured-request replay. GitHub recommends tracking X-GitHub-Delivery, which remains the same for a redelivery.

These mechanisms differ. Implement the provider’s documented verification protocol rather than a generic webhook abstraction that guesses. Clock-based freshness requires synchronized server time. Delivery-ID retention must cover the period in which redelivery is possible plus an operational margin.

Do not reject a legitimate provider retry merely because it is a retry. Accept it idempotently, record it, and return the response that stops unnecessary future delivery.

Build reconciliation, not faith

Webhook delivery is a notification channel, not always the complete ledger. GitHub documents that failed deliveries are not automatically redelivered in all cases. Network errors, disabled endpoints, expired retry windows, and operator mistakes create gaps.

Run a reconciliation job against the provider’s authoritative API when supported. Track checkpoints, compare expected object state, and enqueue repairs through the same idempotent processor. Alert on oldest pending inbox item, failure rate, retry count, signature failures, and reconciliation drift.

Test the ambiguous outcomes

Exercise duplicate delivery, out-of-order delivery, invalid signatures, valid old signatures, rotated secrets, storage failure before acknowledgement, worker crash after side effect but before status update, and provider API outage during reconciliation.

The hardest test is the ambiguous write: the downstream action succeeds, but the worker loses the response. The retry must resolve the outcome through an idempotency key or read-before-retry—not hope.

The reliable webhook system is intentionally boring at the edge. Authenticate, commit, acknowledge, and let a replayable state machine do the dangerous work.

References

>