Webhook Retries Without Duplicate Side Effects in Node.js

A webhook endpoint can return an error even after it has changed data. It can also commit successfully while the acknowledgement disappears on the network. In both cases, the sender may deliver the same event again. If the handler treats delivery as proof that work has not happened, one provider event can create two credits, emails, or state transitions.

The operating rule is direct: authenticate the exact bytes received, insert a durable receipt under a provider-scoped unique key, and commit each local domain change in the same transaction as its processing state. A retry may still reach the application, but it cannot claim the same local operation twice.

This note builds that boundary in Node.js and PostgreSQL. It also marks where the boundary ends: a local transaction cannot make a call to an unrelated provider atomic. After reading, you should be able to reason about concurrent deliveries, lost acknowledgements, worker crashes, ordering, payload changes, retention, and replay without using “exactly once” as a shortcut.

Delivery can repeat even when processing should not

Webhook delivery and business processing are different events.

A delivery is one HTTP attempt from a sender. Processing is the application work caused by the event. A sender can make several delivery attempts while the receiver commits the business change once. Conversely, a receiver can process a single delivery twice because two application paths race.

In practical terms, at-least-once delivery means the sender prefers a possible duplicate over silent loss. It does not mean every provider follows the same schedule. Stripe documents automatic live-mode retries for up to three days with exponential backoff. GitHub documents no automatic redelivery for failed webhook deliveries, though an operator or scheduled recovery job can request a redelivery. Treat retry behavior as provider configuration, not as part of HTTP.

The response can also be lost. Suppose the receiver commits a receipt and sends 200 OK, but the TCP connection disappears before the sender observes the response. The sender only knows that delivery was uncertain. Redelivery is the safe choice. The receiver must recognize the stable event identifier.

This is why the useful target is not exactly-once delivery. The useful target is one committed local effect for one stable business operation, under stated retention and transaction assumptions.

Authenticate bytes before trusting identifiers

An event ID is useful only after the request is authenticated. Otherwise an attacker can send a guessed ID first and occupy the uniqueness key, causing the real event to be treated as a duplicate.

Most signed webhook schemes compute a message authentication code over the body and sometimes a timestamp. Verify that signature before JSON parsing, schema conversion, receipt insertion, or queue publication.

Why signature checks need the raw body

JSON objects do not preserve their original byte representation. Parsing and serializing may change whitespace, key order, escaping, or Unicode encoding. The sender signed the bytes it transmitted, not an equivalent JavaScript object. Stripe states this requirement explicitly, and GitHub warns that a proxy or framework must not modify the payload before validation.

The following verifier shows the mechanism. A real integration should use the provider SDK when it supplies a maintained verifier because header parsing and key rotation rules are provider-specific.

import { createHmac, timingSafeEqual } from "node:crypto";

function verifyHmac(rawBody: Buffer, header: string, secret: string): boolean {
  const prefix = "sha256=";
  if (!header.startsWith(prefix)) return false;

  const suppliedHex = header.slice(prefix.length);
  if (!/^[0-9a-f]{64}$/i.test(suppliedHex)) return false;

  const expected = createHmac("sha256", secret).update(rawBody).digest();
  const supplied = Buffer.from(suppliedHex, "hex");
  return supplied.length === expected.length && timingSafeEqual(supplied, expected);
}

Node.js documents that timingSafeEqual throws when byte lengths differ and that the surrounding code must also avoid timing leaks. The length check is not cosmetic. It prevents malformed input from turning authentication into a server error.

Timestamp tolerance is replay policy, not deduplication

A signed timestamp lets the receiver reject an old captured request. It assumes the receiver clock is reasonably synchronized and the provider defines how the timestamp enters the signed message. Stripe Node 20.0.0, inspected on 14 October 2026, uses a default tolerance of 300 seconds. That value is library behavior, not a standard for all webhooks.

A tolerance window limits how long a captured signed request remains acceptable. It does not replace the receipt table. Two valid deliveries can arrive inside the window, and a legitimate manual replay may occur much later. Monitor clock offset, follow the provider SDK’s signing format, and store the verified event ID for the full replay policy.

A durable inbox is the acknowledgement boundary

The HTTP route should do only the work needed to accept custody:

1. Read the body once as bytes and enforce a size limit.
2. Verify the signature and its timestamp policy.
3. Parse and validate the event envelope.
4. Insert the durable receipt.
5. Commit, then return a successful response.

Do not acknowledge before step 5. An early 2xx can lose the event if the process exits before durable storage. Do not perform a slow remote call before acknowledging either. Stripe recommends returning a successful response before complex work, and GitHub requires a 2xx response within 10 seconds.

The receipt is an inbox record: evidence that this application accepted one provider event. A minimal PostgreSQL 16 or 17 schema is:

CREATE TABLE webhook_receipts (
  provider text NOT NULL,
  event_id text NOT NULL,
  event_type text NOT NULL,
  payload jsonb NOT NULL,
  received_at timestamptz NOT NULL DEFAULT clock_timestamp(),
  processed_at timestamptz,
  PRIMARY KEY (provider, event_id)
);

Scope the key by provider or endpoint. Identifiers from unrelated senders can collide. For a multi-account integration, include the provider account when the provider only guarantees uniqueness inside that account.

The payload is stored here because the worker needs durable input. That choice has privacy and storage consequences. Encrypt or omit sensitive fields, restrict access, and give receipts an explicit retention policy.

The unique constraint handles simultaneous delivery

“Check, then insert” is unsafe:

request A: SELECT finds no receipt
request B: SELECT finds no receipt
request A: performs the effect
request B: performs the effect

An in-memory Set has the same problem across processes and disappears on restart. A Redis lock can expire or be lost between the lock and the database commit. A processed boolean has no value if two rows for the same event can be created or if the flag commits separately from the effect.

Put the race at the database uniqueness boundary instead:

INSERT INTO webhook_receipts (provider, event_id, event_type, payload)
VALUES ($1, $2, $3, $4::jsonb)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING event_id;

PostgreSQL uses the primary-key index as the conflict arbiter. Under concurrent inserts, one transaction can insert the key. Other transactions wait for the conflicting transaction to commit or abort, then follow the conflict outcome. The application does not need to predict which request arrives first.

Return 2xx for an authenticated duplicate whose original receipt is durable. A duplicate is not a client error, and returning failure only invites more delivery attempts. Log it as a duplicate receipt so the retry remains visible.

Put the local effect in the same transaction

Persisting a receipt before processing gives fast acknowledgement, but a receipt alone does not prove the business work finished. The worker needs explicit states such as pending, processing, processed, and failed, plus a lease or retry time if multiple workers can claim work.

For a local PostgreSQL effect, make the receipt transition and domain update one transaction. Here is the core operation from the accompanying lab:

async function applyEvent(event: CreditEvent) {
  const client = await pool.connect();
  try {
    await client.query("BEGIN");

    const receipt = await client.query(
      `INSERT INTO webhook_receipts
         (provider, event_id, event_type, payload)
       VALUES ($1, $2, $3, $4::jsonb)
       ON CONFLICT (provider, event_id) DO NOTHING
       RETURNING event_id`,
      [event.provider, event.id, event.type, JSON.stringify(event.data)],
    );

    if (receipt.rowCount === 0) {
      await client.query("ROLLBACK");
      return { outcome: "duplicate" as const };
    }

    await client.query(
      `INSERT INTO account_credits
         (provider, event_id, account_id, amount_cents)
       VALUES ($1, $2, $3, $4)`,
      [event.provider, event.id, event.data.accountId, event.data.amountCents],
    );
    await client.query(
      `UPDATE webhook_receipts SET processed_at = clock_timestamp()
       WHERE provider = $1 AND event_id = $2`,
      [event.provider, event.id],
    );

    await client.query("COMMIT");
    return { outcome: "applied" as const };
  } catch (error) {
    await client.query("ROLLBACK");
    throw error;
  } finally {
    client.release();
  }
}

The schema also gives account_credits a primary key on (provider, event_id). That second constraint is defense in depth. The receipt claim, domain insert, and processed timestamp either commit together or all roll back.

If the worker crashes before commit, PostgreSQL rolls the transaction back and a later attempt can apply the event. If it crashes after commit, a later attempt meets the unique key and returns duplicate. The exact worker-claim query may use FOR UPDATE SKIP LOCKED, a queue, or another leasing mechanism. None of those choices should move the correctness boundary out of the transaction.

The BullMQ retry reproduction applies the same principle to queued jobs: queue delivery behavior and database effects require separate reasoning.

The concurrency experiment

The repository includes a small lab in research/webhook-inbox-lab. It ran on Ubuntu 24.04 in the project container with Node.js 24.19.0, pg 8.16.3, and a disposable PostgreSQL 16.15 cluster. PostgreSQL used its defaults: 100 maximum connections, 128 MB shared buffers, locale C, and data checksums disabled.

The dataset began empty. One test submitted the same event through 20 concurrent pool calls. A second test inserted a receipt and rolled the transaction back, then retried the event. Node’s built-in test runner measured assertions, not throughput. There was no warm-up because this was a correctness test. One run was sufficient for the deterministic assertions; the concurrent test was also run five additional times during review with the same counts.

cd research/webhook-inbox-lab
npm install
psql "$DATABASE_URL" -f schema.sql
npm test

The result supports a narrow claim. The unique key and transaction produced one local database effect for one provider event ID in this environment. The lab does not measure latency, sustained load, failover, network partitions, or remote API behavior.

Failure matrix: what survives each crash window

This matrix separates sender delivery, receiver storage, local work, and remote work. “Safe” means a retry does not repeat the stated local effect under the schema and transaction shown above.

Notice the final remote-call row. PostgreSQL cannot roll back an email provider, payment API, or calendar service. If the remote provider accepts a request and the local worker loses the response, the outcome is unknown.

Use a stable provider-side idempotency key when available. Persist the outbound command before sending it. Record the remote operation identifier. On uncertain outcomes, query provider state before resending. If the provider supports none of those mechanisms, route uncertainty to an operator instead of retrying an irreversible action without a bound.

Queue handoff: receipt table or transactional outbox

There are two sound handoff shapes.

The first uses the receipt table itself as the work queue. Workers claim pending rows in bounded batches. This has few moving parts and keeps acceptance durable, but PostgreSQL now carries queue polling and retention traffic.

The second inserts both the receipt and an outbox message in one transaction. A relay publishes the outbox message to the queue and marks it published. The relay may publish twice if it crashes after publication but before marking the row. That is acceptable only because the consumer remains idempotent.

Publishing directly to a queue after the receipt commit leaves a gap: the process can exit between commit and publish. Publishing first leaves the opposite gap: the worker can run before the receipt exists. A transactional outbox closes the local database-to-queue omission window. It does not turn the broker and database into one transaction, and duplicate publication remains possible.

Ordering needs a business rule

Deduplication answers “have I accepted this event ID?” It does not answer “is this event newer than the state I hold?” Stripe explicitly does not guarantee delivery order, and separate event objects can describe related changes.

Choose an ordering rule from the domain:

• For state snapshots, fetch the current resource from the provider and apply that authoritative state.
• For monotonic versions, update only when the incoming version exceeds the stored version.
• For transitions, model allowed state changes and quarantine an impossible predecessor rather than forcing the transition.
• For commutative facts such as ledger entries, key each fact independently and derive the balance from accepted facts.

Do not use receipt time as event order. Network delay and retry scheduling make it an observation time. Provider timestamps may be too coarse or may describe creation rather than causal order. Use a documented sequence or resource version when one exists.

Payload versions and changing handlers

A stored payload can outlive the code that first accepted it. Record the provider’s event type, account scope, API version when supplied, receipt time, and a handler version. Preserve the verified bytes or a faithful parsed envelope according to audit and privacy requirements.

When a handler changes, do not silently reinterpret old receipts with a new schema. Either keep a versioned decoder, normalize the event into an internal command at receipt time, or migrate pending payloads explicitly. Unknown fields should usually be tolerated. Missing required fields should produce a permanent failure state with enough metadata for repair.

Retention and replay must be designed together

Deleting the receipt reopens its event ID. Retention therefore defines how long the idempotency guarantee lasts.

Keep receipts longer than the largest of:

• the provider’s automatic retry period;
• its manual redelivery period;
• your own queue and dead-letter retention;
• the longest outage after which operators may replay work;
• any audit period required to explain the effect.

Stripe’s current documentation illustrates why one number is insufficient: live automatic retries can continue for three days, Dashboard resends are available for 15 days, and CLI resends for 30 days. Other providers differ. If privacy or cost requires earlier payload deletion, retain a minimal keyed tombstone after removing the full body.

Partitioning receipts by time can make deletion predictable, but a PostgreSQL unique constraint on a partitioned table has version-specific restrictions. Do not weaken uniqueness to obtain easy retention. A separate compact key table is often clearer.

Observability and reconciliation

Log one structured record for every receipt attempt with:

• provider and endpoint identifier;
• event ID, type, and provider account;
• signature result and timestamp age, without the secret;
• receipt outcome: inserted, duplicate, rejected, or failed;
• handler version and processing attempt;
• local transaction outcome and remote operation ID;
• request correlation ID and trace ID.

Useful counters include accepted events, authenticated duplicates, signature failures, receipt insert failures, pending age, processing retries, permanent failures, and reconciliation mismatches. Alert on growing oldest-pending age and repeated permanent failures, not on every duplicate. Duplicates are expected under a retrying transport; a sudden change in their rate is the signal.

Reconciliation closes gaps that request handling cannot observe. On a schedule, list provider objects or events for a bounded time range, compare them with local receipts and domain state, insert missing commands through the same idempotent path, and record a checkpoint only after the page is complete. Make the overlap larger than expected clock error and pagination delay. Re-reading an overlap is safe because the receipt key remains the arbiter.

Misleading fixes and the guarantee they miss

These mechanisms are not useless. They solve different problems. The mistake is asking a coordination aid or transport option to guarantee a database effect.

Production operating checklist

Before accepting traffic:

1. Confirm the provider’s stable event identifier and its uniqueness scope.
2. Verify signatures over raw bytes with the provider’s current SDK or exact documented scheme.
3. Monitor clock synchronization if the scheme enforces timestamp age.
4. Commit the receipt before returning 2xx.
5. Put the receipt key under a PostgreSQL unique constraint.
6. Commit the receipt state and local domain effect in one transaction.
7. Use an outbox when a queue publish must follow a database commit.
8. Define an ordering rule that does not depend on arrival time.
9. Keep idempotency keys through all automatic, manual, and operator replay windows.
10. Give remote calls stable idempotency keys or an explicit uncertain state.
11. Reconcile provider state through the same idempotent path.
12. Test concurrent delivery, lost acknowledgement, rollback, crash after commit, and manual replay.

The boundary is now precise. The transport may deliver more than once. The application may execute more than once. PostgreSQL admits one provider-scoped receipt and one local effect because both are protected by uniqueness and one transaction. Anything beyond that database needs a new boundary and new evidence.