Reliable Third-Party Integrations: Timeouts, Backoff, and Reconciliation

A user asks your SaaS to provision a remote account. The provider receives the POST, creates the account, and its response disappears when the connection breaks. Your database still says “pending.” Sending the request again may create a second account; marking it failed may lie; waiting forever strands the user.

The operating rule is to persist intent before network work, give every business operation a stable identity, and model an uncertain remote outcome explicitly. Timeouts, retries, backoff, circuit breakers, and queues protect capacity. They do not tell you whether the provider committed a side effect. Idempotency and reconciliation close that gap, within their documented limits.

This note builds a Node.js SaaS integration around that distinction. Afterward, you should be able to classify a failure, schedule a bounded retry, identify unknown outcomes, reconcile them, and give an operator a safe replay path.

Model the whole outbound lifecycle

An integration request begins before HTTP and ends after local state agrees with remote state:

1. Validate the user’s requested operation and authorization.
2. Commit the domain change and an outbound command in one local transaction.
3. Dispatch the command under a per-provider concurrency and rate limit.
4. Resolve DNS, connect TCP, negotiate TLS, send headers and body, wait for response headers, and consume the required response body.
5. Classify the response or transport failure.
6. Record success, schedule a retry, mark a permanent rejection, or mark the outcome unknown.
7. Accept a provider webhook or poll provider state until terminal.
8. Reconcile local and remote records, then retain or erase evidence according to policy.

These steps cross two transaction domains. PostgreSQL cannot atomically commit with an arbitrary provider over HTTP. The provider can succeed while the local commit fails, or the local intent can commit while no request is ever sent. Reliable integration means preserving enough durable state to detect and repair both windows.

Give each phase a timeout and the operation a deadline

A single “30 second timeout” hides where time went. A request can spend time in DNS, a connection pool, TCP setup, TLS negotiation, request upload, time to first response header, and response-body download. Name the clocks your HTTP client actually exposes. Some clients combine DNS with connection time; document that boundary rather than inventing precision.

Use three levels:

• Phase timeouts stop a stuck connect, TLS handshake, upload, header wait, or body read. They identify which resource remained occupied.
• Attempt timeout bounds one complete HTTP attempt. Node.js 24’s AbortSignal.timeout(delay) can supply a signal for this outer bound, but it does not report phase timings by itself.
• Operation deadline bounds the initial attempt, waits, retries, queue delay, and reconciliation before the business workflow escalates.

Choose values from measured provider latency and the caller’s budget. A timeout below the provider’s normal tail manufactures retries. A timeout above your own request deadline keeps consuming sockets and concurrency after the caller has left. Warm connections can make a test look healthy while production requests pay DNS and TLS setup; measure both cold and reused paths.

Cancellation is local. Aborting a fetch tells the client to stop waiting. It does not roll back bytes already received by the provider. If the request body may have crossed the network, the outcome is unknown until an idempotent retry or a status lookup establishes it.

Classify outcomes before retrying

HTTP status alone is not the business result. A 202 Accepted can start an asynchronous operation, a 409 Conflict can mean duplicate success or invalid state, and a 500 can be returned after a provider commits work. Build the classification from the provider contract, HTTP semantics, method, and request phase.

RFC 6585 defines 429 Too Many Requests and allows a response to include Retry-After. RFC 9110 defines Retry-After as either a non-negative number of seconds or an HTTP date. Treat it as a lower bound suggested by the server, then also enforce your deadline and add jitter if a large local fleet would otherwise wake at once. Reject malformed or absurd values according to a documented cap.

Do not retry every 4xx, and do not assume every 5xx means “nothing happened.” The code in the accompanying lab deliberately classifies only a conservative set of statuses. Each provider adapter must narrow or extend that set from its contract.

Persist a stable operation identity

An idempotency key identifies one intended business operation across attempts. Generate it once when the command is created, store it durably, and send the same key with byte-equivalent semantic parameters on every retry. Do not create a new key per attempt.

Idempotency is provider-specific. Record:

• which methods and endpoints accept a key;
• how long results are retained;
• whether concurrent equal keys are serialized or rejected;
• whether parameter mismatch is detected;
• which errors are cached;
• whether a key scopes to account, endpoint, region, or API version;
• how a status lookup finds the resulting remote object.

For one concrete boundary, Stripe’s API v1 documentation read on 2026-10-14 says it stores the first status and response after endpoint execution begins, including 500 responses. Keys can be removed after they are at least 24 hours old, and reuse after pruning starts a new request. That is not a universal idempotency contract, and it means a command cannot wait indefinitely then be blindly replayed with the old key.

Use a random, opaque key or a non-sensitive command identifier. Never put an email, card detail, or other personal data in the key, logs, metric labels, or trace attributes.

Commit intent with a transactional outbox

The API request that accepts the user’s command should not both commit local state and call the provider as if those actions were atomic. Commit the domain record and an outbox command in one PostgreSQL transaction:

CREATE TABLE integration_commands (
  id uuid PRIMARY KEY,
  tenant_id uuid NOT NULL,
  provider text NOT NULL,
  operation text NOT NULL,
  idempotency_key text NOT NULL,
  request_payload jsonb NOT NULL,
  state text NOT NULL CHECK (state IN (
    'ready', 'in_flight', 'retry_wait', 'pending_remote',
    'needs_reconciliation', 'succeeded', 'failed_permanently', 'dead'
  )),
  attempt_count integer NOT NULL DEFAULT 0,
  next_attempt_at timestamptz,
  lease_expires_at timestamptz,
  remote_id text,
  last_error_code text,
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (provider, idempotency_key)
);

Keep secrets outside the payload. Encrypt sensitive fields where required, and store the minimum request needed for a retry or audit. A separate immutable attempt table can record timings and classifications without overwriting the command’s current state.

A dispatcher claims due commands with a short database transaction, increments the attempt, writes a lease, and commits before HTTP. PostgreSQL’s FOR UPDATE SKIP LOCKED is useful for queue-like consumers because workers can skip rows another worker holds. It gives row-claim behavior, not end-to-end exactly-once execution. If the lease expires after a worker crash, another worker will send the command again.

Never hold the row lock or database transaction open while waiting for the provider. That couples remote latency to database locks and pool capacity.

Treat command state as a recovery protocol

Figure: network uncertainty enters needs_reconciliation; only observed provider state or a safe idempotent retry leaves it.

State transitions need compare-and-set protection so a late HTTP response does not overwrite a newer webhook result. For example, update in_flight to succeeded only when the expected attempt number still owns the lease. Terminal provider success should dominate an older timeout.

The most dangerous window is easy to name:

provider commits -> response is lost -> worker cannot commit local success

If the provider honors the stored idempotency key, a retry can return the prior result within the provider’s retention and scope. If it offers lookup by your business reference, reconciliation can find the object. If neither exists, a blind retry may duplicate the effect. Route the command to human review with the request evidence; do not turn uncertainty into failed for dashboard comfort.

A webhook can complete a pending outbound command. Match it using a verified provider event, remote object ID, or stored business reference, then apply a monotonic state transition. The inbound handler still needs durable receipt and deduplication. The related webhook retry field note covers that transaction boundary without repeating it here.

Back off without manufacturing a retry wave

Capped exponential backoff increases the maximum wait between attempts:

ceiling(retry) = min(cap, base * 2^retry)

Without randomness, clients that fail together remain synchronized. Common jitter variants choose different delay distributions:

• Full jitter: random value from zero to the exponential ceiling.
• Equal jitter: half the ceiling plus a random value over the other half.
• Decorrelated jitter: random value between the base and three times the previous delay, capped.

Full jitter is a sound default when the provider does not specify an algorithm. Equal jitter avoids very short waits but can retain more clustering. Decorrelated jitter spreads successive attempts differently and can extend the operation tail. These are scheduling policies, not correctness mechanisms.

The project lab scheduled five retries for 10,000 simultaneous clients with a 500 ms base, 8,000 ms cap, and 100 ms bins. It ran five deterministic seeds on Node.js 24.19.0, Linux x64:

Run it with:

cd research/integration-retry-lab
npm test

The lab measures scheduled collision, not provider throughput. It omits timer precision, network duration, event-loop delay, successes that cancel later attempts, and server recovery. It supports the narrow claim that randomization spread this configured retry population; it does not rank one algorithm for all systems.

Bound retries by attempts, time, and load

An attempt limit alone is weak. Ten thousand commands each allowed three retries can still create thirty thousand extra calls during an outage. Apply all of these bounds:

• a maximum attempt count;
• an operation deadline after which automated retry stops;
• a per-provider retry budget, such as tokens replenished by successful first attempts;
• per-tenant fairness so one customer cannot consume the provider pool;
• a global concurrency limit and a rate limiter aligned to documented quota scopes.

A retry budget is admission control for retry traffic. When depleted, new business commands may remain queued or fail according to product policy, and retryable commands wait for reconciliation or later capacity. AWS SDK standard retry mode is a concrete example: its retry quota uses tokens, stops retries when the quota is empty, and restores tokens after success. Your numbers must come from your traffic, SLO, and provider quota rather than copying the SDK’s.

Rate and concurrency are different. A token bucket can limit starts per second; a semaphore limits simultaneous in-flight calls. A slow provider can exhaust concurrency even below its request-per-second quota. Use separate bulkheads for providers, and sometimes for expensive operations within one provider, so a slow document export cannot consume every payment slot.

Use circuit breakers only for the failure they isolate

A circuit breaker stops calls after a failure threshold, then probes for recovery. It can reduce wasted work against a broadly failing dependency. It can also cause harm:

• a threshold based on mixed endpoints can block healthy operations;
• an open circuit can hide recovery and lengthen an outage;
• synchronized half-open probes can create another burst;
• local instances can disagree about circuit state;
• a provider 4xx caused by one tenant’s credentials can trip everyone;
• rejecting reconciliation calls can prevent uncertain operations from being resolved.

Prefer endpoint- and failure-class-specific breakers, limited probes with jitter, and explicit observability of the open state. A concurrency bulkhead and retry budget often provide smoother overload protection because they do not switch all traffic into a new mode. Test the breaker’s transitions, not only its happy-path wrapper.

Dead letters are evidence, not resolution

A command becomes a poison message when repeated handling cannot make progress, often because of invalid data, a deterministic code defect, an incompatible API version, or a permanent remote rule. After bounded attempts or the operation deadline, move it to a durable dead state or dead-letter queue with:

• command ID, tenant, provider, operation, and schema version;
• attempt count and first/last timestamps;
• normalized error class and provider request ID;
• idempotency key reference and remote ID when safe;
• payload reference subject to access and retention policy;
• the code and adapter version that classified it.

Do not automatically replay an entire dead-letter queue after deploying a fix. The provider may already have performed some operations, keys may have expired, and payload meaning may have changed. A replay tool should show current provider state, redact secrets, require a reason and operator identity, reuse the original operation identity when valid, and create an audit record. Reconciliation comes before replay for any unknown outcome.

Reconcile local intent against provider truth

Run reconciliation continuously for pending_remote, expired in_flight, and needs_reconciliation commands, plus periodically for recent successes where the business cost justifies it.

One practical algorithm is:

for each due command claimed under a short lease:
  query provider by remote_id, idempotency key, or unique business reference

  if provider shows the intended terminal state:
    transactionally record remote identifiers and local success
  else if provider shows a terminal rejection:
    record permanent failure and the normalized reason
  else if provider shows work still pending:
    persist provider state and schedule the next observation
  else if provider proves no operation exists and key is still valid:
    return command to retry_wait with the same identity
  else:
    keep needs_reconciliation and escalate for bounded manual review

Make it convergent: running it twice should move the record to the same state. Use pagination checkpoints for provider list APIs, account for their consistency model, and record the observation time. Absence from an eventually consistent listing is not proof of nonexistence.

Reconciliation also catches missed webhooks, manual provider changes, and local write failures. Track mismatch age, not only mismatch count. A single payment unknown for two days can matter more than a thousand five-second provisioning delays.

Observe the business operation, not only HTTP calls

Propagate a trace context when the provider supports it, but retain your own command ID and the provider’s request ID. One trace rarely spans a command that waits hours for a webhook, so logs and metrics must correlate across executions.

Useful low-cardinality metrics include:

• command creations and terminal outcomes by provider and operation;
• attempt latency by phase and outcome class;
• retry rate, retry-budget exhaustion, and Retry-After delay;
• in-flight concurrency, queue age, and oldest due command;
• commands in unknown, pending, permanent-failure, and dead states;
• reconciliation mismatch count and age;
• webhook completion lag and polling fallback rate.

Define SLOs on business completion, such as “99 percent of accepted provisioning commands reach a known terminal state within ten minutes,” not only HTTP success rate. Alert on burn rate, oldest command, sustained unknown outcomes, depleted retry budget, and reconciliation lag. A provider returning fast 401 responses can have excellent latency and zero useful completions.

Keep attempt logs long enough to investigate and reconcile, but no longer than the declared purpose requires. Hash or omit payload values in logs, restrict operator views by tenant, and ensure deletion propagates to command payloads, dead letters, traces, backups, and vendor data where contracts require it.

Operator runbook

When a provider degrades:

1. Confirm scope by provider, endpoint, account, region, error class, and request phase. Separate throttling from authentication, validation, and timeout.
2. Check queue age, in-flight concurrency, retry budget, rate limiter, breaker state, and your remaining operation deadlines.
3. Read provider status and documentation, but trust observed request IDs and account-specific quota evidence over a green status page.
4. Reduce admission or pause retryable work if retries are adding load. Preserve durable commands; do not discard or mass-replay them.
5. Keep reconciliation traffic available when it can resolve unknown outcomes without repeating side effects.
6. After recovery, release work gradually with jitter and per-tenant fairness.
7. Reconcile the incident window, then review unknown and dead commands before any manual replay.
8. Record provider response samples with secrets removed, policy changes, and the exact time normal completion resumed.

Before launching any integration, answer five questions in writing: What is the stable operation identity? Which timeout phases does the client really cover? Which outcomes are safe to retry? How is remote truth queried? Who can resolve a dead or unknown command? If one answer is missing, another retry wrapper will not supply it.

Limitations

The architecture assumes the provider offers at least one usable correlation mechanism: an idempotency key, remote operation ID, webhook reference, or lookup by a unique business key. When none exists, some timeout windows cannot be resolved automatically. A compensating action may be possible, but compensation is a new fallible operation, not a rollback.

The lab is a scheduling simulation, not a network benchmark. The SQL omits payload encryption, partitioning, attempt-history tables, and tenant foreign keys to keep the state model readable. Adapt retention, indexes, leases, and timeouts from measured workload and provider contracts.