Why BullMQ Jobs Run Twice: Retries, Stalls, and Idempotent Side Effects

A BullMQ job can be added once and still reach your processor twice. That does not automatically mean BullMQ created a duplicate job. It can mean the queue is recovering work whose previous execution did not finish cleanly.

That distinction matters. If a worker writes to a database, sends a notification, or calls another service before it fails, a second execution can repeat the business effect even though the queue is behaving as designed.

I tested this in a NestJS 12 service using BullMQ 6.3.8, Redis 8.10.1, PostgreSQL 17.11, and Node.js 24.19.0. The lab forces a retry after a database write, kills a worker after the same write, and submits the same business operation twice. Each failure path then runs again with a durable operation key. The assertions run against real PostgreSQL and Redis services in CI.

The complete implementation is public in the NestJS BullMQ idempotency lab. It includes separate API and worker processes, a transactional outbox, durable command state, integration tests, operational endpoints, and the controlled failure runner used for the result table.

The queue may process an accepted job again. The handler must decide whether the business operation has already been applied.

The result in one table

The important result is not that the processor ran once. It did not. The safe version allowed repeated processing but prevented a repeated business effect.

First identify what was duplicated

“The job ran twice” can describe several different failures. Checking the wrong layer usually leads to a fix that only hides the symptom.

Start by logging the BullMQ job ID and a separate business operation key. A queue ID identifies BullMQ’s record. A business key identifies the action your system must apply once, such as apply-credit-482, send-registration-email-731, or sync-calendar-event-944.

Those identifiers answer different questions. You need both.

Why BullMQ can process a job again

There are two recovery paths worth understanding before changing any settings.

A failed processor can be retried

BullMQ moves a job to failed when its processor throws an error. When attempts is greater than one, BullMQ can move that job back for another attempt according to its backoff configuration. The behavior is documented in BullMQ’s retry guide.

The dangerous window is simple:

1. The handler commits a side effect.
2. Something throws before the processor returns successfully.
3. BullMQ retries the job.
4. The handler commits the side effect again.

Retry policy controls when work is attempted again. It does not make the work safe to repeat.

An active job can become stalled

When a worker starts processing a job, BullMQ places a lock on it. The worker renews that lock while it is active. If the worker exits, loses connectivity, or keeps the Node.js event loop busy long enough to miss renewal, another worker cannot know whether the first process finished the business operation.

BullMQ’s stalled-job documentation explains that a stalled job can return to waiting and be processed by another worker. This is a recovery mechanism. Without it, a crashed worker could leave work stuck forever.

From the queue’s perspective, retrying is safer than losing the job. From the application’s perspective, that choice means the handler must tolerate another execution.

The controlled reproduction

The companion repository runs a separate NestJS API and worker, a PostgreSQL transactional outbox, BullMQ on Redis, and a durable credit ledger. The controlled runner inside the same repository isolates the exception, worker termination, and duplicate producer paths without replacing the production-shaped implementation.

The pinned environment is:

Start from a clean clone and follow the README. After PostgreSQL and Redis are running, this command applies migrations, seeds the account, tests the complete API-to-worker path, and executes every controlled failure:

npm run verify:integration

The failure runner prints structured events and stops with a non-zero exit code if the expected execution and side-effect counts change. The repository also contains architecture decisions, an operations runbook, health and readiness checks, Prometheus metrics, OpenAPI documentation, CodeQL analysis, and a production container build.

Test 1: throw after the side effect

The unsafe worker writes the effect and then fails on its first attempt:

const worker = new Worker(
  queueName,
  async (job) => {
    await writeBusinessEffect(job.data.operationKey);

    if (job.attemptsMade === 0) {
      throw new Error("controlled failure after side effect");
    }
  },
  { connection },
);

await queue.add("apply-credit", data, {
  attempts: 2,
  backoff: { type: "fixed", delay: 100 },
});

The first database write succeeds. The thrown error causes another attempt. The second execution reaches the same write again.

The structured log showed the same BullMQ job ID in both executions:

jobId=1 attemptsMade=0 execution=1 sideEffectInserted=true
jobId=1 attemptsMade=1 execution=2 sideEffectInserted=true

The job eventually completed, but the database contained two effects. A green queue status did not mean the business result was correct.

With the durable operation key in place, the processor still ran twice:

jobId=1 attemptsMade=0 execution=1 sideEffectInserted=true
jobId=1 attemptsMade=1 execution=2 sideEffectInserted=false

The second execution found that the operation had already been claimed and committed. It returned without writing the effect again.

Test 2: terminate the worker after the write

The second test removes the clean error path. A child worker writes the effect and exits immediately, before BullMQ can record successful completion.

The test uses a short lock window so the recovery is fast enough to run locally. This timing is a reproduction setting, not a recommended production setting.

The observed sequence was:

Without protection, both workers wrote an effect. With the durable operation key, Worker B still processed the job but its insert was rejected as an existing operation. The final counts were two processor executions and one effect.

One detail from this test is easy to miss: the recovery execution still reported attemptsMade=0. The job had stalled rather than failed through the normal retry path. If monitoring only looks for an increased attempt count, it can miss this kind of repeated processing. Record stalled events and worker lifecycle data as well.

Test 3: submit the same operation twice

The first two tests accepted one job and processed it again. The third test asks a different question: what if the producer creates two jobs for the same business operation?

With default BullMQ IDs, the two calls returned job IDs 1 and 2. Both jobs ran and both wrote an effect.

With one deterministic job ID, both producer calls returned operation-producer-job-id. Only one queue job ran while that job record remained in the queue.

await queue.add("send-notification", payload, {
  jobId: `notification-${payload.registrationId}`,
});

BullMQ documents this behavior in its job ID guide. Adding a job with an existing ID is ignored while the existing job record is present. The same documentation also states an important limit: after a job is removed, the same ID can be added again.

A custom job ID is useful ingress protection. It does not replace an idempotent handler. It cannot stop one accepted job from being reprocessed after a failure, and retention settings affect how long its duplicate protection lasts.

The protection belongs at two layers

The tests support a layered design rather than one queue option presented as a complete fix.

Layer 1: suppress duplicate enqueueing

Use a deterministic job ID or BullMQ’s dedicated deduplication options when multiple producers can submit the same logical work.

Build the identifier from stable business data, not from request time or a new random value. Prefix numeric identifiers because BullMQ does not accept custom job IDs made only from digits. Avoid colons because BullMQ reserves that separator in job IDs.

This layer reduces unnecessary queue work. It is not the final correctness boundary.

Layer 2: make the business effect idempotent

The full implementation locks the command, claims a unique ledger operation, updates the account, and records the completed result in one PostgreSQL transaction. The central part is:

return this.database.withTransaction(async (client) => {
  const command = await this.lockCommand(client, commandId);

  if (command.status === "completed") {
    return this.completedResult(command);
  }

  const ledger = await client.query(
    `INSERT INTO credit_ledger
       (operation_key, command_id, account_id, amount_cents, source_job_id)
     VALUES ($1, $2, $3, $4, $5)
     ON CONFLICT (operation_key) DO NOTHING
     RETURNING id`,
    [
      command.operation_key,
      command.id,
      command.account_id,
      command.amount_cents,
      sourceJobId,
    ],
  );

  if (ledger.rowCount === 0) {
    throw new Error("Operation exists but its command is incomplete");
  }

  const account = await client.query(
    `UPDATE credit_accounts
     SET balance_cents = balance_cents + $2,
         version = version + 1,
         updated_at = now()
     WHERE id = $1
     RETURNING balance_cents::text`,
    [command.account_id, command.amount_cents],
  );

  await client.query(
    `UPDATE credit_commands
     SET status = 'completed',
         result = $2::jsonb,
         completed_at = now(),
         updated_at = now()
     WHERE id = $1`,
    [command.id, JSON.stringify(result)],
  );

  return result;
});

The unique constraint is necessary, but the transaction is the protection boundary. If the claim commits without the account update, a later attempt may skip work that never happened. If the account update commits without the claim, a later attempt may repeat it.

In MongoDB, the equivalent design normally uses a unique index on the operation key and a transaction when the marker and domain update span multiple documents. The syntax changes, but the atomic boundary does not.

BullMQ also recommends designing jobs to be idempotent and keeping them simple enough to retry safely in its idempotent-jobs pattern.

External APIs leave another failure window

A database transaction cannot atomically commit your local idempotency record and a request to an unrelated external API.

Consider this order:

1. Call the payment, email, calendar, or notification provider.
2. The provider accepts the request.
3. The worker exits before the local completion record is committed.
4. BullMQ processes the job again.

The local database does not know whether the provider completed step 2. Marking the operation complete before the API call creates the opposite problem: a crash can leave the marker present even though the API request was never sent.

Use the strongest option the provider and workflow support:

• Pass a stable idempotency key to the external API when it supports one.
• Store an outbox record in the same transaction as the local domain change.
• Reconcile provider state before retrying when the API exposes a safe lookup.
• Split a large job into smaller state transitions that can be resumed.
• Send uncertain cases to review instead of retrying blindly.

Do not hide this gap behind a Redis lock. The lock coordinates workers. It does not create a transaction across your database and another company’s API.

Why increasing the lock duration is not the fix

A longer lock duration may reduce false stalls for a known workload. Sandboxed processors can also help keep CPU-heavy work from blocking the main worker’s event loop. These are operational improvements, not correctness guarantees.

A process can still be terminated. A host can restart. A network can fail after the remote side has accepted a request. A deployment can interrupt work at the wrong instruction.

Tune lock and stall settings from observed execution time and worker health. Do not use them as proof that the handler will only run once.

What to log in production

The difference between a duplicate enqueue, a normal retry, and stalled recovery should be visible without reconstructing the incident from scattered strings.

I would log these fields for every processor start and finish:

logger.info({
  event: "queue-job-started",
  queue: job.queueName,
  jobName: job.name,
  jobId: job.id,
  operationKey: job.data.operationKey,
  attemptsMade: job.attemptsMade,
  workerPid: process.pid,
});

Also record:

• active, completed, failed, and stalled events
• whether the idempotency claim was inserted or already existed
• the error class and retry decision
• the worker instance or process ID
• timestamps for the business effect and queue completion
• the final state of jobs that reach the retry limit

Do not place sensitive payload data in these logs. The identifiers should be enough to join the timeline without exposing user information.

What this reproduction does not prove

This was a correctness test, not a throughput benchmark. It uses one PostgreSQL service, one Redis service, and controlled failure timing. It does not measure production latency, cluster failover, Redis persistence, network partitions, or the cost of the idempotency tables under sustained load.

It also does not prove that every report of a duplicated BullMQ job has the same cause. Duplicate producers, retry configuration, stalled workers, repeatable-job logic, application bugs, and duplicate logging need separate checks.

The reproduction proves a narrower point: once the same business operation can reach a handler more than once, a durable atomic idempotency boundary prevents a second database effect in the tested failure paths.

The operating rule

Clone the companion NestJS repository, run the complete path, and then change the failure timing or operation-key strategy to test the design against your own assumptions.

Use deterministic job IDs or deduplication to control what enters the queue. Use a durable business operation key to control what changes state. Keep the marker and the database effect in one transaction. Treat external calls as a separate consistency problem.

Retries are not the defect. They expose a handler that assumed one execution. Design the side effect so a second execution is boring.