Background Jobs in Node.js: Queues, Workers and Scheduled Tasks

A Node.js API that sends email, resizes images or builds reports inside the request handler makes the caller wait for that work and loses it when the process restarts mid-job. Background jobs fix both problems by moving the work out of the request: the handler records that something needs doing, returns, and a worker picks it up with retries if it fails.

This guide builds a report generator with a queue, a worker with retries and a dead-letter queue, a nightly schedule and a bucket for the results, using Encore, and then compares BullMQ, pg-boss, SQS and in-process timers. The Encore code below was run with encore run and the outputs are from that run.

Node.js background job options at a glance

How do you run Node.js background jobs with Encore?

Encore combines an open source infrastructure SDK with a platform that provisions and deploys into your own AWS or GCP account. For TypeScript it is a backend framework with built-in infrastructure: the queue, the worker, the schedule and the bucket are declared in the same file as the API, and Encore runs them locally and provisions them in the cloud.

Set up the app

Each service has an encore.service.ts file:

// reports/encore.service.ts
import { Service } from "encore.dev/service";

export default new Service("reports");

The queue, the worker and the schedule

A Pub/Sub topic is the queue, a subscription is the worker, an object storage bucket holds the finished reports, and a cron job enqueues the nightly run:

// reports/reports.ts
import { api, APIError } from "encore.dev/api";
import { CronJob } from "encore.dev/cron";
import { Subscription, Topic } from "encore.dev/pubsub";
import { Bucket } from "encore.dev/storage/objects";
import { getAuthData } from "~encore/auth";

// Finished reports. S3 on AWS, Cloud Storage on GCP, a local emulator in `encore run`.
const reports = new Bucket("reports", { versioned: false });

interface ReportJob {
  accountId: string;
  day: string; // YYYY-MM-DD
}

// The queue. SNS and SQS on AWS, Pub/Sub on GCP.
export const reportJobs = new Topic<ReportJob>("report-jobs", {
  deliveryGuarantee: "at-least-once",
});

function reportKey(job: ReportJob): string {
  return `${job.accountId}/${job.day}.json`;
}

// The worker. Throwing an error retries the message with backoff, and after
// maxRetries attempts it moves to a dead-letter queue.
const _ = new Subscription(reportJobs, "build-report", {
  handler: async (job) => {
    const summary = {
      accountId: job.accountId,
      day: job.day,
      generatedAt: new Date().toISOString(),
    };
    // A fixed key makes a redelivered job overwrite the same file.
    await reports.upload(reportKey(job), Buffer.from(JSON.stringify(summary)), {
      contentType: "application/json",
    });
  },
  maxConcurrency: 10,
  ackDeadline: "2m",
  retryPolicy: { minBackoff: "10s", maxBackoff: "5m", maxRetries: 5 },
});

// Enqueue a report and return right away. The account comes from the caller,
// not from the request body.
export const enqueue = api(
  { expose: true, auth: true, method: "POST", path: "/reports" },
  async (job: { day: string }): Promise<{ queued: boolean }> => {
    if (!/^\d{4}-\d{2}-\d{2}$/.test(job.day)) {
      throw APIError.invalidArgument("day must be YYYY-MM-DD");
    }
    await reportJobs.publish({ accountId: getAuthData()!.userID, day: job.day });
    return { queued: true };
  },
);

// Check whether the caller's report has been built. The body stays in the bucket.
export const status = api(
  { expose: true, auth: true, method: "GET", path: "/reports/:day" },
  async (p: { day: string }): Promise<{ ready: boolean }> => {
    if (!/^\d{4}-\d{2}-\d{2}$/.test(p.day)) {
      throw APIError.invalidArgument("day must be YYYY-MM-DD");
    }
    const ready = await reports.exists(reportKey({ accountId: getAuthData()!.userID, day: p.day }));
    return { ready };
  },
);

// Called by the nightly cron job. Cron endpoints take no parameters.
export const enqueueNightly = api({}, async (): Promise<void> => {
  const day = new Date(Date.now() - 86_400_000).toISOString().slice(0, 10);
  for (const accountId of ["acme", "globex"]) {
    await reportJobs.publish({ accountId, day });
  }
});

// 02:00 UTC every day, scheduled by the Encore platform once deployed.
const nightly = new CronJob("nightly-reports", {
  title: "Build yesterday's reports",
  schedule: "0 2 * * *",
  endpoint: enqueueNightly,
});

Delivery is at least once, so the handler has to be safe to run twice. Here the report key is derived from the job, and a redelivered message overwrites the same file instead of creating a second one. Jobs that charge a card or send an email need a stored idempotency key instead.

Run it locally

encore run starts the queue and the bucket on your machine. Cron jobs do not fire locally, so the nightly endpoint is called by hand. It is not exposed publicly, and local calls still reach it:

Publishing the same job a second time replaced the file with a new generatedAt and left one report, which is the idempotent behavior the handler was written for. The local development dashboard shows each publish and delivery in the trace view.

Deploy

On AWS, Encore provisions an SNS topic and an SQS queue with a dead-letter queue for the subscription, an S3 bucket for the reports, and the compute for the service, and it provisions an IAM role for the service. On GCP it uses Pub/Sub and Cloud Storage. Cron schedules are managed by the Encore platform and need no extra cloud resources, as described in the infrastructure overview. Messages that keep failing land in the dead-letter queue instead of blocking the topic.

Best fit: TypeScript backends that want queues, workers and schedules declared next to the API, with the same setup locally and in your own AWS or GCP account.

BullMQ

BullMQ is a widely used Node.js queue library that keeps jobs in Redis. Workers are separate processes that call new Worker(queueName, processor), and retries are configured per job with attempts and a backoff strategy. Repeatable jobs cover schedules, and flows model parent and child jobs. The cost is Redis as a durable store: it needs persistence turned on, a size that fits the backlog, and in production a managed service such as ElastiCache, plus a worker deployment of its own. Redis eviction policies other than noeviction can drop jobs.

Best fit: Node.js teams that already run Redis and want rich job features such as priorities, rate limits and parent-child flows.

pg-boss

pg-boss stores jobs in Postgres tables and uses SKIP LOCKED to hand them to workers, so an app that already has Postgres gets a queue without another service. Jobs can be created inside the same transaction as the data they refer to, which removes a class of lost-job bugs. Throughput is lower than a dedicated broker, and heavy job traffic adds load and vacuum work to the main database.

Best fit: apps already on Postgres with moderate job volume that want jobs and data in the same transaction.

SQS with the AWS SDK

Using Amazon SQS directly means SendMessageCommand from the API and a long-polling loop of ReceiveMessageCommand and DeleteMessageCommand in a worker. Retries come from the visibility timeout, and a redrive policy moves messages to a dead-letter queue after a set number of receives. You provision the queue, the DLQ, IAM permissions for both the publisher and the worker and the worker service itself, and local development needs LocalStack or a real queue.

Best fit: teams on AWS that want the managed queue and are prepared to own the queue definitions and worker deployment in Terraform or CDK.

In-process timers and promises

Firing a promise without awaiting it, or running setInterval in the API process, is the zero-setup option. It works for work that can be lost, such as warming a cache. A deploy, a crash or a scale-in event drops every job in flight, nothing retries, and with more than one instance a setInterval schedule runs once per instance.

Best fit: best-effort work where losing a job has no consequence.

How to choose

Pick Encore when you want the queue, the worker, the schedule and the storage declared in TypeScript and provisioned in your own AWS or GCP account, with the whole flow running locally. Pick BullMQ when Redis is already in the stack and you need advanced job features, pg-boss when Postgres is the only datastore and volumes are moderate, SQS directly when your team already manages AWS resources in code, and in-process timers only for work you can afford to lose.

Related guides

• Cron jobs in TypeScript
• How to add Pub/Sub to a TypeScript API
• BullMQ alternatives
• Using SQS from Node.js
• Background jobs in Go