Nov 20, 20266 min read

SQS vs Redis as a Job Queue

How Amazon SQS and Redis compare for background jobs: durability, delivery guarantees, retries and dead letters, delayed jobs, throughput, operations and cost shape, and which one fits which workload.

Illustration for SQS vs Redis as a Job Queue

Background jobs need somewhere to wait, and on AWS the two most common answers are SQS, a managed queue with nothing to run, and Redis, usually already deployed as a cache, and both move work off the request path.

They disagree about durability, and that matters most on the day a node restarts with ten thousand jobs in memory. This comparison covers delivery guarantees, failure handling, features and operations.

At a glance

Amazon SQS (standard)Redis as a queue
StorageManaged, replicated across AZsIn memory, persisted by RDB or AOF
DeliveryAt least onceDepends on pattern: lists, BLMOVE or Streams
In-flight trackingVisibility timeout, up to 12 hoursProcessing list or Streams pending entries
Retries and dead lettersRedrive policy to a DLQImplemented by the library or your code
Delayed jobsUp to 15 minutes per messageSorted sets, any delay
Max payload1 MiBUp to 512 MB per value, memory permitting
LatencyTens of millisecondsSub-millisecond
RetentionUp to 14 daysUntil memory runs out
OperationsNoneMemory sizing, persistence, failover

Durability and delivery

SQS stores each message redundantly before acknowledging the send, and a received message stays hidden for the visibility timeout until the consumer deletes it. A worker that crashes mid-job lets the timeout expire, and the message reappears for another worker. The guarantee is at least once, so handlers must be idempotent.

Redis offers several queue patterns with different guarantees, and a plain LPUSH and BRPOP loses the job if the worker dies after popping it. BLMOVE moves the job atomically into a per-worker processing list, which a reaper can requeue, and Streams with consumer groups track unacknowledged entries in a pending list that XAUTOCLAIM can reassign. All of these sit on top of Redis persistence, and the persistence docs are explicit that AOF with appendfsync everysec can lose about a second of writes in a crash. A failover to an asynchronous replica can lose more.

Failure handling and features

SQS handles the unhappy path in configuration: after maxReceiveCount failed receives, the redrive policy moves the message to a dead-letter queue, and a redrive back to the source queue is one console action once the bug is fixed. Messages are limited to 1 MiB according to the SQS message quotas, and delays cap at 15 minutes, so jobs scheduled hours ahead need another mechanism.

Redis queues get their features from libraries, where BullMQ for Node.js and asynq for Go add priorities, rate limiting, repeatable jobs, arbitrary delays and dashboards, which SQS does not have natively. The trade is that those features, and their retry and dead-letter behavior, are code running in your workers rather than guarantees of the storage layer.

Throughput, latency and cost shape

Standard SQS queues scale throughput without configuration, and FIFO queues reach thousands of messages per second with batching and far more in high-throughput mode. Each send, receive and delete is a billed request, so long polling and batching of up to 10 messages keep the request count down.

Redis is faster per operation and has no per-request charge, but the queue lives in the same memory as everything else on that instance. A backlog that grows during an outage competes with the cache for memory, and when maxmemory is reached with an eviction policy enabled, Redis can evict queue keys along with cache keys. Running a separate Redis for queues with noeviction avoids that at the cost of another instance to operate.

When to pick which

  • SQS: jobs that must not be lost, such as payments, emails and webhooks, teams that do not want to operate a queue, and workloads with bursty backlogs.
  • Redis: high-rate, short-lived jobs where latency matters more than the rare loss, and teams who want BullMQ-style features and already run Redis with persistence and failover configured.

Best fit: SQS for durable business jobs, Redis for fast, feature-rich queues where a lost job is recoverable.

Queues with Encore

Encore

Encore brings infrastructure from code to background jobs: a topic and its subscriptions are declared in the service, and on AWS, Encore provisions SNS and SQS for them in your own account, with Google Cloud Pub/Sub on GCP, as the infrastructure docs describe. Retries, the dead-letter queue and the ack deadline come from the subscription config in code rather than a separate redrive policy:

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

package email import ( "context" "time" "encore.dev/pubsub" "encore.dev/storage/sqldb" ) type SendEmail struct { MessageID string `json:"message_id"` // unique per email, used for idempotency To string `json:"to"` Template string `json:"template"` } var Outbox = pubsub.NewTopic[*SendEmail]("send-email", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var db = sqldb.NewDatabase("email", sqldb.DatabaseConfig{Migrations: "./migrations"}) var _ = pubsub.NewSubscription(Outbox, "deliver", pubsub.SubscriptionConfig[*SendEmail]{ Handler: deliver, MaxConcurrency: 20, AckDeadline: 60 * time.Second, RetryPolicy: &pubsub.RetryPolicy{ MinBackoff: 10 * time.Second, MaxBackoff: 10 * time.Minute, MaxRetries: 8, // then the message goes to the subscription's dead-letter queue }, }) func deliver(ctx context.Context, msg *SendEmail) error { // Delivery is at least once, so skip messages that were already sent. var sent bool if err := db.QueryRow(ctx, `SELECT EXISTS (SELECT 1 FROM sent_emails WHERE message_id = $1)`, msg.MessageID).Scan(&sent); err != nil { return err } if sent { return nil } // The provider deduplicates on MessageID, which covers a crash between send and insert. if err := sendWithProvider(ctx, msg); err != nil { return err // retried with backoff, then dead-lettered } _, err := db.Exec(ctx, `INSERT INTO sent_emails (message_id) VALUES ($1) ON CONFLICT DO NOTHING`, msg.MessageID) return err } func sendWithProvider(ctx context.Context, msg *SendEmail) error { // Email provider call goes here, with msg.MessageID as its idempotency key. return nil }

encore run runs topics and subscriptions in-process on your machine, so publishing from an endpoint and watching the handler run takes no AWS account, and each delivery shows up in the local development dashboard. The Pub/Sub docs cover ordering keys and exactly-once topics, and in production the queues are visible in the SQS console of your account like any other, as listed in the AWS infrastructure docs.

With the queue, its retry policy and the idempotency check in one file, an AI coding agent adding a new job type writes all three in one change and can trigger it locally before anyone reviews it.

Best fit: services on AWS or GCP that want durable queues with retries and dead letters declared in code.

How to choose

Default to SQS for jobs with business consequences, since durability and dead letters come without operating anything. Choose Redis when per-job latency or library features like priorities and rate limits matter more, and run it as a dedicated instance with persistence and noeviction if the jobs matter at all.

Frequently asked questions

Is SQS or Redis better for a job queue?

SQS is better when losing a job is unacceptable and you want retries and dead-letter queues without running anything. Redis is better when you need sub-millisecond latency, rich job features like priorities and rate limits from a library such as BullMQ, and already run Redis with persistence configured.

Can Redis lose queued jobs?

Yes. Redis keeps data in memory and persists it with RDB snapshots or an append-only file. With appendfsync everysec, a crash can lose up to about one second of writes, and a failover to a replica can lose writes that had not replicated yet.

How large can an SQS message be?

An SQS message can be up to 1 MiB. Larger payloads go to S3 with only a reference in the message, which the extended client libraries automate for Java and Python.

What queue does Encore use on AWS?

Encore Pub/Sub runs on SNS and SQS when deployed to AWS and on Google Cloud Pub/Sub on GCP. Each subscription gets its own queue, with retries and a dead-letter queue configured from the RetryPolicy in code.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders