SQS vs Kafka: Queue or Event Log for Your Backend?

SQS and Kafka both move messages between services, and teams often compare them when the first asynchronous job shows up: sending emails after signup, processing an upload, syncing to a CRM. They are built on different models, and picking the wrong one either means running a Kafka cluster for a job queue or rebuilding on Kafka later because SQS cannot replay events.

Amazon SQS is a managed queue where each message is processed and deleted. Apache Kafka is a distributed, replicated log where events stay for a retention period and many consumers read them independently. This guide compares the two on the points that decide the choice and covers how Encore (infrastructure from code) handles queues for Go and TypeScript services.

SQS vs Kafka at a glance

Pub/Sub on SNS and SQS from Go or TypeScript with Encore

Encore is infrastructure from code for TypeScript and Go, with a platform that provisions and deploys into your own AWS or GCP account. A topic and its subscriptions are declared in code, and on AWS Encore provisions an SNS topic with an SQS queue and dead-letter queue per subscription, or GCP Pub/Sub with dead-letter topics on GCP, as described in the AWS infrastructure docs.

Set up the app

The order service publishes an event and the receipt service handles it, with retries and a dead-letter queue configured on the subscription:

// order/order.go
package order

import (
	"context"

	"encore.dev/pubsub"
	"encore.dev/storage/sqldb"
)

var db = sqldb.NewDatabase("orders", sqldb.DatabaseConfig{
	Migrations: "./migrations",
})

type OrderPlaced struct {
	OrderID int64  `json:"order_id"`
	Email   string `json:"email"`
}

var OrderPlacedTopic = pubsub.NewTopic[*OrderPlaced]("order-placed", pubsub.TopicConfig{
	DeliveryGuarantee: pubsub.AtLeastOnce,
})

type PlaceParams struct {
	Email string `json:"email"`
}

//encore:api public method=POST path=/orders
func Place(ctx context.Context, p *PlaceParams) error {
	var id int64
	err := db.QueryRow(ctx,
		`INSERT INTO orders (email) VALUES ($1) RETURNING id`, p.Email).Scan(&id)
	if err != nil {
		return err
	}
	_, err = OrderPlacedTopic.Publish(ctx, &OrderPlaced{OrderID: id, Email: p.Email})
	return err
}

-- order/migrations/1_create_orders.up.sql
CREATE TABLE orders (
    id BIGSERIAL PRIMARY KEY,
    email TEXT NOT NULL
);

// receipt/receipt.go
package receipt

import (
	"context"
	"time"

	"encore.app/order"
	"encore.dev/pubsub"
	"encore.dev/rlog"
)

var _ = pubsub.NewSubscription(order.OrderPlacedTopic, "send-receipt",
	pubsub.SubscriptionConfig[*order.OrderPlaced]{
		Handler: SendReceipt,
		RetryPolicy: &pubsub.RetryPolicy{
			MinBackoff: 1 * time.Second,
			MaxRetries: 5,
		},
	},
)

func SendReceipt(ctx context.Context, e *order.OrderPlaced) error {
	rlog.Info("sending receipt", "order_id", e.OrderID)
	return nil
}

encore run runs both services with an in-memory Pub/Sub, and the local development dashboard traces each message from publish to handler. A handler that returns an error is retried, and after MaxRetries the message goes to the subscription's dead-letter queue, as covered in the Go Pub/Sub docs. The same model exists in TypeScript. Encore provisions an IAM role for the service, so no queue URLs or access keys sit in config.

Encore Pub/Sub is a queue and fan-out system, and it does not keep history for replay. When an AI coding agent adds a new consumer, it writes one more NewSubscription call and tests it locally, and the SQS queue and dead-letter queue for it arrive with the deploy.

Best fit: Go and TypeScript teams who need queues and fan-out between services on AWS or GCP with retries and dead-letter queues, declared in their service code.

Amazon SQS

Amazon SQS is a fully managed queue with no servers or partitions to plan. A consumer polls with long polling, receives up to 10 messages, and deletes each after processing. While a message is being processed it is hidden for the visibility timeout, and if the consumer does not delete it in time it becomes visible again and is retried. A redrive policy moves messages to a dead-letter queue after a set number of receives. Messages can be up to 1 MiB and are kept for up to 14 days.

Standard queues deliver at least once with best-effort ordering and scale with little planning. FIFO queues add ordering within a message group and deduplication, with lower throughput limits. SQS has no fan-out of its own, so publishing one event to several consumers means an SNS topic in front of one queue per consumer.

Good to know

Once a message is deleted it is gone, so SQS cannot rebuild a read model or feed a new consumer with past events. Teams that need that keep an event table in their database or move to a log.

Apache Kafka

Apache Kafka stores events in topics split into partitions, each an ordered, replicated log on the brokers. Producers append, and consumers in a consumer group divide the partitions between them and commit offsets. Events stay until a time or size retention limit, so a new consumer group can read from the beginning, and a broken consumer can be rewound and replayed. Ordering holds within a partition, and throughput scales with partitions and brokers into millions of events per second.

The ecosystem is Kafka's other strength: Kafka Connect moves data to and from databases and warehouses, and Kafka Streams and Flink process streams. Kafka 4.x runs only in KRaft mode without ZooKeeper, and Kafka 4.2 made share groups production-ready, which adds queue-style consumption with per-record acknowledgement.

Good to know

Kafka is a cluster you run: brokers and controllers, partition counts that are hard to reduce later, replication settings, upgrades and monitoring of consumer lag. Amazon MSK runs the brokers on AWS, and the partition planning, client configuration and dead-letter handling stay with you.

Which one to use

Pick SQS for job queues and decoupling services when each message is processed once and then discarded, which covers emails, webhooks, uploads and most background work. Pick Kafka when events are a shared record that several systems read at their own pace, when you need replay, or when throughput is sustained and high enough to justify a cluster. Running both is common, with Kafka for analytics and change data capture and a queue for application jobs.

Pick Encore when you want queues and fan-out between Go or TypeScript services, provisioned as SNS and SQS in your own AWS account from the code that publishes and handles the messages.

Related guides

• Queueing: an interactive study of queueing strategies
• Event-driven microservices on AWS: Encore vs DIY
• Message queues vs Pub/Sub
• Add Pub/Sub to a Go API
• Set up a queue on AWS with an AI agent