Kinesis vs SQS: Event Stream or Message Queue on AWS?

This comparison usually comes up when one service has to tell several others that something happened, and Kinesis is proposed as the more scalable option. Amazon Kinesis Data Streams and Amazon SQS both move messages between producers and consumers, but they are built on different models, and picking the stream for application events means managing shards, checkpoints and iterator ages for a workload that a queue would have handled.

This guide compares Kinesis and SQS on ordering, replay, consumers, retention and scaling, and shows how Encore covers the common case, ordered events fanned out to several consumers, on SNS and SQS from Go code.

Kinesis vs SQS at a glance

Kinesis numbers are from the Kinesis quotas page and SQS numbers from the SQS message quotas.

How does Encore handle event fan-out on AWS?

Encore brings infrastructure from code to messaging: you declare a topic and its subscriptions as Go values, and Encore provisions SNS topics and SQS queues in your own AWS account, one queue per subscription, with a dead-letter queue for each. That covers what Kinesis is often picked for in this case, several consumers each reading every event, along with per-key ordering, with no shards or checkpoint tables.

Set up the app

A document editor publishes a change event each time a document is saved. Search indexing and the audit trail consume the same events independently, and OrderingAttribute keeps changes to one document in order for each of them:

// documents/events.go
package documents

import (
	"context"
	"time"

	"encore.dev/pubsub"
	"encore.dev/rlog"
)

type DocumentChanged struct {
	DocumentID string `pubsub-attr:"document_id"`
	Version    int64  `json:"version"`
	Editor     string `json:"editor"`
	Action     string `json:"action"` // "created", "edited", "deleted"
}

// SNS + SQS on AWS, Google Pub/Sub on GCP. Messages with the same
// document_id reach each subscriber in the order they were published.
var DocumentChanges = pubsub.NewTopic[*DocumentChanged]("document-changes", pubsub.TopicConfig{
	DeliveryGuarantee: pubsub.AtLeastOnce,
	OrderingAttribute: "document_id",
})

type PublishParams struct {
	DocumentID string `json:"document_id"`
	Version    int64  `json:"version"`
	Editor     string `json:"editor"`
	Action     string `json:"action"`
}

// Record publishes a change. Called by the editor service after it saves.
//
//encore:api private method=POST path=/documents/changes
func Record(ctx context.Context, p *PublishParams) error {
	_, err := DocumentChanges.Publish(ctx, &DocumentChanged{
		DocumentID: p.DocumentID, Version: p.Version, Editor: p.Editor, Action: p.Action,
	})
	return err
}

// Each subscription gets its own queue, so search indexing and the audit
// trail consume every change independently, like two stream consumers.
var _ = pubsub.NewSubscription(DocumentChanges, "search-index", pubsub.SubscriptionConfig[*DocumentChanged]{
	Handler: func(ctx context.Context, ev *DocumentChanged) error {
		// Upserts keyed by document ID and version are safe to repeat.
		rlog.Info("index document", "id", ev.DocumentID, "version", ev.Version, "action", ev.Action)
		return nil
	},
	AckDeadline: 30 * time.Second,
})

var _ = pubsub.NewSubscription(DocumentChanges, "audit-trail", pubsub.SubscriptionConfig[*DocumentChanged]{
	Handler: func(ctx context.Context, ev *DocumentChanged) error {
		// Insert with ON CONFLICT (document_id, version) DO NOTHING in a real store.
		rlog.Info("audit", "id", ev.DocumentID, "version", ev.Version, "editor", ev.Editor)
		return nil
	},
	RetryPolicy: &pubsub.RetryPolicy{MaxRetries: 10},
})

Under encore run the topic runs in memory, and calling the endpoint delivers the event to both subscribers, which log the change in the local run. Ordering has no effect locally, so test order-dependent logic against a deployed environment. On deploy, Encore creates the SNS topic and SQS queues with the permissions the service needs, and ordered topics are limited to 300 messages per second on AWS, as the Pub/Sub docs explain, which covers application events but not a firehose of telemetry.

Adding a third consumer, say a notification service, is one more pubsub.NewSubscription call, and it starts with its own queue at the next deploy. A coding agent adding a consumer writes a typed handler it can run against local events right away.

Best fit: application events that several services consume, with retries, dead-lettering and optional per-key ordering, on SNS and SQS in your own AWS account.

Amazon Kinesis Data Streams

Kinesis is a durable, ordered log where producers write records with a partition key, records with the same key land on the same shard in order, and consumers read from a position they track themselves, so a new consumer can start from the oldest retained record and a buggy one can rewind and reprocess:

package main

import (
	"context"
	"encoding/json"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/kinesis"
)

type DocumentChanged struct {
	DocumentID string `json:"document_id"`
	Version    int64  `json:"version"`
}

func main() {
	ctx := context.Background()
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		log.Fatal(err)
	}
	client := kinesis.NewFromConfig(cfg)

	data, _ := json.Marshal(DocumentChanged{DocumentID: "d1", Version: 2})
	// Records with the same partition key go to the same shard, in order.
	_, err = client.PutRecord(ctx, &kinesis.PutRecordInput{
		StreamName:   aws.String("document-changes"),
		PartitionKey: aws.String("d1"),
		Data:         data,
	})
	if err != nil {
		log.Fatal(err)
	}
}

Each provisioned shard takes up to 1,000 records or 1 MB per second of writes, and on-demand streams start at 4 MB/s of writes and scale from there, with records up to 10 MiB and retention from 24 hours to 365 days. Consumers share 2 MB/s of reads per shard unless you register enhanced fan-out consumers, which get their own throughput, up to 20 per stream in the standard modes. Reading a stream reliably means using the Kinesis Client Library, which is Java-first and stores shard leases and checkpoints in DynamoDB, or Lambda event source mappings. A record that keeps failing blocks its shard until you skip it or route it elsewhere, since Kinesis has no per-record dead-letter queue of its own.

Best fit: high-volume streams such as clickstreams, telemetry and change data capture, where several consumers read everything and replay matters.

Amazon SQS

SQS is a managed queue with almost nothing to size: standard queues scale with traffic, each message is handled by one consumer and deleted afterwards, and a failed message reappears after the visibility timeout until a redrive policy moves it to a dead-letter queue. FIFO queues add ordering per message group and deduplication, at lower throughput per partition. Messages can be up to 1 MiB and stay in the queue for up to 14 days.

SQS has no fan-out or replay of its own, so two services that both need every event each need their own queue subscribed to an SNS topic, and once a message is deleted it is gone, so rebuilding a search index means reading from the source of truth rather than rewinding the queue. For background jobs and service-to-service events, that model fits.

Best fit: work queues and background jobs, and fan-out when paired with SNS.

How to choose

Pick Encore when the job is application events that several services consume, with retries, dead-letter queues and per-key ordering declared in Go and provisioned on SNS and SQS in your own AWS account. Pick Kinesis when you need replay, retention beyond 14 days or ordered throughput beyond what queues offer, for telemetry and analytics pipelines. Pick plain SQS for single-consumer work queues.

Related guides

• SNS vs SQS
• SQS vs Kafka
• SQS vs EventBridge
• Kafka vs RabbitMQ
• IoT telemetry ingestion backend