Amazon SNS with Terraform: Topics, SQS Subscriptions, Filters and Dead-Letter Queues
Terraform for an encrypted SNS topic with a publisher-only topic policy, one SQS queue per consumer with body filter policies, subscription dead-letter queues and the IAM a publisher needs, plus how Encore provisions SNS and SQS from topics declared in code.
An SNS topic is one resource in Terraform, and a topic that services can rely on takes a dozen more: encryption, a policy that says who may publish, a queue per consumer, a policy on each queue so SNS can write to it, filter policies, and a place for messages that never reached their queue. Each consumer added later repeats most of that list, and mistakes such as a missing queue policy or a filter on the wrong scope show up as messages that silently do not arrive.
This guide builds that setup with the AWS provider 6.x, using a payments topic with three consumers, and then shows how Encore provisions SNS and SQS from topics and subscriptions declared in TypeScript.
What an SNS Terraform setup needs
An encrypted topic with a publisher-only policy
terraform {
required_version = ">= 1.6"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
provider "aws" {
region = "eu-central-1"
}
data "aws_caller_identity" "current" {}
variable "publisher_role_arn" {
type = string
description = "IAM role of the service that publishes payment events"
}
# Customer managed key for the topic. Publishers get kms:GenerateDataKey* and
# kms:Decrypt through their IAM policy, which the root statement allows.
resource "aws_kms_key" "events" {
description = "SNS payment events"
enable_key_rotation = true
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Sid = "AccountAdmin"
Effect = "Allow"
Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
Action = "kms:*"
Resource = "*"
}]
})
}
resource "aws_sns_topic" "payments" {
name = "payment-events"
kms_master_key_id = aws_kms_key.events.arn
}
# Only the payments service may publish, and only over TLS.
resource "aws_sns_topic_policy" "payments" {
arn = aws_sns_topic.payments.arn
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "PublisherOnly"
Effect = "Allow"
Principal = { AWS = var.publisher_role_arn }
Action = "sns:Publish"
Resource = aws_sns_topic.payments.arn
},
{
Sid = "DenyInsecureTransport"
Effect = "Deny"
Principal = "*"
Action = "sns:Publish"
Resource = aws_sns_topic.payments.arn
Condition = { Bool = { "aws:SecureTransport" = "false" } }
}
]
})
}
Publishing to a topic encrypted with a KMS key requires kms:GenerateDataKey* and kms:Decrypt on that key, per the SNS key management docs. The root statement lets IAM policies in the account grant those to your own publishers, while AWS services that publish to the topic, such as CloudWatch alarms or EventBridge rules, need their service principal added to the key policy itself. The topic policy replaces the default one, so the explicit Allow for the publisher role is the only way in, and the Deny rejects any publish that does not use TLS.
One queue per consumer, with filters and a delivery DLQ
# One queue per consumer, each with an optional filter on the message body.
locals {
consumers = {
receipts = { filter = { type = ["payment.succeeded"] } }
risk = { filter = { type = ["payment.failed", "payment.disputed"] } }
ledger = { filter = null } # receives every event
}
}
resource "aws_sqs_queue" "consumer_dlq" {
for_each = local.consumers
name = "${each.key}-dlq"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
}
resource "aws_sqs_queue" "consumer" {
for_each = local.consumers
name = each.key
visibility_timeout_seconds = 60
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.consumer_dlq[each.key].arn
maxReceiveCount = 5
})
}
# Messages SNS could not deliver to a queue at all (for example a broken queue
# policy) land here, separate from messages a consumer failed to process.
resource "aws_sqs_queue" "sns_undeliverable" {
name = "payment-events-undeliverable"
message_retention_seconds = 1209600
sqs_managed_sse_enabled = true
}
resource "aws_sns_topic_subscription" "consumer" {
for_each = local.consumers
topic_arn = aws_sns_topic.payments.arn
protocol = "sqs"
endpoint = aws_sqs_queue.consumer[each.key].arn
raw_message_delivery = true
filter_policy_scope = each.value.filter == null ? null : "MessageBody"
filter_policy = each.value.filter == null ? null : jsonencode(each.value.filter)
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.sns_undeliverable.arn
})
}
Each consumer queue has an SQS redrive policy for messages its consumer received and failed to process. The subscription's own redrive_policy catches a different failure: messages SNS could not deliver to the queue at all, which SNS otherwise discards after client-side errors such as a changed queue policy, as the SNS dead-letter queue docs explain. That queue must be in the same account and region as the subscription.
filter_policy_scope = "MessageBody" makes the filter match fields in the JSON body. Leaving it out applies the same filter to message attributes, the default scope according to the filter policy scope docs, and a body-style filter on the attribute scope matches nothing. raw_message_delivery sends the published JSON to the queue as is instead of wrapping it in the SNS envelope, and changing it later breaks every consumer that parses the other format.
Queue policies and publisher IAM
# SNS needs permission to write to every queue it delivers to, the
# undeliverable queue included.
locals {
sns_targets = merge(
{ for k, q in aws_sqs_queue.consumer : k => q },
{ undeliverable = aws_sqs_queue.sns_undeliverable },
)
}
resource "aws_sqs_queue_policy" "from_sns" {
for_each = local.sns_targets
queue_url = each.value.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "sns.amazonaws.com" }
Action = "sqs:SendMessage"
Resource = each.value.arn
Condition = { ArnEquals = { "aws:SourceArn" = aws_sns_topic.payments.arn } }
}]
})
}
# Attach to the publisher's role: publish, plus the KMS calls an encrypted topic needs.
data "aws_iam_policy_document" "publish" {
statement {
actions = ["sns:Publish"]
resources = [aws_sns_topic.payments.arn]
}
statement {
actions = ["kms:GenerateDataKey*", "kms:Decrypt"]
resources = [aws_kms_key.events.arn]
}
}
output "publisher_policy_json" {
value = data.aws_iam_policy_document.publish.json
}
The queue policies cover the undeliverable queue too, as SNS writes to it on behalf of the subscription. The whole configuration passes terraform validate with Terraform 1.13 and the AWS provider 6.x. Consumers then need their own role with sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes on their queue, and the topic ARN and queue URLs reach each service through environment variables per environment.
What still takes work after the first apply
- A new consumer means a new entry in
local.consumers, a new IAM role, and a Terraform apply before the consumer code can run, usually in a separate pull request. - Filter changes take effect in SNS, and the code that reads the queue has no record of which events it was meant to receive.
- Alarms on both kinds of dead-letter queue have to be added separately, or messages sit there until someone looks.
- Local development needs an emulator or a shared AWS account, so filter policies and redrive behavior are first tested after deploy.
How does Encore provision SNS and SQS from code?
Encore takes the infrastructure-from-code route for TypeScript and Go: a topic is declared in the service that publishes it and a subscription in the service that consumes it, and when you deploy to AWS the Encore platform creates the SNS topic, an SQS queue with a dead-letter queue for each subscription, and the IAM permissions for publishing and consuming services, as listed in the AWS infrastructure docs. The publish and consume grants follow from which service publishes and which one subscribes.
Set up the app
Run in your terminal to get started locally.
The payments service declares the topic, and its webhook handler publishes with await paymentEvents.publish({ ... }):
// payments/events.ts
import { Topic } from "encore.dev/pubsub";
export interface PaymentEvent {
paymentId: string;
type: "payment.succeeded" | "payment.failed" | "payment.disputed";
customerEmail: string;
amountCents: number;
}
// An SNS topic on AWS; each subscription gets its own SQS queue and DLQ.
export const paymentEvents = new Topic<PaymentEvent>("payment-events", {
deliveryGuarantee: "at-least-once",
});
Each service directory has an encore.service.ts file, which is how Encore knows receipts is a service of its own with its own permissions:
// receipts/encore.service.ts
import { Service } from "encore.dev/service";
export default new Service("receipts");
The receipts consumer does what the receipts filter policy did in Terraform, and claims each payment before sending so a redelivered event is a no-op:
// receipts/receipts.ts
import { Subscription } from "encore.dev/pubsub";
import { secret } from "encore.dev/config";
import { SQLDatabase } from "encore.dev/storage/sqldb";
import { paymentEvents } from "../payments/events";
const db = new SQLDatabase("receipts", { migrations: "./migrations" });
const mailApiKey = secret("MailApiKey");
const _ = new Subscription(paymentEvents, "send-receipt", {
handler: async (event) => {
// The filter policy from the Terraform version, as code you can test.
if (event.type !== "payment.succeeded") return;
// Claim the payment first, so a redelivered event sends nothing twice.
const claimed = await db.queryRow<{ payment_id: string }>`
INSERT INTO receipts_sent (payment_id) VALUES (${event.paymentId})
ON CONFLICT (payment_id) DO NOTHING
RETURNING payment_id
`;
if (!claimed) return;
const res = await fetch("https://mail.example.com/v1/send", {
method: "POST",
headers: { Authorization: `Bearer ${mailApiKey()}`, "Content-Type": "application/json" },
body: JSON.stringify({ to: event.customerEmail, template: "receipt", paymentId: event.paymentId }),
});
if (!res.ok) {
// Release the claim so the retry can send it.
await db.exec`DELETE FROM receipts_sent WHERE payment_id = ${event.paymentId}`;
throw new Error(`mail provider returned ${res.status}`);
}
},
retryPolicy: { maxRetries: 8, minBackoff: "30s", maxBackoff: "10m" },
});
The claim table is one migration, applied locally by encore run and to RDS on deploy:
-- receipts/migrations/1_create_receipts_sent.up.sql
CREATE TABLE receipts_sent (
payment_id TEXT PRIMARY KEY,
sent_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
The risk and ledger consumers are each another new Subscription in their own service, and each gets its own queue and dead-letter queue. Routing that lived in filter_policy is an if in the handler, covered by the same tests as the rest of it, and the Pub/Sub docs cover retry policies, maxConcurrency and ordered topics. encore run delivers events locally, so a coding agent that adds a consumer can publish a test payment and read the handler's trace before opening a pull request. The mail key is set per environment with encore secret set and is not stored in the repository, per the secrets docs, and a topic that other systems already publish to can be imported.
Best fit: TypeScript and Go teams on AWS who want SNS fan-out with a queue and dead-letter queue per consumer, created in their own account from the code that publishes and consumes.
Good to know
- Renaming a queue or topic in Terraform destroys and recreates it, and messages still in the old queue are lost.
- FIFO topics need a
.fifoname suffix andfifo_topic = true, and the dead-letter queue for a FIFO subscription must also be a FIFO queue. - Large payloads go through the extended client libraries, which store the body in S3 and publish a pointer, as the large payload docs describe.
Related guides
Frequently asked questions
How do I subscribe an SQS queue to an SNS topic in Terraform?
Create an aws_sns_topic_subscription with protocol = "sqs" and the queue ARN as endpoint, then add an aws_sqs_queue_policy that allows the sns.amazonaws.com principal to call sqs:SendMessage on the queue with an aws:SourceArn condition matching the topic ARN. Without the queue policy, the subscription exists but messages never arrive.
What is the difference between an SNS dead-letter queue and an SQS dead-letter queue?
An SNS dead-letter queue is set with redrive_policy on the subscription and receives messages SNS could not deliver to the endpoint, for example after the queue policy was changed. An SQS dead-letter queue is set with redrive_policy on the queue and receives messages a consumer received but failed to process after maxReceiveCount attempts.
How do I filter SNS messages by fields in the message body with Terraform?
Set filter_policy_scope = "MessageBody" on the aws_sns_topic_subscription and pass the filter as JSON in filter_policy, for example jsonencode({ type = ["payment.succeeded"] }). Without filter_policy_scope, the filter applies to message attributes, which is the default.
How do I create an SNS topic with SQS subscribers from TypeScript code?
With Encore, declare new Topic<PaymentEvent>("payment-events", { deliveryGuarantee: "at-least-once" }) and add consumers with new Subscription(paymentEvents, "send-receipt", { handler, retryPolicy }). On AWS, Encore provisions the SNS topic, an SQS queue and a dead-letter queue for each subscription, and the IAM permissions for the services that publish and consume.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.
