Nov 20, 20266 min read

Use Amazon SNS from Python

Publish messages with boto3, add message attributes and batch publishes, fan out to SQS queues with filter policies, read SNS envelopes in a consumer, and test the whole flow locally with moto.

Illustration for Use Amazon SNS from Python

Amazon SNS is how a Python backend on AWS announces that an order was placed, so several consumers react without the publisher knowing about them. Publishing is one boto3 call, and the effort goes into attributes for filtering, partly failed batches, the envelope SQS consumers receive and the queue policies behind fan-out.

This guide covers each part with boto3, including tests that run without AWS.

SNS in Python at a glance

Taskboto3 callWatch out for
Publish one messagesns.publish(TopicArn=..., Message=...)Message is a string
Publish manysns.publish_batch(...)10 entries per call, check Failed
Route to some consumersMessageAttributes plus a filter policyFilters match attributes by default, not the body
Fan out to queuessubscribe(Protocol="sqs", ...)The queue policy must allow SNS
Read in a consumerParse Body, then MessageOr enable raw message delivery
Ordering and deduplicationFIFO topic with MessageGroupIdEach message also needs a deduplication ID or content-based deduplication

How do you publish messages with boto3?

Create one client per process. It resolves credentials through the boto3 credential chain, and the topic ARN usually comes from configuration because the topic is created outside the application:

import json import uuid import boto3 sns = boto3.client("sns", region_name="eu-west-1") TOPIC_ARN = "arn:aws:sns:eu-west-1:123456789012:order-events" def publish_order_placed(order_id: str, total_cents: int, country: str) -> str: resp = sns.publish( TopicArn=TOPIC_ARN, Message=json.dumps({"order_id": order_id, "total_cents": total_cents}), MessageAttributes={ "event_type": {"DataType": "String", "StringValue": "order.placed"}, "country": {"DataType": "String", "StringValue": country}, }, ) return resp["MessageId"] def publish_many(orders: list[dict]) -> list[str]: ids = [] for start in range(0, len(orders), 10): # PublishBatch takes at most 10 entries chunk = orders[start : start + 10] resp = sns.publish_batch( TopicArn=TOPIC_ARN, PublishBatchRequestEntries=[ {"Id": str(uuid.uuid4()), "Message": json.dumps(o)} for o in chunk ], ) if resp.get("Failed"): raise RuntimeError(f"{len(resp['Failed'])} messages failed: {resp['Failed'][0]['Code']}") ids += [s["MessageId"] for s in resp["Successful"]] return ids

Messages are limited to 256 KiB by default, and since September 2026 a topic's MaximumMessageSize attribute can raise that to 1 MiB, with the PublishBatch API applying the same limit to the whole batch. A partly failed batch returns normally with entries in Failed, which is why the helper raises instead of trusting the HTTP status. For a FIFO topic, every publish also needs a MessageGroupId, plus a MessageDeduplicationId unless content-based deduplication is enabled on the topic.

How do you fan out to SQS queues?

Fan-out means one SQS queue per consumer, each subscribed to the topic. Two pieces of configuration trip people up. The queue needs a resource policy that allows sns.amazonaws.com to call sqs:SendMessage with a condition on the topic ARN, and without it the subscription is confirmed but nothing arrives. A consumer that only wants some events gets a filter policy on its subscription:

sns.subscribe( TopicArn=TOPIC_ARN, Protocol="sqs", Endpoint="arn:aws:sqs:eu-west-1:123456789012:dach-fulfilment", Attributes={ "FilterPolicy": json.dumps({"country": ["DE", "AT", "CH"]}), "RawMessageDelivery": "true", }, ReturnSubscriptionArn=True, )

Filter policies match message attributes unless you set FilterPolicyScope to MessageBody, so a publisher that forgets the country attribute silently reaches no filtered subscriber.

How does a consumer read SNS messages?

Without raw delivery, SNS wraps the payload in an envelope with Type, MessageId, TopicArn, Message and MessageAttributes, so the consumer parses twice:

import json import boto3 sqs = boto3.client("sqs", region_name="eu-west-1") def handle(event: dict) -> None: print("order placed", event["order_id"]) def poll(queue_url: str) -> None: resp = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=20) for msg in resp.get("Messages", []): envelope = json.loads(msg["Body"]) # SNS wraps the payload unless raw delivery is on handle(json.loads(envelope["Message"])) sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg["ReceiptHandle"])

SNS to SQS delivery is at least once, so handle has to tolerate the same order twice, for example by inserting the order ID into a table with a unique constraint before acting.

Which IAM permissions are needed?

The publisher's role needs sns:Publish on the topic, which also covers PublishBatch. The consumer needs sqs:ReceiveMessage and sqs:DeleteMessage on its queue. If the topic uses a customer managed KMS key, the SNS service principal needs kms:GenerateDataKey and kms:Decrypt on that key, and an encrypted queue needs the same grant, which the SNS encryption docs spell out.

How do you test it locally?

moto mocks SNS and SQS together, including subscriptions and filter policies, so a test can publish and read the queue in-process:

import json import boto3 from moto import mock_aws import orders_events @mock_aws def test_filtered_fan_out(): sns = boto3.client("sns", region_name="eu-west-1") sqs = boto3.client("sqs", region_name="eu-west-1") orders_events.sns = sns topic_arn = sns.create_topic(Name="order-events")["TopicArn"] queue_url = sqs.create_queue(QueueName="dach")["QueueUrl"] queue_arn = sqs.get_queue_attributes(QueueUrl=queue_url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"] sns.subscribe(TopicArn=topic_arn, Protocol="sqs", Endpoint=queue_arn, Attributes={"FilterPolicy": json.dumps({"country": ["DE", "AT", "CH"]})}) orders_events.publish_order_placed("o1", 4200, "DE") orders_events.publish_order_placed("o2", 1800, "US") msgs = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)["Messages"] assert [json.loads(json.loads(m["Body"])["Message"])["order_id"] for m in msgs] == ["o1"]

The test assumes the publisher module is orders_events.py and that its TOPIC_ARN matches the mocked topic, which it does here because moto uses account 123456789012.

Topics declared in Python with Encore

Encore

Encore works alongside your Python framework, here FastAPI, as an SDK that declares infrastructure in code, and a Pub/Sub topic with its subscriptions is two declarations. Encore for Python is in private beta, and you can request an invite. On AWS each topic becomes an SNS topic and each subscription its own SQS queue, so the fan-out above, queue policies included, comes from the code.

Application code passes through Encore and infrastructure guardrails to resources provisioned in your own AWS or GCP account.
Encore provisions the infrastructure your application needs in your own AWS or GCP account.
# orders/main.py import encore from encore import pubsub from encore.storage import postgres from fastapi import FastAPI from pydantic import BaseModel db = postgres.Database("orders", migrations="./migrations") class OrderPlaced(BaseModel): order_id: str total_cents: int country: str # SNS on AWS, GCP Pub/Sub on GCP, a local broker under `encore run`. order_events = pubsub.Topic[OrderPlaced]("order-events") async def send_receipt(message: pubsub.Message[OrderPlaced]) -> None: # Delivery is at least once: the unique key turns a repeat into a no-op. result = await db.exec( "INSERT INTO receipts (order_id) VALUES ($1) ON CONFLICT DO NOTHING", message.data.order_id, ) if result.rows_affected == 0: return print("sending receipt for", message.data.order_id) receipts = pubsub.Subscription[OrderPlaced](order_events, "send-receipt", handler=send_receipt) app = FastAPI() @app.post("/orders") async def place_order(order: OrderPlaced) -> dict: message_id = await order_events.publish(order) return {"message_id": message_id} if __name__ == "__main__": encore.run(app)

encore run starts the database and a local broker, applies the migration that creates the receipts table, and delivers each published order to send_receipt, with the publish and the handler in one trace. The service is registered with an encore.service.json file next to the code, and the SDK needs Python 3.9 or later. On deploy to your own AWS account, Encore creates the topic, the queue and its dead-letter queue, as listed in the AWS infrastructure docs, and provisions an IAM role for the service.

When a coding agent needs a second consumer, say a fraud check on large orders, it adds another Subscription in the same change as the handler and watches both receive the event locally, with no queue policy or subscription ARN to wire up by hand.

Request a private-beta invite to try this setup:

Try Encore →

Frequently asked questions

How do I publish a message to SNS in Python?

Create a client with boto3.client("sns") and call publish(TopicArn=..., Message=...). Message is a string, so serialize structured data with json.dumps, and put routing data such as an event type in MessageAttributes so subscription filter policies can match on it.

How many messages can publish_batch send at once?

Up to 10 entries per call, and the combined size of the batch must fit within the topic's MaximumMessageSize, which defaults to 256 KiB and can be raised to 1 MiB. The response lists Successful and Failed entries separately, and a partly failed batch does not raise an exception, so check the Failed list and retry those entries.

Why does my SQS consumer receive JSON with a Message field?

SNS wraps each message in a JSON envelope with Type, MessageId, TopicArn, Message and MessageAttributes fields when it delivers to SQS. Parse the body, then parse the Message field, or enable RawMessageDelivery on the subscription to receive the original payload unchanged.

Can I run SNS locally for development?

No. There is no official local SNS. Tests usually use moto, which mocks SNS and SQS in-process, or LocalStack in a container. Encore takes a different route, where encore run serves declared topics and subscriptions from a local broker, so no AWS emulator is involved. Encore for Python is in private beta (sign up at /py).

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders