Nov 17, 202614 min read

S3 Event Notifications: Send Bucket Events to SQS, SNS, Lambda or EventBridge

How Amazon S3 event notifications work, how to choose between SQS, SNS, Lambda and EventBridge, the queue, topic and function permissions each needs, Terraform for S3 to SQS to Lambda, and how to write consumers that handle duplicates, plus how Encore handles upload events.

Illustration for S3 Event Notifications: Send Bucket Events to SQS, SNS, Lambda or EventBridge

S3 event notifications are how a bucket tells the rest of your system that something happened, usually that a file finished uploading and a thumbnail, virus scan or CSV import should start. The configuration is a few lines, and most of the effort goes into what surrounds it: a queue or topic policy that lets S3 write, a KMS key S3 is allowed to use, and a consumer that copes with events arriving twice, late or out of order.

This guide covers how notifications work, which destination to pick, the permissions each one needs, a Terraform setup for S3 to SQS to Lambda, an idempotent Go consumer, and how Encore handles upload events.

S3 event notification destinations at a glance

SQSSNSLambdaEventBridge
Best forOne worker pool, bufferingFan-out to several queuesSmall, direct handlersMany rules, rich filters
FilteringPrefix and suffixPrefix and suffixPrefix and suffixKey, size, metadata, event type
PermissionQueue policyTopic policyFunction resource policyTurn on per bucket
FIFONot supportedNot supportedn/aCan target SQS FIFO
RetriesQueue redrive to DLQSubscription retry and DLQAsync retries, failure destinationTarget retries, DLQ
RegionSame as bucketSame as bucketSame as bucketSame as bucket

Destinations and limits are from the S3 event notification guide.

How S3 event notifications work

A bucket has one notification configuration, a list of rules that each name event types, an optional prefix or suffix filter and one destination. When a matching event happens, S3 sends a JSON message with a Records array describing the bucket, the object key, size, ETag, version and a sequencer. Event types are grouped, so s3:ObjectCreated:* covers Put, Post, Copy and CompleteMultipartUpload, and s3:ObjectRemoved:* covers deletes and delete markers.

Delivery is at least once, typically within seconds but sometimes a minute or longer, and S3 does not guarantee that events arrive in the order they happened, as the ordering and duplicates notes explain. For PUT and DELETE events on the same key, the later event has the larger sequencer, which lets a consumer ignore an older event that arrives after a newer one. Object keys in the message are URL-encoded, with spaces as +, so red flower.jpg arrives as red+flower.jpg.

Two rules for the same event type cannot have overlapping prefixes or suffixes, so uploads/ and uploads/images/ for ObjectCreated is rejected. Writing the configuration replaces the whole thing, so a script that adds one rule with put-bucket-notification-configuration silently deletes the others. And a consumer that writes back into the same prefix it listens on triggers itself in a loop, so outputs go to a different prefix or bucket.

Set up S3 to SQS with the AWS CLI

S3 needs permission to send to the queue, scoped to your bucket and account so other buckets cannot use it:

{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "sqs:SendMessage", "Resource": "arn:aws:sqs:us-east-1:123456789012:uploads-events", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::acme-uploads-prod" }, "StringEquals": { "aws:SourceAccount": "123456789012" } } }] }

Save it as queue-policy.json, attach it to the queue, then add the notification rule:

~
$ QUEUE_URL=$(aws sqs create-queue --queue-name uploads-events --query QueueUrl --output text)$ aws sqs set-queue-attributes --queue-url "$QUEUE_URL" \ --attributes "$(jq -c '{Policy: tostring}' queue-policy.json)" # Replaces any existing notification configuration on the bucket$ aws s3api put-bucket-notification-configuration --bucket acme-uploads-prod \ --notification-configuration '{ "QueueConfigurations": [{ "Id": "incoming-to-queue", "QueueArn": "arn:aws:sqs:us-east-1:123456789012:uploads-events", "Events": ["s3:ObjectCreated:*"], "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "incoming/"}]}} }] }' # S3 sends an s3:TestEvent to the queue when the configuration is saved$ aws sqs receive-message --queue-url $QUEUE_URL --query 'Messages[0].Body'

S3 validates the destination when you save the configuration, so a missing or wrong queue policy fails right there with Unable to validate the following destination configurations. The permissions guide has the matching SNS topic policy, which allows SNS:Publish with the same two conditions.

If the queue or topic is encrypted with SSE-KMS, the key has to be a customer managed key whose key policy lets the s3.amazonaws.com principal call kms:GenerateDataKey and kms:Decrypt; the AWS managed aws/sqs key cannot be changed to allow that, so events never arrive. SQS-managed encryption (SSE-SQS) works without any key policy.

S3 to SQS to Lambda with Terraform

Putting SQS between the bucket and Lambda gives you batching, a retry budget per message and a dead-letter queue to inspect, which direct Lambda invocation does not:

# s3_notifications.tf variable "lambda_function_arn" { type = string } # the Go handler above resource "aws_s3_bucket" "uploads" { bucket = "acme-uploads-prod" } resource "aws_sqs_queue" "uploads_dlq" { name = "uploads-events-dlq" message_retention_seconds = 1209600 # 14 days } resource "aws_sqs_queue" "uploads" { name = "uploads-events" visibility_timeout_seconds = 180 # at least 6x the Lambda timeout (30s) sqs_managed_sse_enabled = true redrive_policy = jsonencode({ deadLetterTargetArn = aws_sqs_queue.uploads_dlq.arn maxReceiveCount = 5 }) } # Let this bucket, in this account, send to the queue. resource "aws_sqs_queue_policy" "uploads" { queue_url = aws_sqs_queue.uploads.id policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Service = "s3.amazonaws.com" } Action = "sqs:SendMessage" Resource = aws_sqs_queue.uploads.arn Condition = { ArnEquals = { "aws:SourceArn" = aws_s3_bucket.uploads.arn } StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id } } }] }) } data "aws_caller_identity" "current" {} # Replaces the bucket's whole notification configuration. resource "aws_s3_bucket_notification" "uploads" { bucket = aws_s3_bucket.uploads.id queue { queue_arn = aws_sqs_queue.uploads.arn events = ["s3:ObjectCreated:*"] filter_prefix = "incoming/" } # S3 checks it can write to the queue when the configuration is saved. depends_on = [aws_sqs_queue_policy.uploads] } resource "aws_lambda_event_source_mapping" "uploads" { event_source_arn = aws_sqs_queue.uploads.arn function_name = var.lambda_function_arn batch_size = 10 function_response_types = ["ReportBatchItemFailures"] }

The configuration validates with Terraform 1.13 and AWS provider 6.73, and the depends_on matters: without it Terraform can save the notification before the queue policy exists, and S3 rejects it. With ReportBatchItemFailures, the handler below tells Lambda which messages failed, so one bad file does not send the other nine back to the queue. The SQS dead-letter queue guide covers redrive once something lands in the DLQ.

Write an idempotent consumer

Duplicates are rare but guaranteed to happen eventually, and a retried Lambda batch or a redelivered SQS message adds more. This Go handler, built with aws-lambda-go and the AWS SDK for Go v2, decodes the key, skips events it has finished before, and records completion only after the work succeeds:

// main.go package main import ( "context" "encoding/json" "fmt" "log" "net/url" "os" "strconv" "time" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/dynamodb" "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" ) var ( ddb *dynamodb.Client table = os.Getenv("PROCESSED_TABLE") // partition key "pk" (string), TTL on "expires_at" ) func handler(ctx context.Context, ev events.SQSEvent) (events.SQSEventResponse, error) { var resp events.SQSEventResponse for _, msg := range ev.Records { if err := handleMessage(ctx, msg.Body); err != nil { log.Printf("message %s: %v", msg.MessageId, err) // Only this message is retried; the rest of the batch is deleted. resp.BatchItemFailures = append(resp.BatchItemFailures, events.SQSBatchItemFailure{ItemIdentifier: msg.MessageId}) } } return resp, nil } func handleMessage(ctx context.Context, body string) error { // S3 sends s3:TestEvent when the notification is configured; it has no Records. var s3ev events.S3Event if err := json.Unmarshal([]byte(body), &s3ev); err != nil { return fmt.Errorf("decode: %w", err) } for _, r := range s3ev.Records { // Keys arrive URL-encoded, with spaces as "+". key, err := url.QueryUnescape(r.S3.Object.Key) if err != nil { return fmt.Errorf("unescape key: %w", err) } // Bucket, key and sequencer identify one event on one object. The // sequencer is set for PUT and DELETE events; fall back to version and ETag. id := r.S3.Object.Sequencer if id == "" { id = r.S3.Object.VersionID + ":" + r.S3.Object.ETag } pk := r.S3.Bucket.Name + "/" + key + "#" + id done, err := alreadyProcessed(ctx, pk) if err != nil { return err } if done { continue // duplicate delivery } if err := process(ctx, r.S3.Bucket.Name, key, r.S3.Object.Size); err != nil { return err } if err := markProcessed(ctx, pk); err != nil { return err } } return nil } func alreadyProcessed(ctx context.Context, pk string) (bool, error) { out, err := ddb.GetItem(ctx, &dynamodb.GetItemInput{ TableName: aws.String(table), Key: map[string]types.AttributeValue{"pk": &types.AttributeValueMemberS{Value: pk}}, ConsistentRead: aws.Bool(true), }) if err != nil { return false, fmt.Errorf("check %s: %w", pk, err) } return out.Item != nil, nil } func markProcessed(ctx context.Context, pk string) error { expires := time.Now().Add(7 * 24 * time.Hour).Unix() _, err := ddb.PutItem(ctx, &dynamodb.PutItemInput{ TableName: aws.String(table), Item: map[string]types.AttributeValue{ "pk": &types.AttributeValueMemberS{Value: pk}, "expires_at": &types.AttributeValueMemberN{Value: strconv.FormatInt(expires, 10)}, }, }) return err } // process must be safe to run twice for the same object, for example by // writing its output to a key derived from the input key. func process(ctx context.Context, bucket, key string, size int64) error { log.Printf("processing s3://%s/%s (%d bytes)", bucket, key, size) return nil } func main() { cfg, err := config.LoadDefaultConfig(context.Background()) if err != nil { log.Fatal(err) } ddb = dynamodb.NewFromConfig(cfg) lambda.Start(handler) }

Check, process, record is the order that keeps files from being lost: recording first and processing second loses the file for good if processing then fails, since the retry sees the record and skips it. Recording last means two concurrent deliveries of the same event can both process it, which is why the work itself writes to output keys derived from the input key, so a second run overwrites the first with the same result.

Send S3 events directly to Lambda

Direct invocation skips the queue: S3 invokes the function asynchronously, and the function's resource policy has to allow it, scoped to the bucket and account:

~
$ aws lambda add-permission --function-name scan-upload \ --statement-id s3-acme-uploads --action lambda:InvokeFunction \ --principal s3.amazonaws.com \ --source-arn arn:aws:s3:::acme-uploads-prod --source-account 123456789012 # Replaces any existing notification configuration; include other rules here if needed.$ aws s3api put-bucket-notification-configuration --bucket acme-uploads-prod \ --notification-configuration '{ "LambdaFunctionConfigurations": [{ "Id": "direct-scan-upload", "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:scan-upload", "Events": ["s3:ObjectCreated:*"] }] }'

Asynchronous invocations retry twice on error and then drop the event unless the function has an on-failure destination or DLQ configured, which is the main reason to prefer the SQS path for anything you cannot afford to lose.

Send S3 events to EventBridge

EventBridge is switched on per bucket rather than per rule. Once on, every event from the bucket goes to the default event bus, and rules decide what goes where:

~
$ aws s3api put-bucket-notification-configuration --bucket acme-uploads-prod \ --notification-configuration '{"EventBridgeConfiguration": {}}' $ aws events put-rule --name uploads-incoming-created --event-pattern '{ "source": ["aws.s3"], "detail-type": ["Object Created"], "detail": { "bucket": {"name": ["acme-uploads-prod"]}, "object": {"key": [{"prefix": "incoming/"}], "size": [{"numeric": [">", 0]}]} }}'

That put-bucket-notification-configuration call also replaces any SQS, SNS or Lambda rules, so include them in the same document if you keep both. EventBridge events use a different shape from direct notifications, with the bucket and object under detail instead of a Records array, so a consumer written for one does not parse the other. Rules can target SQS FIFO queues, Step Functions and several targets at once, and the bus can archive and replay events, as covered in the S3 EventBridge guide.

How Encore handles upload events

Encore

Encore gives you infrastructure from code for both halves of this setup, a bucket and a topic with subscriptions, but it does not configure S3 event notifications. On AWS a declared bucket is an S3 bucket in your account and a topic is SNS with an SQS queue and dead-letter queue per subscription, as the AWS infrastructure docs describe, and the event comes from your API rather than from S3: the endpoint that handed out the signed upload URL checks the object exists and publishes a message.

Set up the app

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

~
$ encore app create media --example=ts/hello-world$ cd media && mkdir -p uploads/migrations
// uploads/encore.service.ts import { Service } from "encore.dev/service"; export default new Service("uploads");
// uploads/uploads.ts import { api, APIError, Gateway, Header } from "encore.dev/api"; import { authHandler } from "encore.dev/auth"; import { getAuthData } from "~encore/auth"; import { Bucket, ObjectNotFound } from "encore.dev/storage/objects"; import { SQLDatabase } from "encore.dev/storage/sqldb"; import { Subscription, Topic } from "encore.dev/pubsub"; import log from "encore.dev/log"; import { createHash, randomUUID } from "node:crypto"; // An S3 bucket on AWS, a GCS bucket on GCP, local storage under encore run. export const uploads = new Bucket("uploads", { versioned: false }); const db = new SQLDatabase("uploads", { migrations: "./migrations" }); const MAX_BYTES = 20 * 1024 * 1024; interface AuthParams { authorization: Header<"Authorization">; } interface AuthData { userID: string; } export const sessionAuth = authHandler<AuthParams, AuthData>(async ({ authorization }) => { const token = authorization.replace(/^Bearer\s+/i, ""); if (!token) throw APIError.unauthenticated("missing token"); const row = await db.queryRow<{ user_id: string }>` SELECT user_id FROM sessions WHERE token_hash = ${createHash("sha256").update(token).digest("hex")} AND expires_at > now()`; if (!row) throw APIError.unauthenticated("invalid token"); return { userID: row.user_id }; }); export const gateway = new Gateway({ authHandler: sessionAuth }); export interface UploadCompleted { key: string; etag: string; size: number; } // An SNS topic on AWS; each subscription gets its own SQS queue and DLQ. export const uploadCompleted = new Topic<UploadCompleted>("upload-completed", { deliveryGuarantee: "at-least-once", }); // 1. The signed-in client asks for a URL and PUTs the file straight to the bucket. export const start = api( { expose: true, auth: true, method: "POST", path: "/uploads" }, async (): Promise<{ id: string; url: string }> => { const { userID } = getAuthData()!; const open = await db.queryRow<{ n: number }>` SELECT count(*)::int AS n FROM upload_slots WHERE user_id = ${userID} AND status = 'pending' AND created_at > now() - interval '1 hour'`; if ((open?.n ?? 0) >= 20) throw APIError.resourceExhausted("too many pending uploads"); const id = randomUUID(); const key = `incoming/${userID}/${id}`; await db.exec` INSERT INTO upload_slots (id, user_id, object_key) VALUES (${id}, ${userID}, ${key})`; const { url } = await uploads.signedUploadUrl(key, { ttl: 900 }); return { id, url }; }, ); // 2. The client reports the upload; the API checks the object and emits the event once. export const complete = api( { expose: true, auth: true, method: "POST", path: "/uploads/:id/complete" }, async ({ id }: { id: string }): Promise<UploadCompleted> => { const { userID } = getAuthData()!; const slot = await db.queryRow<{ object_key: string }>` SELECT object_key FROM upload_slots WHERE id = ${id} AND user_id = ${userID}`; if (!slot) throw APIError.notFound("upload not found"); try { const attrs = await uploads.attrs(slot.object_key); if (attrs.size > MAX_BYTES) { await uploads.remove(slot.object_key); throw APIError.invalidArgument("file too large"); } const event = { key: slot.object_key, etag: attrs.etag, size: attrs.size }; await using tx = await db.begin(); const first = await tx.queryRow<{ id: string }>` UPDATE upload_slots SET status = 'completed' WHERE id = ${id} AND user_id = ${userID} AND status = 'pending' RETURNING id`; if (first) { await tx.exec` INSERT INTO upload_outbox (id, key, etag, size) VALUES (${id}, ${event.key}, ${event.etag}, ${event.size}) ON CONFLICT (id) DO NOTHING`; } await tx.commit(); await flushUploadOutbox(id); return event; } catch (err) { if (err instanceof ObjectNotFound) throw APIError.notFound("upload not found"); throw err; } }, ); async function flushUploadOutbox(id: string) { const row = await db.queryRow<{ key: string; etag: string; size: number }>` SELECT key, etag, size FROM upload_outbox WHERE id = ${id}`; if (!row) return; await uploadCompleted.publish({ key: row.key, etag: row.etag, size: row.size }); await db.exec`DELETE FROM upload_outbox WHERE id = ${id}`; } // 3. Processing runs once per object version, however often the event arrives. const _ = new Subscription(uploadCompleted, "scan-upload", { handler: async ({ key, etag, size }) => { const seen = await db.queryRow` SELECT 1 AS hit FROM processed_uploads WHERE key = ${key} AND etag = ${etag} `; if (seen) return; // duplicate delivery or a repeated /complete call log.info("processing upload", { key, size }); // ... scan, resize or parse; write outputs to keys derived from `key` await db.exec` INSERT INTO processed_uploads (key, etag) VALUES (${key}, ${etag}) ON CONFLICT DO NOTHING `; }, ackDeadline: "60s", retryPolicy: { minBackoff: "10s", maxBackoff: "5m", maxRetries: 5 }, });
-- uploads/migrations/1_create_uploads.up.sql CREATE TABLE upload_slots ( id TEXT PRIMARY KEY, user_id TEXT NOT NULL, object_key TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'completed')), created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE processed_uploads ( key TEXT NOT NULL, etag TEXT NOT NULL, processed_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (key, etag) ); CREATE TABLE upload_outbox ( id TEXT PRIMARY KEY REFERENCES upload_slots (id), key TEXT NOT NULL, etag TEXT NOT NULL, size BIGINT NOT NULL );
~/media
$ encore run$ curl -X POST localhost:4000/uploads \ -H "Authorization: Bearer <session token>"{"id":"6f1c...","url":"http://localhost:..."}$ curl -X PUT --data-binary @report.csv "<url from above>"$ curl -X POST localhost:4000/uploads/6f1c.../complete \ -H "Authorization: Bearer <session token>"{"key":"incoming/<user-id>/6f1c...","etag":"...","size":18234}

encore run gives you a local bucket, a local Postgres and local Pub/Sub, so the whole flow runs on a laptop, and the local development dashboard shows the publish and the subscriber's trace. On AWS the subscriber's SQS queue has a dead-letter queue, and the retryPolicy and ackDeadline above set the retry backoff and how long a delivery may run, as the Pub/Sub docs explain. The subscriber follows the same check, process, record order as the Lambda handler, with an Encore Postgres database in place of DynamoDB.

Publishing from the API instead of from S3 changes two things. The event only fires for uploads your app knows about, after it has checked the object, so there is no s3:TestEvent, no URL-encoded keys and no stray objects from other writers, and the message carries whatever your app needs, such as the user who uploaded it. In exchange, a client that uploads and never calls complete produces no event, which a periodic sweep or an expiry on the incoming/ prefix handles. When other systems write to a bucket directly and you need to react to their objects, S3 event notifications on that bucket, set up as above, are still the tool.

Encore provisions an IAM role for the service.

Best fit: teams on AWS that want upload processing with retries and dead-letter queues declared next to the code that handles the files.

Frequently asked questions

How do I send S3 events to an SQS queue?

Add a queue policy that allows the s3.amazonaws.com principal to call sqs:SendMessage with aws:SourceArn set to the bucket ARN and aws:SourceAccount to your account, then run aws s3api put-bucket-notification-configuration with a QueueConfigurations entry listing the queue ARN, the events such as s3:ObjectCreated:* and an optional prefix or suffix filter. The queue must be a standard queue in the bucket's region.

Are S3 event notifications delivered exactly once?

S3 delivers event notifications at least once, not exactly once, usually within seconds but sometimes after a minute or longer, and not necessarily in the order the events happened. Consumers should deduplicate on the bucket, key and sequencer, and compare sequencers to order events for the same key.

Should I use S3 event notifications or EventBridge?

Direct notifications to SQS, SNS or Lambda are simplest for one or two consumers with prefix and suffix filters. EventBridge receives every event from the bucket once enabled, supports richer filters on key, size and metadata, many targets per rule, archive and replay, and can deliver to SQS FIFO queues, which direct notifications cannot.

Does Encore configure S3 event notifications?

Encore provisions S3 buckets and SNS topics with SQS subscriptions from code, but it does not set up bucket notifications. The Encore pattern is for the API that handed out the signed upload URL to check the object and publish a Pub/Sub event, which an idempotent subscriber processes with retries and a dead-letter queue.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders