How Amazon S3 event notifications work, how to choose between SQS, SNS, Lambda and EventBridge, the queue, topic and function permissions each needs, Terraform for S3 to SQS to Lambda, and how to write consumers that handle duplicates, plus how Encore handles upload events.

S3 event notifications are how a bucket tells the rest of your system that something happened, usually that a file finished uploading and a thumbnail, virus scan or CSV import should start. The configuration is a few lines, and most of the effort goes into what surrounds it: a queue or topic policy that lets S3 write, a KMS key S3 is allowed to use, and a consumer that copes with events arriving twice, late or out of order.
This guide covers how notifications work, which destination to pick, the permissions each one needs, a Terraform setup for S3 to SQS to Lambda, an idempotent Go consumer, and how Encore handles upload events.
Destinations and limits are from the S3 event notification guide.
A bucket has one notification configuration, a list of rules that each name event types, an optional prefix or suffix filter and one destination. When a matching event happens, S3 sends a JSON message with a Records array describing the bucket, the object key, size, ETag, version and a sequencer. Event types are grouped, so s3:ObjectCreated:* covers Put, Post, Copy and CompleteMultipartUpload, and s3:ObjectRemoved:* covers deletes and delete markers.
Delivery is at least once, typically within seconds but sometimes a minute or longer, and S3 does not guarantee that events arrive in the order they happened, as the ordering and duplicates notes explain. For PUT and DELETE events on the same key, the later event has the larger sequencer, which lets a consumer ignore an older event that arrives after a newer one. Object keys in the message are URL-encoded, with spaces as +, so red flower.jpg arrives as red+flower.jpg.
Two rules for the same event type cannot have overlapping prefixes or suffixes, so uploads/ and uploads/images/ for ObjectCreated is rejected. Writing the configuration replaces the whole thing, so a script that adds one rule with put-bucket-notification-configuration silently deletes the others. And a consumer that writes back into the same prefix it listens on triggers itself in a loop, so outputs go to a different prefix or bucket.
S3 needs permission to send to the queue, scoped to your bucket and account so other buckets cannot use it:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "Service": "s3.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:123456789012:uploads-events",
"Condition": {
"ArnEquals": { "aws:SourceArn": "arn:aws:s3:::acme-uploads-prod" },
"StringEquals": { "aws:SourceAccount": "123456789012" }
}
}]
}
Save it as queue-policy.json, attach it to the queue, then add the notification rule:
S3 validates the destination when you save the configuration, so a missing or wrong queue policy fails right there with Unable to validate the following destination configurations. The permissions guide has the matching SNS topic policy, which allows SNS:Publish with the same two conditions.
If the queue or topic is encrypted with SSE-KMS, the key has to be a customer managed key whose key policy lets the s3.amazonaws.com principal call kms:GenerateDataKey and kms:Decrypt; the AWS managed aws/sqs key cannot be changed to allow that, so events never arrive. SQS-managed encryption (SSE-SQS) works without any key policy.
Putting SQS between the bucket and Lambda gives you batching, a retry budget per message and a dead-letter queue to inspect, which direct Lambda invocation does not:
# s3_notifications.tf
variable "lambda_function_arn" { type = string } # the Go handler above
resource "aws_s3_bucket" "uploads" {
bucket = "acme-uploads-prod"
}
resource "aws_sqs_queue" "uploads_dlq" {
name = "uploads-events-dlq"
message_retention_seconds = 1209600 # 14 days
}
resource "aws_sqs_queue" "uploads" {
name = "uploads-events"
visibility_timeout_seconds = 180 # at least 6x the Lambda timeout (30s)
sqs_managed_sse_enabled = true
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.uploads_dlq.arn
maxReceiveCount = 5
})
}
# Let this bucket, in this account, send to the queue.
resource "aws_sqs_queue_policy" "uploads" {
queue_url = aws_sqs_queue.uploads.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "s3.amazonaws.com" }
Action = "sqs:SendMessage"
Resource = aws_sqs_queue.uploads.arn
Condition = {
ArnEquals = { "aws:SourceArn" = aws_s3_bucket.uploads.arn }
StringEquals = { "aws:SourceAccount" = data.aws_caller_identity.current.account_id }
}
}]
})
}
data "aws_caller_identity" "current" {}
# Replaces the bucket's whole notification configuration.
resource "aws_s3_bucket_notification" "uploads" {
bucket = aws_s3_bucket.uploads.id
queue {
queue_arn = aws_sqs_queue.uploads.arn
events = ["s3:ObjectCreated:*"]
filter_prefix = "incoming/"
}
# S3 checks it can write to the queue when the configuration is saved.
depends_on = [aws_sqs_queue_policy.uploads]
}
resource "aws_lambda_event_source_mapping" "uploads" {
event_source_arn = aws_sqs_queue.uploads.arn
function_name = var.lambda_function_arn
batch_size = 10
function_response_types = ["ReportBatchItemFailures"]
}
The configuration validates with Terraform 1.13 and AWS provider 6.73, and the depends_on matters: without it Terraform can save the notification before the queue policy exists, and S3 rejects it. With ReportBatchItemFailures, the handler below tells Lambda which messages failed, so one bad file does not send the other nine back to the queue. The SQS dead-letter queue guide covers redrive once something lands in the DLQ.
Duplicates are rare but guaranteed to happen eventually, and a retried Lambda batch or a redelivered SQS message adds more. This Go handler, built with aws-lambda-go and the AWS SDK for Go v2, decodes the key, skips events it has finished before, and records completion only after the work succeeds:
// main.go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"net/url"
"os"
"strconv"
"time"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
var (
ddb *dynamodb.Client
table = os.Getenv("PROCESSED_TABLE") // partition key "pk" (string), TTL on "expires_at"
)
func handler(ctx context.Context, ev events.SQSEvent) (events.SQSEventResponse, error) {
var resp events.SQSEventResponse
for _, msg := range ev.Records {
if err := handleMessage(ctx, msg.Body); err != nil {
log.Printf("message %s: %v", msg.MessageId, err)
// Only this message is retried; the rest of the batch is deleted.
resp.BatchItemFailures = append(resp.BatchItemFailures,
events.SQSBatchItemFailure{ItemIdentifier: msg.MessageId})
}
}
return resp, nil
}
func handleMessage(ctx context.Context, body string) error {
// S3 sends s3:TestEvent when the notification is configured; it has no Records.
var s3ev events.S3Event
if err := json.Unmarshal([]byte(body), &s3ev); err != nil {
return fmt.Errorf("decode: %w", err)
}
for _, r := range s3ev.Records {
// Keys arrive URL-encoded, with spaces as "+".
key, err := url.QueryUnescape(r.S3.Object.Key)
if err != nil {
return fmt.Errorf("unescape key: %w", err)
}
// Bucket, key and sequencer identify one event on one object. The
// sequencer is set for PUT and DELETE events; fall back to version and ETag.
id := r.S3.Object.Sequencer
if id == "" {
id = r.S3.Object.VersionID + ":" + r.S3.Object.ETag
}
pk := r.S3.Bucket.Name + "/" + key + "#" + id
done, err := alreadyProcessed(ctx, pk)
if err != nil {
return err
}
if done {
continue // duplicate delivery
}
if err := process(ctx, r.S3.Bucket.Name, key, r.S3.Object.Size); err != nil {
return err
}
if err := markProcessed(ctx, pk); err != nil {
return err
}
}
return nil
}
func alreadyProcessed(ctx context.Context, pk string) (bool, error) {
out, err := ddb.GetItem(ctx, &dynamodb.GetItemInput{
TableName: aws.String(table),
Key: map[string]types.AttributeValue{"pk": &types.AttributeValueMemberS{Value: pk}},
ConsistentRead: aws.Bool(true),
})
if err != nil {
return false, fmt.Errorf("check %s: %w", pk, err)
}
return out.Item != nil, nil
}
func markProcessed(ctx context.Context, pk string) error {
expires := time.Now().Add(7 * 24 * time.Hour).Unix()
_, err := ddb.PutItem(ctx, &dynamodb.PutItemInput{
TableName: aws.String(table),
Item: map[string]types.AttributeValue{
"pk": &types.AttributeValueMemberS{Value: pk},
"expires_at": &types.AttributeValueMemberN{Value: strconv.FormatInt(expires, 10)},
},
})
return err
}
// process must be safe to run twice for the same object, for example by
// writing its output to a key derived from the input key.
func process(ctx context.Context, bucket, key string, size int64) error {
log.Printf("processing s3://%s/%s (%d bytes)", bucket, key, size)
return nil
}
func main() {
cfg, err := config.LoadDefaultConfig(context.Background())
if err != nil {
log.Fatal(err)
}
ddb = dynamodb.NewFromConfig(cfg)
lambda.Start(handler)
}
Check, process, record is the order that keeps files from being lost: recording first and processing second loses the file for good if processing then fails, since the retry sees the record and skips it. Recording last means two concurrent deliveries of the same event can both process it, which is why the work itself writes to output keys derived from the input key, so a second run overwrites the first with the same result.
Direct invocation skips the queue: S3 invokes the function asynchronously, and the function's resource policy has to allow it, scoped to the bucket and account:
Asynchronous invocations retry twice on error and then drop the event unless the function has an on-failure destination or DLQ configured, which is the main reason to prefer the SQS path for anything you cannot afford to lose.
EventBridge is switched on per bucket rather than per rule. Once on, every event from the bucket goes to the default event bus, and rules decide what goes where:
That put-bucket-notification-configuration call also replaces any SQS, SNS or Lambda rules, so include them in the same document if you keep both. EventBridge events use a different shape from direct notifications, with the bucket and object under detail instead of a Records array, so a consumer written for one does not parse the other. Rules can target SQS FIFO queues, Step Functions and several targets at once, and the bus can archive and replay events, as covered in the S3 EventBridge guide.
Encore gives you infrastructure from code for both halves of this setup, a bucket and a topic with subscriptions, but it does not configure S3 event notifications. On AWS a declared bucket is an S3 bucket in your account and a topic is SNS with an SQS queue and dead-letter queue per subscription, as the AWS infrastructure docs describe, and the event comes from your API rather than from S3: the endpoint that handed out the signed upload URL checks the object exists and publishes a message.
Run in your terminal to get started locally.
// uploads/encore.service.ts
import { Service } from "encore.dev/service";
export default new Service("uploads");
// uploads/uploads.ts
import { api, APIError, Gateway, Header } from "encore.dev/api";
import { authHandler } from "encore.dev/auth";
import { getAuthData } from "~encore/auth";
import { Bucket, ObjectNotFound } from "encore.dev/storage/objects";
import { SQLDatabase } from "encore.dev/storage/sqldb";
import { Subscription, Topic } from "encore.dev/pubsub";
import log from "encore.dev/log";
import { createHash, randomUUID } from "node:crypto";
// An S3 bucket on AWS, a GCS bucket on GCP, local storage under encore run.
export const uploads = new Bucket("uploads", { versioned: false });
const db = new SQLDatabase("uploads", { migrations: "./migrations" });
const MAX_BYTES = 20 * 1024 * 1024;
interface AuthParams {
authorization: Header<"Authorization">;
}
interface AuthData {
userID: string;
}
export const sessionAuth = authHandler<AuthParams, AuthData>(async ({ authorization }) => {
const token = authorization.replace(/^Bearer\s+/i, "");
if (!token) throw APIError.unauthenticated("missing token");
const row = await db.queryRow<{ user_id: string }>`
SELECT user_id FROM sessions
WHERE token_hash = ${createHash("sha256").update(token).digest("hex")}
AND expires_at > now()`;
if (!row) throw APIError.unauthenticated("invalid token");
return { userID: row.user_id };
});
export const gateway = new Gateway({ authHandler: sessionAuth });
export interface UploadCompleted {
key: string;
etag: string;
size: number;
}
// An SNS topic on AWS; each subscription gets its own SQS queue and DLQ.
export const uploadCompleted = new Topic<UploadCompleted>("upload-completed", {
deliveryGuarantee: "at-least-once",
});
// 1. The signed-in client asks for a URL and PUTs the file straight to the bucket.
export const start = api(
{ expose: true, auth: true, method: "POST", path: "/uploads" },
async (): Promise<{ id: string; url: string }> => {
const { userID } = getAuthData()!;
const open = await db.queryRow<{ n: number }>`
SELECT count(*)::int AS n FROM upload_slots
WHERE user_id = ${userID} AND status = 'pending'
AND created_at > now() - interval '1 hour'`;
if ((open?.n ?? 0) >= 20) throw APIError.resourceExhausted("too many pending uploads");
const id = randomUUID();
const key = `incoming/${userID}/${id}`;
await db.exec`
INSERT INTO upload_slots (id, user_id, object_key) VALUES (${id}, ${userID}, ${key})`;
const { url } = await uploads.signedUploadUrl(key, { ttl: 900 });
return { id, url };
},
);
// 2. The client reports the upload; the API checks the object and emits the event once.
export const complete = api(
{ expose: true, auth: true, method: "POST", path: "/uploads/:id/complete" },
async ({ id }: { id: string }): Promise<UploadCompleted> => {
const { userID } = getAuthData()!;
const slot = await db.queryRow<{ object_key: string }>`
SELECT object_key FROM upload_slots WHERE id = ${id} AND user_id = ${userID}`;
if (!slot) throw APIError.notFound("upload not found");
try {
const attrs = await uploads.attrs(slot.object_key);
if (attrs.size > MAX_BYTES) {
await uploads.remove(slot.object_key);
throw APIError.invalidArgument("file too large");
}
const event = { key: slot.object_key, etag: attrs.etag, size: attrs.size };
await using tx = await db.begin();
const first = await tx.queryRow<{ id: string }>`
UPDATE upload_slots SET status = 'completed'
WHERE id = ${id} AND user_id = ${userID} AND status = 'pending'
RETURNING id`;
if (first) {
await tx.exec`
INSERT INTO upload_outbox (id, key, etag, size)
VALUES (${id}, ${event.key}, ${event.etag}, ${event.size})
ON CONFLICT (id) DO NOTHING`;
}
await tx.commit();
await flushUploadOutbox(id);
return event;
} catch (err) {
if (err instanceof ObjectNotFound) throw APIError.notFound("upload not found");
throw err;
}
},
);
async function flushUploadOutbox(id: string) {
const row = await db.queryRow<{ key: string; etag: string; size: number }>`
SELECT key, etag, size FROM upload_outbox WHERE id = ${id}`;
if (!row) return;
await uploadCompleted.publish({ key: row.key, etag: row.etag, size: row.size });
await db.exec`DELETE FROM upload_outbox WHERE id = ${id}`;
}
// 3. Processing runs once per object version, however often the event arrives.
const _ = new Subscription(uploadCompleted, "scan-upload", {
handler: async ({ key, etag, size }) => {
const seen = await db.queryRow`
SELECT 1 AS hit FROM processed_uploads WHERE key = ${key} AND etag = ${etag}
`;
if (seen) return; // duplicate delivery or a repeated /complete call
log.info("processing upload", { key, size });
// ... scan, resize or parse; write outputs to keys derived from `key`
await db.exec`
INSERT INTO processed_uploads (key, etag) VALUES (${key}, ${etag})
ON CONFLICT DO NOTHING
`;
},
ackDeadline: "60s",
retryPolicy: { minBackoff: "10s", maxBackoff: "5m", maxRetries: 5 },
});
-- uploads/migrations/1_create_uploads.up.sql
CREATE TABLE upload_slots (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
object_key TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'completed')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE processed_uploads (
key TEXT NOT NULL,
etag TEXT NOT NULL,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (key, etag)
);
CREATE TABLE upload_outbox (
id TEXT PRIMARY KEY REFERENCES upload_slots (id),
key TEXT NOT NULL,
etag TEXT NOT NULL,
size BIGINT NOT NULL
);
encore run gives you a local bucket, a local Postgres and local Pub/Sub, so the whole flow runs on a laptop, and the local development dashboard shows the publish and the subscriber's trace. On AWS the subscriber's SQS queue has a dead-letter queue, and the retryPolicy and ackDeadline above set the retry backoff and how long a delivery may run, as the Pub/Sub docs explain. The subscriber follows the same check, process, record order as the Lambda handler, with an Encore Postgres database in place of DynamoDB.
Publishing from the API instead of from S3 changes two things. The event only fires for uploads your app knows about, after it has checked the object, so there is no s3:TestEvent, no URL-encoded keys and no stray objects from other writers, and the message carries whatever your app needs, such as the user who uploaded it. In exchange, a client that uploads and never calls complete produces no event, which a periodic sweep or an expiry on the incoming/ prefix handles. When other systems write to a bucket directly and you need to react to their objects, S3 event notifications on that bucket, set up as above, are still the tool.
Encore provisions an IAM role for the service.
Best fit: teams on AWS that want upload processing with retries and dead-letter queues declared next to the code that handles the files.
Add a queue policy that allows the s3.amazonaws.com principal to call sqs:SendMessage with aws:SourceArn set to the bucket ARN and aws:SourceAccount to your account, then run aws s3api put-bucket-notification-configuration with a QueueConfigurations entry listing the queue ARN, the events such as s3:ObjectCreated:* and an optional prefix or suffix filter. The queue must be a standard queue in the bucket's region.
S3 delivers event notifications at least once, not exactly once, usually within seconds but sometimes after a minute or longer, and not necessarily in the order the events happened. Consumers should deduplicate on the bucket, key and sequencer, and compare sequencers to order events for the same key.
Direct notifications to SQS, SNS or Lambda are simplest for one or two consumers with prefix and suffix filters. EventBridge receives every event from the bucket once enabled, supports richer filters on key, size and metadata, many targets per rule, archive and replay, and can deliver to SQS FIFO queues, which direct notifications cannot.
Encore provisions S3 buckets and SNS topics with SQS subscriptions from code, but it does not set up bucket notifications. The Encore pattern is for the API that handed out the signed upload URL to check the object and publish a Pub/Sub event, which an idempotent subscriber processes with retries and a dead-letter queue.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.