The claim check pattern stores a large payload in object storage and sends only a small reference through the queue. When message size limits force it, how to design the reference, clean up payloads, and what to do about retries, with a Go example on Encore.

A queue works best with small messages: when an event has to carry a 40 MB video, a large CSV or a rendered report, putting the bytes in the message makes the broker slow and expensive, and eventually unable to accept it.
The claim check pattern stores the payload in object storage and publishes only a reference. This guide covers when to use it, how to design the reference and the cleanup, and a Go version with a bucket and a topic.
The producer writes the payload to storage and receives a key. It publishes a message with that key, plus the metadata a consumer needs to route it. The consumer receives the message, reads the object, processes it and acknowledges. The Azure Architecture Center describes the same split, with the message bus carrying the check and storage holding the item.
The queue then handles what it is good at, ordering, retries and fan-out, while storage handles size and durability. Only one party ever needs write access to the payload and only the consumers need read access.
Limits differ per broker, and they are the usual reason for the pattern. SQS accepts messages up to 1 MiB and Google Cloud Pub/Sub up to 10 MB. Even under the limit, large messages count against throughput and cost, and some consumers load whole messages into memory, so switching to references at a few hundred kilobytes is common practice.
Keep the reference small and stable: bucket and key, and usually a content type, size and checksum. A key built from a unique ID, such as incoming/2026-11-20/<uuid>.json, avoids collisions and makes lifecycle rules by prefix possible. The checksum lets the consumer detect a truncated upload. Do not put presigned URLs in messages, since they expire before a delayed retry can use them.
Brokers deliver at least once, so the consumer may fetch the same object twice and the handler must be idempotent, as described in idempotent message consumers. That also means the consumer should not delete the payload on first success if replays matter. A bucket lifecycle rule that expires the incoming/ prefix after a few days is simpler and survives dead-letter replays.
Write the object before publishing: if the publish fails, the object is orphaned and expires with the rest, which is harmless. The reverse order produces messages that point at nothing.
Encore is a backend framework with built-in infrastructure, so the bucket and the Pub/Sub topic are declared in the Go service that uses them. On deploy they become an S3 bucket and SNS with SQS in your AWS account, or Cloud Storage and Pub/Sub on GCP, and encore run provides both locally.
Run in your terminal to get started locally.
// reports/reports.go
package reports
import (
"context"
"fmt"
"io"
"encore.dev/pubsub"
"encore.dev/rlog"
"encore.dev/storage/objects"
"github.com/google/uuid"
)
var payloads = objects.NewBucket("payloads", objects.BucketConfig{})
type PayloadReady struct {
Key string `json:"key"`
ContentType string `json:"content_type"`
Size int64 `json:"size"`
}
var Ready = pubsub.NewTopic[*PayloadReady]("payload-ready", pubsub.TopicConfig{
DeliveryGuarantee: pubsub.AtLeastOnce,
})
// Publish stores the payload and sends only the reference.
func Publish(ctx context.Context, data []byte, contentType string) (string, error) {
key := "incoming/" + uuid.NewString()
w := payloads.Upload(ctx, key, objects.WithUploadAttrs(objects.UploadAttrs{ContentType: contentType}))
if _, err := w.Write(data); err != nil {
w.Abort(err)
return "", err
}
if err := w.Close(); err != nil {
return "", err
}
_, err := Ready.Publish(ctx, &PayloadReady{Key: key, ContentType: contentType, Size: int64(len(data))})
return key, err
}
var _ = pubsub.NewSubscription(Ready, "process-payload", pubsub.SubscriptionConfig[*PayloadReady]{
Handler: func(ctx context.Context, m *PayloadReady) error {
r := payloads.Download(ctx, m.Key)
defer r.Close()
data, err := io.ReadAll(r)
if err != nil {
return fmt.Errorf("fetch %s: %w", m.Key, err)
}
rlog.Info("processing payload", "key", m.Key, "bytes", len(data))
// Process data here. Reading the same key twice is safe.
return nil
},
})
The message carries a key of about 50 bytes whatever the payload size. Add a lifecycle rule on the bucket in your cloud account or delete keys in the handler once replays no longer matter. Each publish and consume shows up in a trace, which makes it easy to see which key a failing delivery was holding.
A coding agent can add the bucket, the topic and the subscriber in the same change as the handler logic, and run the full round trip locally with no cloud console work.
Best fit: Teams moving large files or documents between Go services who want references on the queue and payloads in object storage in their own AWS or GCP account.
The producer stores the large payload in external storage such as an S3 or Cloud Storage bucket and publishes a small message containing only a reference to it. The consumer reads the reference, fetches the payload from storage, and processes it, the same way a coat check hands back an item for a ticket.
Use it when payloads exceed or approach the broker's message size limit, or when large messages make the queue slow and expensive to operate. Pub/Sub accepts messages up to 10 MB and SQS up to 1 MiB, and switching to references well below those limits keeps the queue lean.
Either the consumer deletes the object after successful processing, or a lifecycle rule on the bucket expires objects after a fixed period. A lifecycle rule is safer, since a retry or a replay from a dead-letter queue still finds the payload for a few days after the first attempt.
Yes. A Go service declares an object storage bucket and a Pub/Sub topic, uploads the payload to the bucket and publishes the object key. Encore provisions the bucket and topic in your AWS or GCP account, and both run locally under encore run.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.