Nov 16, 20268 min read

Pub/Sub Dead-Letter Topics on Google Cloud: Setup, IAM and Replay

How dead-letter topics work in Google Cloud Pub/Sub, the IAM bindings they need, how to choose max delivery attempts and a retry policy, how to keep and replay dead-lettered messages, and how Encore sets them up per subscription.

Illustration for Pub/Sub Dead-Letter Topics on Google Cloud: Setup, IAM and Replay

A Pub/Sub subscription without a dead-letter topic retries a failing message until it expires from the subscription, seven days by default, and with the default retry policy it redelivers immediately each time. One malformed message can then burn CPU for a week and hide real failures in the noise. A dead-letter topic caps the attempts and moves the message somewhere you can inspect and replay it.

This guide covers how dead-letter topics work in Google Cloud Pub/Sub, the IAM bindings they need, choosing delivery attempts and backoff, keeping and replaying dead-lettered messages, and how Encore sets them up for every subscription declared in code.

Pub/Sub dead-letter topics at a glance

SettingValueNotes
Configured onThe subscriptionEach subscription can use its own dead-letter topic
Max delivery attempts5 to 100Approximate, forwarded on a best-effort basis
IAMService agent: publisher on the DLT, subscriber on the sourceAttempts are only counted with both in place
Retry policyImmediate by default, or exponential backoffBackoff between 0 and 600 seconds, defaults 10s and 600s
Added attributesCloudPubSubDeadLetterSource*Source subscription, project and delivery count
RetentionNeeds a subscription on the DLTMessages to a topic with no subscription are dropped
Encore on GCPDead-letter topic per subscriptionAttempts and backoff from RetryPolicy

Values are from Google Cloud's dead-letter topics guide and retry policy guide.

How dead-letter topics work

Pub/Sub tracks delivery attempts per message on a subscription. A message counts as failed when the subscriber nacks it, lets its ack deadline expire, or, for a push subscription, returns a non-2xx status. Once the attempts pass max_delivery_attempts, Pub/Sub wraps the message and publishes it to the dead-letter topic with attributes including CloudPubSubDeadLetterSourceSubscription and CloudPubSubDeadLetterSourceDeliveryCount, then acks it on the source subscription.

The count is approximate: Google notes that Pub/Sub may forward a message after fewer attempts than configured or try a few more times first, and that the tracked count can reset to zero, especially for pull subscriptions with inactive subscribers, so a handler can see a message more times than the limit. The count is also exposed to subscribers as the delivery attempt on each message, which is useful for logging and for treating the last attempt differently.

IAM: the step that silently breaks it

Forwarding is done by the Pub/Sub service agent of the project, service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com, not by your code. It needs roles/pubsub.publisher on the dead-letter topic to publish there and roles/pubsub.subscriber on the source subscription to ack the original. Without them, the subscription accepts the configuration but Pub/Sub does not count delivery attempts, so nothing is ever dead-lettered and the message keeps retrying. The Cloud console flags this in the subscription's details panel, and Terraform has to grant both explicitly.

Set up a dead-letter topic with Terraform

A source topic, a dead-letter topic with a hold subscription so forwarded messages are kept, a source subscription with exponential backoff and five attempts, and the two IAM bindings:

# main.tf terraform { required_providers { google = { source = "hashicorp/google" version = ">= 7.0" } } } variable "project_id" { type = string } data "google_project" "this" { project_id = var.project_id } locals { pubsub_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com" } resource "google_pubsub_topic" "orders" { project = var.project_id name = "order-placed" } resource "google_pubsub_topic" "orders_dlt" { project = var.project_id name = "order-placed-dlt" } # Without a subscription, messages forwarded to the dead-letter topic are dropped. resource "google_pubsub_subscription" "orders_dlt_hold" { project = var.project_id name = "order-placed-dlt-hold" topic = google_pubsub_topic.orders_dlt.id message_retention_duration = "604800s" ack_deadline_seconds = 60 expiration_policy { ttl = "" } } resource "google_pubsub_subscription" "send_receipt" { project = var.project_id name = "send-receipt" topic = google_pubsub_topic.orders.id ack_deadline_seconds = 60 retry_policy { minimum_backoff = "10s" maximum_backoff = "600s" } dead_letter_policy { dead_letter_topic = google_pubsub_topic.orders_dlt.id max_delivery_attempts = 5 } } # The Pub/Sub service agent forwards failed messages, so it needs to publish to # the dead-letter topic and to acknowledge them on the source subscription. resource "google_pubsub_topic_iam_member" "dlt_publisher" { project = var.project_id topic = google_pubsub_topic.orders_dlt.name role = "roles/pubsub.publisher" member = local.pubsub_agent } resource "google_pubsub_subscription_iam_member" "source_subscriber" { project = var.project_id subscription = google_pubsub_subscription.send_receipt.name role = "roles/pubsub.subscriber" member = local.pubsub_agent }

terraform validate passes on this with the Google provider, and the hold subscription has an empty expiration_policy so it does not expire after 31 days without subscribers, which is the default for subscriptions. Seven days of retention gives the team a week to notice before dead-lettered messages expire.

Choosing attempts and backoff

The default retry policy redelivers immediately, which suits short blips and is the wrong default for a downstream outage: every message retries as fast as the subscriber can fail it. Exponential backoff spreads attempts out between the minimum and maximum, and together with the attempt limit sets how long a message has before it is dead-lettered. With a 10 second minimum, a 600 second maximum and five attempts, a message gets a few minutes of retries, enough for a deploy or a short provider outage, before it moves to the DLT.

Code should also separate failures that can pass from ones that never will. A message with an invalid email address will fail on every attempt, so the handler should record it and ack it instead of spending five attempts on it, leaving the dead-letter topic for failures that need a fix and a replay.

How does Encore set up dead-letter topics on GCP?

Encore

Encore brings infrastructure from code to Google Cloud: a Pub/Sub topic and its subscriptions are declarations in Go or TypeScript, and when the app deploys to your GCP project Encore creates the topics and subscriptions and configures dead-letter topics for them, as listed in the GCP infrastructure docs. Encore provisions an IAM role for the service, or a service account on GCP.

Set up the app

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

~
$ encore app create shop --example=hello-world$ cd shop && mkdir receipts
// receipts/receipts.go package receipts import ( "context" "net/mail" "time" "encore.dev/pubsub" "encore.dev/rlog" ) type OrderPlaced struct { OrderID string `json:"order_id"` Email string `json:"email"` TotalCents int64 `json:"total_cents"` } // On GCP this is a Pub/Sub topic, and each subscription gets a dead-letter topic. var Orders = pubsub.NewTopic[*OrderPlaced]("order-placed", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var _ = pubsub.NewSubscription(Orders, "send-receipt", pubsub.SubscriptionConfig[*OrderPlaced]{ Handler: sendReceipt, AckDeadline: 60 * time.Second, RetryPolicy: &pubsub.RetryPolicy{ MinBackoff: 10 * time.Second, MaxBackoff: 10 * time.Minute, MaxRetries: 5, }, }) func sendReceipt(ctx context.Context, o *OrderPlaced) error { // A message that can never succeed: record it and ack, so it does not // burn five attempts and land in the dead-letter topic with the real failures. if _, err := mail.ParseAddress(o.Email); err != nil { rlog.Warn("receipt skipped: invalid email", "order", o.OrderID, "email", o.Email) return nil } // A failure that may pass: return it, and Encore retries with backoff // until MaxRetries, then the message moves to the dead-letter topic. if err := deliver(ctx, o); err != nil { rlog.Warn("receipt failed, will retry", "order", o.OrderID, "err", err) return err } rlog.Info("receipt sent", "order", o.OrderID) return nil } // deliver stands in for your email provider's client. func deliver(ctx context.Context, o *OrderPlaced) error { return nil } type PlaceParams struct { OrderID string `json:"order_id"` Email string `json:"email"` TotalCents int64 `json:"total_cents"` } //encore:api public method=POST path=/orders func Place(ctx context.Context, p *PlaceParams) error { _, err := Orders.Publish(ctx, &OrderPlaced{OrderID: p.OrderID, Email: p.Email, TotalCents: p.TotalCents}) return err }

With encore run, a valid order logs receipt sent, an order with not-an-email logs receipt skipped once and is acked, and when deliver returns an error the handler logs receipt failed, will retry again after the backoff, about 10 seconds later in our run. The RetryPolicy is the same settings as the Terraform above in one place: MinBackoff and MaxBackoff set the delay between attempts and MaxRetries how many attempts happen before the message is dead-lettered. The handler decides which errors are worth retrying, and the infrastructure to catch the rest exists for every subscription without a separate resource to remember.

An AI coding agent adding a new consumer declares the subscription with its retry policy in the same file, and the deploy creates the subscription along with its dead-letter topic.

Best fit: teams on GCP that want every subscription to have retries with backoff and a dead-letter topic, declared next to the handler.

Replaying dead-lettered messages

Once the cause is fixed, move the held messages back to the source topic. Replay should strip the dead-letter attributes, ack each message only after the republish succeeds and run with a time limit, so it stops when the hold subscription is drained. A small replay command with the Pub/Sub client for Go:

// replay/main.go // replay moves messages from the dead-letter hold subscription back to the // source topic once the bug that made them fail is fixed. package main import ( "context" "log" "os" "strings" "sync/atomic" "time" "cloud.google.com/go/pubsub/v2" ) func main() { ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) defer cancel() client, err := pubsub.NewClient(ctx, os.Getenv("PROJECT_ID")) if err != nil { log.Fatal(err) } defer client.Close() publisher := client.Publisher("order-placed") defer publisher.Stop() sub := client.Subscriber("order-placed-dlt-hold") var replayed atomic.Int64 err = sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) { // Drop the attributes Pub/Sub added when it dead-lettered the message. attrs := map[string]string{} for k, v := range m.Attributes { if !strings.HasPrefix(k, "CloudPubSubDeadLetter") { attrs[k] = v } } res := publisher.Publish(ctx, &pubsub.Message{Data: m.Data, Attributes: attrs}) if _, err := res.Get(ctx); err != nil { log.Printf("republish %s: %v", m.ID, err) m.Nack() return } m.Ack() replayed.Add(1) }) if err != nil && ctx.Err() == nil { log.Fatal(err) } log.Printf("replayed %d messages", replayed.Load()) }

Republishing creates new messages with new IDs, so consumers that deduplicate should use a business key such as the order ID, not the Pub/Sub message ID. Watch the dead-letter topic with an alert on the hold subscription's num_undelivered_messages metric, so a new failure mode shows up within minutes instead of at the next incident review.

Frequently asked questions

What is a dead-letter topic in Pub/Sub?

A dead-letter topic is a Pub/Sub topic that a subscription forwards messages to after they fail a set number of delivery attempts, between 5 and 100. It is configured on the subscription, not the source topic, and the forwarded message carries attributes naming the source subscription and the number of attempts.

Why are my Pub/Sub messages not going to the dead-letter topic?

Usually IAM. The Pub/Sub service agent, service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com, needs roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription, and Pub/Sub only counts delivery attempts when the dead-letter topic is configured with those permissions. Also check that the dead-letter topic has a subscription, or forwarded messages are not kept.

How do I replay messages from a Pub/Sub dead-letter topic?

Attach a subscription to the dead-letter topic so messages are retained, fix the cause of the failures, then run a small consumer that pulls from that subscription, republishes each message to the original topic without the CloudPubSubDeadLetter attributes, and acks it only after the publish succeeds.

Does Encore configure Pub/Sub dead-letter topics?

Yes. When an Encore app deploys to GCP, each Pub/Sub subscription declared in code gets a dead-letter topic, and the subscription's RetryPolicy sets the backoff and the number of attempts before a message is dead-lettered. Encore provisions an IAM role for the service, or a service account on GCP.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders