How dead-letter topics work in Google Cloud Pub/Sub, the IAM bindings they need, how to choose max delivery attempts and a retry policy, how to keep and replay dead-lettered messages, and how Encore sets them up per subscription.

A Pub/Sub subscription without a dead-letter topic retries a failing message until it expires from the subscription, seven days by default, and with the default retry policy it redelivers immediately each time. One malformed message can then burn CPU for a week and hide real failures in the noise. A dead-letter topic caps the attempts and moves the message somewhere you can inspect and replay it.
This guide covers how dead-letter topics work in Google Cloud Pub/Sub, the IAM bindings they need, choosing delivery attempts and backoff, keeping and replaying dead-lettered messages, and how Encore sets them up for every subscription declared in code.
Values are from Google Cloud's dead-letter topics guide and retry policy guide.
Pub/Sub tracks delivery attempts per message on a subscription. A message counts as failed when the subscriber nacks it, lets its ack deadline expire, or, for a push subscription, returns a non-2xx status. Once the attempts pass max_delivery_attempts, Pub/Sub wraps the message and publishes it to the dead-letter topic with attributes including CloudPubSubDeadLetterSourceSubscription and CloudPubSubDeadLetterSourceDeliveryCount, then acks it on the source subscription.
The count is approximate: Google notes that Pub/Sub may forward a message after fewer attempts than configured or try a few more times first, and that the tracked count can reset to zero, especially for pull subscriptions with inactive subscribers, so a handler can see a message more times than the limit. The count is also exposed to subscribers as the delivery attempt on each message, which is useful for logging and for treating the last attempt differently.
Forwarding is done by the Pub/Sub service agent of the project, service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com, not by your code. It needs roles/pubsub.publisher on the dead-letter topic to publish there and roles/pubsub.subscriber on the source subscription to ack the original. Without them, the subscription accepts the configuration but Pub/Sub does not count delivery attempts, so nothing is ever dead-lettered and the message keeps retrying. The Cloud console flags this in the subscription's details panel, and Terraform has to grant both explicitly.
A source topic, a dead-letter topic with a hold subscription so forwarded messages are kept, a source subscription with exponential backoff and five attempts, and the two IAM bindings:
# main.tf
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = ">= 7.0"
}
}
}
variable "project_id" {
type = string
}
data "google_project" "this" {
project_id = var.project_id
}
locals {
pubsub_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}
resource "google_pubsub_topic" "orders" {
project = var.project_id
name = "order-placed"
}
resource "google_pubsub_topic" "orders_dlt" {
project = var.project_id
name = "order-placed-dlt"
}
# Without a subscription, messages forwarded to the dead-letter topic are dropped.
resource "google_pubsub_subscription" "orders_dlt_hold" {
project = var.project_id
name = "order-placed-dlt-hold"
topic = google_pubsub_topic.orders_dlt.id
message_retention_duration = "604800s"
ack_deadline_seconds = 60
expiration_policy {
ttl = ""
}
}
resource "google_pubsub_subscription" "send_receipt" {
project = var.project_id
name = "send-receipt"
topic = google_pubsub_topic.orders.id
ack_deadline_seconds = 60
retry_policy {
minimum_backoff = "10s"
maximum_backoff = "600s"
}
dead_letter_policy {
dead_letter_topic = google_pubsub_topic.orders_dlt.id
max_delivery_attempts = 5
}
}
# The Pub/Sub service agent forwards failed messages, so it needs to publish to
# the dead-letter topic and to acknowledge them on the source subscription.
resource "google_pubsub_topic_iam_member" "dlt_publisher" {
project = var.project_id
topic = google_pubsub_topic.orders_dlt.name
role = "roles/pubsub.publisher"
member = local.pubsub_agent
}
resource "google_pubsub_subscription_iam_member" "source_subscriber" {
project = var.project_id
subscription = google_pubsub_subscription.send_receipt.name
role = "roles/pubsub.subscriber"
member = local.pubsub_agent
}
terraform validate passes on this with the Google provider, and the hold subscription has an empty expiration_policy so it does not expire after 31 days without subscribers, which is the default for subscriptions. Seven days of retention gives the team a week to notice before dead-lettered messages expire.
The default retry policy redelivers immediately, which suits short blips and is the wrong default for a downstream outage: every message retries as fast as the subscriber can fail it. Exponential backoff spreads attempts out between the minimum and maximum, and together with the attempt limit sets how long a message has before it is dead-lettered. With a 10 second minimum, a 600 second maximum and five attempts, a message gets a few minutes of retries, enough for a deploy or a short provider outage, before it moves to the DLT.
Code should also separate failures that can pass from ones that never will. A message with an invalid email address will fail on every attempt, so the handler should record it and ack it instead of spending five attempts on it, leaving the dead-letter topic for failures that need a fix and a replay.
Encore brings infrastructure from code to Google Cloud: a Pub/Sub topic and its subscriptions are declarations in Go or TypeScript, and when the app deploys to your GCP project Encore creates the topics and subscriptions and configures dead-letter topics for them, as listed in the GCP infrastructure docs. Encore provisions an IAM role for the service, or a service account on GCP.
Run in your terminal to get started locally.
// receipts/receipts.go
package receipts
import (
"context"
"net/mail"
"time"
"encore.dev/pubsub"
"encore.dev/rlog"
)
type OrderPlaced struct {
OrderID string `json:"order_id"`
Email string `json:"email"`
TotalCents int64 `json:"total_cents"`
}
// On GCP this is a Pub/Sub topic, and each subscription gets a dead-letter topic.
var Orders = pubsub.NewTopic[*OrderPlaced]("order-placed", pubsub.TopicConfig{
DeliveryGuarantee: pubsub.AtLeastOnce,
})
var _ = pubsub.NewSubscription(Orders, "send-receipt", pubsub.SubscriptionConfig[*OrderPlaced]{
Handler: sendReceipt,
AckDeadline: 60 * time.Second,
RetryPolicy: &pubsub.RetryPolicy{
MinBackoff: 10 * time.Second,
MaxBackoff: 10 * time.Minute,
MaxRetries: 5,
},
})
func sendReceipt(ctx context.Context, o *OrderPlaced) error {
// A message that can never succeed: record it and ack, so it does not
// burn five attempts and land in the dead-letter topic with the real failures.
if _, err := mail.ParseAddress(o.Email); err != nil {
rlog.Warn("receipt skipped: invalid email", "order", o.OrderID, "email", o.Email)
return nil
}
// A failure that may pass: return it, and Encore retries with backoff
// until MaxRetries, then the message moves to the dead-letter topic.
if err := deliver(ctx, o); err != nil {
rlog.Warn("receipt failed, will retry", "order", o.OrderID, "err", err)
return err
}
rlog.Info("receipt sent", "order", o.OrderID)
return nil
}
// deliver stands in for your email provider's client.
func deliver(ctx context.Context, o *OrderPlaced) error {
return nil
}
type PlaceParams struct {
OrderID string `json:"order_id"`
Email string `json:"email"`
TotalCents int64 `json:"total_cents"`
}
//encore:api public method=POST path=/orders
func Place(ctx context.Context, p *PlaceParams) error {
_, err := Orders.Publish(ctx, &OrderPlaced{OrderID: p.OrderID, Email: p.Email, TotalCents: p.TotalCents})
return err
}
With encore run, a valid order logs receipt sent, an order with not-an-email logs receipt skipped once and is acked, and when deliver returns an error the handler logs receipt failed, will retry again after the backoff, about 10 seconds later in our run. The RetryPolicy is the same settings as the Terraform above in one place: MinBackoff and MaxBackoff set the delay between attempts and MaxRetries how many attempts happen before the message is dead-lettered. The handler decides which errors are worth retrying, and the infrastructure to catch the rest exists for every subscription without a separate resource to remember.
An AI coding agent adding a new consumer declares the subscription with its retry policy in the same file, and the deploy creates the subscription along with its dead-letter topic.
Best fit: teams on GCP that want every subscription to have retries with backoff and a dead-letter topic, declared next to the handler.
Once the cause is fixed, move the held messages back to the source topic. Replay should strip the dead-letter attributes, ack each message only after the republish succeeds and run with a time limit, so it stops when the hold subscription is drained. A small replay command with the Pub/Sub client for Go:
// replay/main.go
// replay moves messages from the dead-letter hold subscription back to the
// source topic once the bug that made them fail is fixed.
package main
import (
"context"
"log"
"os"
"strings"
"sync/atomic"
"time"
"cloud.google.com/go/pubsub/v2"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
client, err := pubsub.NewClient(ctx, os.Getenv("PROJECT_ID"))
if err != nil {
log.Fatal(err)
}
defer client.Close()
publisher := client.Publisher("order-placed")
defer publisher.Stop()
sub := client.Subscriber("order-placed-dlt-hold")
var replayed atomic.Int64
err = sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
// Drop the attributes Pub/Sub added when it dead-lettered the message.
attrs := map[string]string{}
for k, v := range m.Attributes {
if !strings.HasPrefix(k, "CloudPubSubDeadLetter") {
attrs[k] = v
}
}
res := publisher.Publish(ctx, &pubsub.Message{Data: m.Data, Attributes: attrs})
if _, err := res.Get(ctx); err != nil {
log.Printf("republish %s: %v", m.ID, err)
m.Nack()
return
}
m.Ack()
replayed.Add(1)
})
if err != nil && ctx.Err() == nil {
log.Fatal(err)
}
log.Printf("replayed %d messages", replayed.Load())
}
Republishing creates new messages with new IDs, so consumers that deduplicate should use a business key such as the order ID, not the Pub/Sub message ID. Watch the dead-letter topic with an alert on the hold subscription's num_undelivered_messages metric, so a new failure mode shows up within minutes instead of at the next incident review.
A dead-letter topic is a Pub/Sub topic that a subscription forwards messages to after they fail a set number of delivery attempts, between 5 and 100. It is configured on the subscription, not the source topic, and the forwarded message carries attributes naming the source subscription and the number of attempts.
Usually IAM. The Pub/Sub service agent, service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com, needs roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription, and Pub/Sub only counts delivery attempts when the dead-letter topic is configured with those permissions. Also check that the dead-letter topic has a subscription, or forwarded messages are not kept.
Attach a subscription to the dead-letter topic so messages are retained, fix the cause of the failures, then run a small consumer that pulls from that subscription, republishes each message to the original topic without the CloudPubSubDeadLetter attributes, and acks it only after the publish succeeds.
Yes. When an Encore app deploys to GCP, each Pub/Sub subscription declared in code gets a dead-letter topic, and the subscription's RetryPolicy sets the backoff and the number of attempts before a message is dead-lettered. Encore provisions an IAM role for the service, or a service account on GCP.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.