Nov 18, 20267 min read

Pub/Sub Retry Policy: Exponential Backoff for Subscriptions

How immediate redelivery and exponential backoff work on Google Cloud Pub/Sub subscriptions, how they interact with ack deadlines, push backoff and dead-letter topics, and how to declare the retry policy next to the handler.

Illustration for Pub/Sub Retry Policy: Exponential Backoff for Subscriptions

A Google Cloud Pub/Sub subscription redelivers a message whenever the subscriber nacks it or misses the ack deadline, and by default it does that immediately. For a handler that calls an email provider during an outage, that turns one failing message into a tight loop of requests.

An exponential backoff retry policy spaces those attempts out per message, between a minimum and a maximum delay, and a dead-letter topic catches the messages that never succeed. This guide covers both settings, the limits Google sets on them, and how to declare them next to the handler code in Go with Encore.

Retry policies at a glance

Encore RetryPolicy (Go)Pub/Sub subscription setting
Minimum backoffMinBackoff, default 10 secondsminimumBackoff, 0 to 600 seconds, console default 10 seconds
Maximum backoffMaxBackoff, default 10 minutesmaximumBackoff, 0 to 600 seconds, console default 600 seconds
Retry limitMaxRetries, default 100, then the dead-letter topicmaxDeliveryAttempts on the dead-letter policy, 5 to 100, default 5
Where it is configuredIn the subscription declaration in GoConsole, gcloud, Terraform or the admin API
Dead-letter topicProvisioned with the subscriptionA separate topic plus IAM grants for the Pub/Sub service agent
Default without configBackoff with the defaults aboveImmediate redelivery

The Pub/Sub values come from the subscription retry policy docs and the dead-letter topic docs, and the Encore defaults come from the RetryPolicy type in the encore.dev/pubsub package.

How does exponential backoff work on Pub/Sub?

Backoff starts when a delivery fails, which on Pub/Sub means one of two things: the subscriber sends a negative acknowledgment, or the ack deadline expires before it acknowledges. Pub/Sub then waits at least the minimum backoff before the next attempt of that message, and adds more delay on each consecutive failure until it reaches the maximum backoff.

The policy is applied per message on a best-effort basis. Other messages on the subscription keep flowing while one message backs off, unless the subscription uses ordering keys, where a failing message holds back the later messages with the same key. Google also warns that nacking a large number of messages at once can lead to some of them arriving with less backoff than configured, and that Pub/Sub may slow delivery for the whole subscription in that case.

Push subscriptions also have push backoff: when a push endpoint returns errors, Pub/Sub slows delivery to the endpoint as a whole, and the effective delay for a message is the larger of the push backoff and the retry policy delay. A Cloud Run service that returns 500 for every request during a bad deploy sees both at once, which is covered in more detail in Pub/Sub push vs pull.

How do you configure the retry policy with gcloud?

The gcloud flags map one to one onto the two backoff values. This creates a subscription with a 10 second minimum and a 5 minute maximum:

gcloud pubsub subscriptions create send-receipt \ --topic=orders-placed \ --ack-deadline=30 \ --min-retry-delay=10s \ --max-retry-delay=300s

An existing subscription takes the same flags on gcloud pubsub subscriptions update. Setting a dead-letter topic needs --dead-letter-topic and --max-delivery-attempts, and it only starts counting attempts once the Pub/Sub service agent can publish to the dead-letter topic and subscribe to the source subscription.

How do you configure it with Terraform?

In Terraform the retry policy and the dead-letter policy are blocks on google_pubsub_subscription. The two IAM members at the bottom are the grants that the dead-letter docs require for the Pub/Sub service agent, and forgetting them is the usual reason a dead-letter topic stays empty.

data "google_project" "this" {} resource "google_pubsub_topic" "orders_placed" { name = "orders-placed" } resource "google_pubsub_topic" "send_receipt_dead_letter" { name = "send-receipt-dead-letter" } resource "google_pubsub_subscription" "send_receipt" { name = "send-receipt" topic = google_pubsub_topic.orders_placed.id ack_deadline_seconds = 30 retry_policy { minimum_backoff = "10s" maximum_backoff = "300s" } dead_letter_policy { dead_letter_topic = google_pubsub_topic.send_receipt_dead_letter.id max_delivery_attempts = 8 } } locals { pubsub_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com" } resource "google_pubsub_topic_iam_member" "dead_letter_publisher" { topic = google_pubsub_topic.send_receipt_dead_letter.id role = "roles/pubsub.publisher" member = local.pubsub_agent } resource "google_pubsub_subscription_iam_member" "source_subscriber" { subscription = google_pubsub_subscription.send_receipt.id role = "roles/pubsub.subscriber" member = local.pubsub_agent }

The Pub/Sub with Terraform guide covers the topic, the subscriber service account and the push configuration around this block.

How should you pick the backoff values?

The right values follow from the failure you expect: a rate-limited API that recovers within seconds suits a minimum of 5 to 10 seconds and a maximum of a minute or two, while a dependency that goes down for a deploy or an incident suits a maximum close to the 600 second ceiling. The retry limit then decides how long a message keeps trying in total, and once the delay has grown to the maximum, every further attempt adds up to that maximum again.

Keep the ack deadline longer than the slowest successful run of the handler. A handler that sometimes takes 45 seconds behind a 30 second deadline gets redelivered while the first attempt is still running, which looks like a retry storm and is a timeout problem. The handler also has to be idempotent, since Pub/Sub delivers at least once and a retry can arrive after a success that was never acknowledged, so record the order ID or message ID before sending the email or charging the card.

When the delay is the point, for example "send this reminder in two hours", use a scheduler instead of the retry policy. The retry docs say so directly and point to Cloud Tasks, and Cloud Tasks vs Pub/Sub compares the two.

How do you declare the retry policy with Encore?

Encore

Encore brings infrastructure from code, so the topic, the subscription and its retry policy are Go declarations in the service that handles the message. The Pub/Sub docs describe the behavior: when the handler returns an error, the message is retried according to the subscription's retry policy, and after MaxRetries it goes to a dead-letter queue for that subscriber. On deploy to GCP, Encore provisions the Pub/Sub topic, the subscription and a dead-letter topic as the GCP infrastructure docs describe, and on AWS the same code becomes SNS and SQS with a dead-letter queue.

Application code passes through Encore and infrastructure guardrails to resources provisioned in your own AWS or GCP account.
Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

// orders/orders.go package orders import ( "context" "time" "encore.dev/pubsub" ) type OrderPlaced struct { OrderID string `json:"order_id"` Email string `json:"email"` } var OrdersPlaced = pubsub.NewTopic[*OrderPlaced]("orders-placed", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var _ = pubsub.NewSubscription(OrdersPlaced, "send-receipt", pubsub.SubscriptionConfig[*OrderPlaced]{ Handler: sendReceipt, AckDeadline: 30 * time.Second, RetryPolicy: &pubsub.RetryPolicy{ MinBackoff: 10 * time.Second, MaxBackoff: 5 * time.Minute, MaxRetries: 8, }, }) func sendReceipt(ctx context.Context, msg *OrderPlaced) error { // Returning an error nacks the message, and Encore retries it // with the backoff above until MaxRetries is reached. return deliverReceipt(ctx, msg.OrderID, msg.Email) } func deliverReceipt(ctx context.Context, orderID, email string) error { // Call your email provider here, keyed on orderID so a retry // after an unacknowledged success does not send twice. return nil } //encore:api public method=POST path=/orders func Place(ctx context.Context, p *OrderPlaced) error { _, err := OrdersPlaced.Publish(ctx, p) return err }

deliverReceipt is where the call to your email provider goes. The RetryPolicy values are read at compile time, and the package documentation states that they may be clamped to what the target cloud supports, which on GCP means the 600 second backoff ceiling. MaxRetries also accepts pubsub.NoRetries and pubsub.InfiniteRetries for subscriptions that should dead-letter on the first error or never dead-letter at all.

encore run runs the topic and subscription locally, so a failing handler can be watched retrying in the local trace view before anything is deployed. A coding agent that adds a subscriber writes the retry policy in the same Go file as the handler, and the review shows the backoff and retry limit next to the code they protect.

Best fit: Go backends on GCP or AWS that want the subscription, its backoff and its dead-letter topic declared with the handler and running locally the same way.

Retry policy checklist

  • Turn on exponential backoff for any subscriber that calls an external dependency, since the default is immediate redelivery.
  • Keep the ack deadline above the slowest successful handler run.
  • Choose the maximum backoff from the length of the outages you expect, up to 600 seconds.
  • Set a retry limit and a dead-letter topic, and grant the Pub/Sub service agent its two roles if you manage the IAM yourself.
  • Make the handler idempotent on a business key before it has side effects.
  • Use Cloud Tasks or a scheduled job when you want a planned delay.

Frequently asked questions

What is the default retry policy for a Pub/Sub subscription?

A new Pub/Sub subscription uses immediate redelivery, so a nacked or expired message can come back right away, often to the same subscriber. Exponential backoff is opt-in, and when you enable it the console defaults are 10 seconds of minimum backoff and 600 seconds of maximum backoff.

What is the maximum backoff on Pub/Sub?

Both the minimum and the maximum backoff accept values between 0 and 600 seconds, so the longest wait between two delivery attempts of the same message is 10 minutes. For longer or scheduled delays, Google points to Cloud Tasks instead of the retry policy.

Does the Pub/Sub retry policy apply to push subscriptions?

Yes. The retry policy works per message on both pull and push subscriptions. Push subscriptions also have push backoff, which slows delivery for the whole endpoint when it returns errors, and the effective delay is the larger of the two.

How do I set a Pub/Sub retry policy with Encore?

Pass a RetryPolicy with MinBackoff, MaxBackoff and MaxRetries in the pubsub.SubscriptionConfig of the subscription in Go. Encore provisions the subscription and its dead-letter topic on deploy, and values outside what the cloud supports are clamped to the supported range.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders