Google Cloud Pub/Sub with Terraform: Topics, Subscriptions and Dead Letters

A Pub/Sub topic in Terraform is a two-line resource, and a subscription is not much longer. The settings that decide whether messages survive a bad deploy are separate: a retry policy so failing messages back off, a dead-letter topic with its own subscription, two IAM bindings for a Google-managed service agent, and an expiration policy so quiet subscriptions are not deleted. Leave any of these out and the setup still applies cleanly, then misbehaves later.

This guide covers Terraform for each piece, the IAM for publishers and consumers, and how Encore provisions the same resources from the code that publishes and consumes.

What a Pub/Sub Terraform setup needs

Topic and subscription

variable "project" { type = string }

provider "google" {
  project = var.project
}

data "google_project" "this" {}

locals {
  # Google-managed service agent that moves messages to the dead-letter topic.
  pubsub_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

resource "google_pubsub_topic" "order_placed" {
  name                       = "order-placed"
  message_retention_duration = "86400s" # keep 1 day so subscriptions can seek back
}

resource "google_pubsub_topic" "order_placed_dlq" {
  name = "order-placed-dlq"
}

resource "google_pubsub_subscription" "reserve_stock" {
  name                 = "reserve-stock"
  topic                = google_pubsub_topic.order_placed.id
  ack_deadline_seconds = 60

  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"
  }

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.order_placed_dlq.id
    max_delivery_attempts = 5
  }

  expiration_policy {
    ttl = "" # never expire, even without traffic
  }
}

# A subscription on the dead-letter topic so failed messages are kept and can be inspected.
resource "google_pubsub_subscription" "order_placed_dlq" {
  name                       = "order-placed-dlq"
  topic                      = google_pubsub_topic.order_placed_dlq.id
  message_retention_duration = "604800s" # 7 days, the maximum
  expiration_policy {
    ttl = ""
  }
}

A few of these values deserve a second look:

• ack_deadline_seconds should cover the slowest message the consumer handles. When it runs out, Pub/Sub redelivers the message, possibly to another instance while the first is still working. The client libraries extend the deadline automatically while a handler runs, up to a limit you configure.
• retry_policy turns immediate redelivery into exponential backoff between 10 seconds and 10 minutes, which gives a struggling downstream API time to recover.
• max_delivery_attempts must be between 5 and 100, as described in the dead-letter topics docs. Delivery attempts are counted approximately, so do not build logic around the exact number.
• expiration_policy { ttl = "" } stops a subscription from being deleted after 31 days without activity, the default listed in the subscription properties docs.

IAM for publishers, consumers and dead lettering

variable "orders_sa" { type = string }    # service account of the publishing service
variable "inventory_sa" { type = string } # service account of the consuming service

resource "google_pubsub_topic_iam_member" "orders_publish" {
  topic  = google_pubsub_topic.order_placed.id
  role   = "roles/pubsub.publisher"
  member = "serviceAccount:${var.orders_sa}"
}

resource "google_pubsub_subscription_iam_member" "inventory_consume" {
  subscription = google_pubsub_subscription.reserve_stock.id
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:${var.inventory_sa}"
}

# Without these two bindings, failed messages are never forwarded to the dead-letter topic.
resource "google_pubsub_topic_iam_member" "agent_publish_dlq" {
  topic  = google_pubsub_topic.order_placed_dlq.id
  role   = "roles/pubsub.publisher"
  member = local.pubsub_agent
}

resource "google_pubsub_subscription_iam_member" "agent_ack_source" {
  subscription = google_pubsub_subscription.reserve_stock.id
  role         = "roles/pubsub.subscriber"
  member       = local.pubsub_agent
}

The last two bindings are the easiest to forget, because dead lettering is performed by the Pub/Sub service agent of your project, which needs to publish to the dead-letter topic and acknowledge on the source subscription. Without them, Pub/Sub does not even count delivery attempts, so failing messages keep being redelivered to the consumer until they expire. The topic and subscription bindings above follow the role guidance in the Pub/Sub access control docs. Both files pass terraform validate with the Google provider 7.x.

What still takes work after the first apply

• Each new consumer is another subscription, another dead-letter topic and subscription, and four more IAM bindings, in a repository separate from the consumer's code.
• Local development needs the Pub/Sub emulator, which does not support IAM operations, so a missing binding, including the dead-letter ones, only shows up in the cloud.
• Replaying dead letters is manual: pull from the dead-letter subscription and republish, or seek the main subscription back to a snapshot or timestamp.
• Schema drift between publisher and consumer goes unnoticed unless you add Pub/Sub schemas or validate payloads in code.

How does Encore provision Pub/Sub from code?

Encore applies infrastructure from code to TypeScript and Go backends. A topic and its subscriptions are typed declarations in the services that publish and consume them, and the Encore platform creates the matching cloud resources in your own GCP or AWS account. The GCP infrastructure docs describe Pub/Sub topics with dead-letter topics configured automatically and IAM permissions for each service to publish or consume.

Set up the app

The topic lives in the orders service, which publishes after an order is stored:

// orders/orders.go
package orders

import (
	"context"

	"encore.dev/pubsub"
)

type OrderPlaced struct {
	OrderID  string
	SKU      string
	Quantity int
}

// A Pub/Sub topic on GCP, SNS on AWS; runs locally with `encore run`.
var Placed = pubsub.NewTopic[*OrderPlaced]("order-placed", pubsub.TopicConfig{
	DeliveryGuarantee: pubsub.AtLeastOnce,
})

type PlaceParams struct {
	OrderID  string
	SKU      string
	Quantity int
}

//encore:api public method=POST path=/orders
func Place(ctx context.Context, p *PlaceParams) error {
	// ... insert the order ...
	_, err := Placed.Publish(ctx, &OrderPlaced{OrderID: p.OrderID, SKU: p.SKU, Quantity: p.Quantity})
	return err
}

The inventory service subscribes, with the ack deadline and retry policy from the Terraform example expressed in code:

// inventory/inventory.go
package inventory

import (
	"context"
	"time"

	"encore.app/orders"
	"encore.dev/pubsub"
	"encore.dev/rlog"
)

var _ = pubsub.NewSubscription(orders.Placed, "reserve-stock", pubsub.SubscriptionConfig[*orders.OrderPlaced]{
	Handler:     reserve,
	AckDeadline: 60 * time.Second,
	RetryPolicy: &pubsub.RetryPolicy{
		MinBackoff: 10 * time.Second,
		MaxBackoff: 10 * time.Minute,
		MaxRetries: 5,
	},
})

// reserve must be idempotent: Pub/Sub can deliver the same message twice.
func reserve(ctx context.Context, e *orders.OrderPlaced) error {
	rlog.Info("reserving stock", "order", e.OrderID, "sku", e.SKU, "qty", e.Quantity)
	// ... UPDATE stock ... WHERE NOT EXISTS (reservation for e.OrderID) ...
	return nil
}

When the handler returns an error, the message is retried according to RetryPolicy, and after MaxRetries it goes to the subscription's dead-letter queue, where it can be released for reprocessing once the bug is fixed, as the Pub/Sub docs explain. encore run runs the topic and subscription locally, and the local development dashboard traces each message from publish to handler, so the failure path can be tested before deploying.

When a coding agent adds a second consumer, for example an analytics subscription, it is a new NewSubscription call in a pull request, and the subscription, dead-letter topic and permissions come with the next deploy. Topics that other systems already publish to can be imported.

Best fit: Go and TypeScript teams on GCP who want Pub/Sub with dead letters and per-service IAM, created from the publishing and consuming code.

Good to know

• Pub/Sub delivers at least once by default, while exactly-once delivery is available for pull subscriptions in a single region, at the cost of higher latency, and handlers should stay idempotent either way.
• Ordering keys guarantee order only within a key and only when enable_message_ordering is set on the subscription when it is created. Changing it later replaces the subscription.
• Messages are limited to 10 MB, so store larger payloads in Cloud Storage and publish the object name.

Related guides

• What is Google Cloud Pub/Sub?
• Google Cloud Pub/Sub with Go
• Google Pub/Sub vs SQS
• Amazon SQS with Terraform
• Cloud SQL with Terraform