Nov 19, 20266 min read

How to Use Google Cloud Pub/Sub from Python

Publish and receive Pub/Sub messages with the google-cloud-pubsub library: batching, attributes, streaming pull with flow control, acks and nacks, retries, dead-letter topics and the emulator, plus typed topics declared in Python.

Illustration for How to Use Google Cloud Pub/Sub from Python

Google Cloud Pub/Sub is the managed message bus on GCP, and from Python you use it through the google-cloud-pubsub library. Publishing takes a few lines, but receiving reliably means understanding streaming pull, ack deadlines and flow control.

This guide covers both sides with version 2.x of the client, then the retry and dead-letter settings that decide what happens when a handler fails.

Pub/Sub in Python at a glance

TaskAPINotes
PublishPublisherClient.publish(topic_path, data, **attrs)Returns a future, batches automatically
ReceiveSubscriberClient.subscribe(sub_path, callback)Streaming pull, callbacks on a thread pool
Limit in-flight workFlowControl(max_messages=...)Caps outstanding messages per subscriber
Confirm handlingmessage.ack() / message.nack()Unacked messages return after the deadline
Retries and DLQSubscription retry policy and dead-letter topicSet on the subscription, not in code
Local testingEmulator with PUBSUB_EMULATOR_HOSTTopics and subscriptions created by hand

How do you publish messages from Python?

Install the library with pip install google-cloud-pubsub and create one PublisherClient per process. The client batches messages in the background, and batch settings trade a little latency for fewer requests:

import json import os from google.cloud import pubsub_v1 PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"] # One client per process: it batches messages and keeps gRPC channels open. publisher = pubsub_v1.PublisherClient( batch_settings=pubsub_v1.types.BatchSettings( max_messages=100, # send when 100 messages are queued... max_latency=0.05, # ...or after 50 ms, whichever comes first ) ) topic_path = publisher.topic_path(PROJECT, "order-placed") def publish_order(order_id: str, total_cents: int) -> str: data = json.dumps({"order_id": order_id, "total_cents": total_cents}).encode() # Attributes are string key/value pairs, handy for subscription filters. future = publisher.publish(topic_path, data, source="checkout") return future.result(timeout=30) # the server-assigned message ID if __name__ == "__main__": print(publish_order("ord_123", 4200))

The payload must be bytes, and you own the serialization format. Attributes ride alongside the payload and can drive subscription filters, so a subscriber that only cares about one source never receives the rest. Calling future.result() on every publish serializes the calls, so in a request path add a done callback instead when you do not need the ID before responding.

How do you receive and acknowledge messages?

SubscriberClient.subscribe opens a streaming pull and calls your function on a thread pool for each message. Without limits it can pull far more messages than the process can handle, so set flow control:

import json import os import threading from google.cloud import pubsub_v1 PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"] subscriber = pubsub_v1.SubscriberClient() sub_path = subscriber.subscription_path(PROJECT, "order-placed-email") processed: set[str] = set() # per-process demo; use a unique constraint in the database processed_lock = threading.Lock() def handle(message: pubsub_v1.subscriber.message.Message) -> None: event = json.loads(message.data) try: # Delivery is at least once, so claim the order ID under a lock first. with processed_lock: if event["order_id"] in processed: message.ack() return processed.add(event["order_id"]) print("sending receipt for", event["order_id"]) message.ack() except Exception: with processed_lock: processed.discard(event["order_id"]) message.nack() # redeliver now instead of waiting for the ack deadline raise flow_control = pubsub_v1.types.FlowControl(max_messages=20) streaming_pull = subscriber.subscribe(sub_path, callback=handle, flow_control=flow_control) with subscriber: try: streaming_pull.result() # blocks; callbacks run on a thread pool except KeyboardInterrupt: streaming_pull.cancel() streaming_pull.result()

The client extends the ack deadline while the callback runs, up to a maximum, so long handlers do not get redelivered mid-flight. Callbacks run on threads, which means shared state needs a lock and async code needs its own event loop. Each subscriber process holds a streaming connection, so it runs as a long-lived worker, not inside a request handler.

What happens when a handler fails?

A nacked or expired message is redelivered, by default immediately, and a message that always fails will loop forever. Set a retry policy with exponential backoff and a dead-letter topic on the subscription, as described in the handling failures guide:

~
$ gcloud pubsub topics create order-placed-dlq$ gcloud pubsub subscriptions update order-placed-email \ --min-retry-delay=10s --max-retry-delay=600s \ --dead-letter-topic=order-placed-dlq --max-delivery-attempts=5

The Pub/Sub service agent also needs publish rights on the dead-letter topic and subscribe rights on the source subscription, or failed messages never move. Standard delivery stays at least once even with these settings, so keep handlers idempotent. Exactly-once delivery is a subscription option that removes redelivery after a successful ack, at the cost of lower throughput and checking the result of ack_with_response().

Test locally with the emulator

The emulator runs with gcloud beta emulators pubsub start, and setting PUBSUB_EMULATOR_HOST=localhost:8085 makes the Python client connect to it with anonymous credentials. It starts empty, so a test setup script has to create every topic and subscription the code expects, and features such as dead-letter topics and IAM checks are not fully emulated.

Typed topics declared in Python with Encore

Encore

Encore works alongside your Python framework, here FastAPI, as an SDK that declares the infrastructure in code. Encore for Python is in private beta, and you can request an invite. A topic and subscription are module-level variables, encore run delivers messages through a local broker, and a deploy creates Pub/Sub topics and subscriptions in your own GCP project, or SNS and SQS on AWS.

import encore from encore import pubsub from fastapi import FastAPI from pydantic import BaseModel class OrderPlaced(BaseModel): order_id: str total_cents: int # Pub/Sub on GCP, SNS and SQS on AWS, an in-process broker under `encore run`. order_placed = pubsub.Topic[OrderPlaced]("order-placed") processed: set[str] = set() # stand-in for a table with a unique key async def send_receipt(msg: pubsub.Message[OrderPlaced]) -> None: # Returning acks the message; raising retries it with backoff. if msg.data.order_id in processed: return processed.add(msg.data.order_id) print(f"receipt for {msg.data.order_id} (attempt {msg.delivery_attempt})") receipts = pubsub.Subscription[OrderPlaced]( order_placed, "send-receipt", handler=send_receipt, ack_deadline="30s", max_retries=5, ) app = FastAPI() @app.post("/orders", status_code=202) async def place_order(order: OrderPlaced) -> dict: msg_id = await order_placed.publish(order) return {"queued": msg_id} if __name__ == "__main__": # Serves the app where Encore assigns it and starts the subscriptions. encore.run(app)

The payload is a Pydantic model, so serialization and validation happen on both ends, and the handler is async rather than a thread-pool callback. Retries, backoff and the ack deadline live on the Subscription, next to the handler they apply to. The service is registered with an encore.service.json file next to the code, and the SDK needs Python 3.9 or later.

Encore provisions an IAM role for the service, or a service account on GCP, and local traces follow each order from the endpoint into the handler. An agent adding a second consumer writes one more Subscription and watches its delivery in the local trace, with no gcloud commands or emulator setup scripts.

To run the same code, request a private-beta invite:

Try Encore →

Frequently asked questions

How do I publish a message to Pub/Sub in Python?

Create one pubsub_v1.PublisherClient per process, build the topic path with topic_path(project, topic), and call publish(topic_path, data, **attributes) with the payload as bytes. publish returns a future, and future.result() gives the server-assigned message ID once the batch is sent.

What is the difference between ack and nack in the Python client?

message.ack() tells Pub/Sub the message was handled so it is not delivered again. message.nack() releases it for redelivery right away instead of waiting for the ack deadline. A message that is neither acked nor nacked is redelivered after its deadline expires, which the client extends automatically while your callback runs.

Does Pub/Sub deliver messages more than once?

Yes. Standard subscriptions deliver at least once, so a handler can see the same message twice after a timeout, a crash or a redelivery. Make the side effect idempotent, for example with a unique key in the database, or enable exactly-once delivery on the subscription and check the ack result.

How do I run Pub/Sub locally for Python tests?

Start the Pub/Sub emulator with gcloud beta emulators pubsub start and set PUBSUB_EMULATOR_HOST, and the Python client connects to it instead of Google Cloud. You still create topics and subscriptions in the emulator yourself. With Encore, encore run starts a local broker for the topics declared in code.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders