Publish and receive Pub/Sub messages with the google-cloud-pubsub library: batching, attributes, streaming pull with flow control, acks and nacks, retries, dead-letter topics and the emulator, plus typed topics declared in Python.

Google Cloud Pub/Sub is the managed message bus on GCP, and from Python you use it through the google-cloud-pubsub library. Publishing takes a few lines, but receiving reliably means understanding streaming pull, ack deadlines and flow control.
This guide covers both sides with version 2.x of the client, then the retry and dead-letter settings that decide what happens when a handler fails.
Install the library with pip install google-cloud-pubsub and create one PublisherClient per process. The client batches messages in the background, and batch settings trade a little latency for fewer requests:
import json
import os
from google.cloud import pubsub_v1
PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"]
# One client per process: it batches messages and keeps gRPC channels open.
publisher = pubsub_v1.PublisherClient(
batch_settings=pubsub_v1.types.BatchSettings(
max_messages=100, # send when 100 messages are queued...
max_latency=0.05, # ...or after 50 ms, whichever comes first
)
)
topic_path = publisher.topic_path(PROJECT, "order-placed")
def publish_order(order_id: str, total_cents: int) -> str:
data = json.dumps({"order_id": order_id, "total_cents": total_cents}).encode()
# Attributes are string key/value pairs, handy for subscription filters.
future = publisher.publish(topic_path, data, source="checkout")
return future.result(timeout=30) # the server-assigned message ID
if __name__ == "__main__":
print(publish_order("ord_123", 4200))
The payload must be bytes, and you own the serialization format. Attributes ride alongside the payload and can drive subscription filters, so a subscriber that only cares about one source never receives the rest. Calling future.result() on every publish serializes the calls, so in a request path add a done callback instead when you do not need the ID before responding.
SubscriberClient.subscribe opens a streaming pull and calls your function on a thread pool for each message. Without limits it can pull far more messages than the process can handle, so set flow control:
import json
import os
import threading
from google.cloud import pubsub_v1
PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"]
subscriber = pubsub_v1.SubscriberClient()
sub_path = subscriber.subscription_path(PROJECT, "order-placed-email")
processed: set[str] = set() # per-process demo; use a unique constraint in the database
processed_lock = threading.Lock()
def handle(message: pubsub_v1.subscriber.message.Message) -> None:
event = json.loads(message.data)
try:
# Delivery is at least once, so claim the order ID under a lock first.
with processed_lock:
if event["order_id"] in processed:
message.ack()
return
processed.add(event["order_id"])
print("sending receipt for", event["order_id"])
message.ack()
except Exception:
with processed_lock:
processed.discard(event["order_id"])
message.nack() # redeliver now instead of waiting for the ack deadline
raise
flow_control = pubsub_v1.types.FlowControl(max_messages=20)
streaming_pull = subscriber.subscribe(sub_path, callback=handle, flow_control=flow_control)
with subscriber:
try:
streaming_pull.result() # blocks; callbacks run on a thread pool
except KeyboardInterrupt:
streaming_pull.cancel()
streaming_pull.result()
The client extends the ack deadline while the callback runs, up to a maximum, so long handlers do not get redelivered mid-flight. Callbacks run on threads, which means shared state needs a lock and async code needs its own event loop. Each subscriber process holds a streaming connection, so it runs as a long-lived worker, not inside a request handler.
A nacked or expired message is redelivered, by default immediately, and a message that always fails will loop forever. Set a retry policy with exponential backoff and a dead-letter topic on the subscription, as described in the handling failures guide:
The Pub/Sub service agent also needs publish rights on the dead-letter topic and subscribe rights on the source subscription, or failed messages never move. Standard delivery stays at least once even with these settings, so keep handlers idempotent. Exactly-once delivery is a subscription option that removes redelivery after a successful ack, at the cost of lower throughput and checking the result of ack_with_response().
The emulator runs with gcloud beta emulators pubsub start, and setting PUBSUB_EMULATOR_HOST=localhost:8085 makes the Python client connect to it with anonymous credentials. It starts empty, so a test setup script has to create every topic and subscription the code expects, and features such as dead-letter topics and IAM checks are not fully emulated.
Encore works alongside your Python framework, here FastAPI, as an SDK that declares the infrastructure in code. Encore for Python is in private beta, and you can request an invite. A topic and subscription are module-level variables, encore run delivers messages through a local broker, and a deploy creates Pub/Sub topics and subscriptions in your own GCP project, or SNS and SQS on AWS.
import encore
from encore import pubsub
from fastapi import FastAPI
from pydantic import BaseModel
class OrderPlaced(BaseModel):
order_id: str
total_cents: int
# Pub/Sub on GCP, SNS and SQS on AWS, an in-process broker under `encore run`.
order_placed = pubsub.Topic[OrderPlaced]("order-placed")
processed: set[str] = set() # stand-in for a table with a unique key
async def send_receipt(msg: pubsub.Message[OrderPlaced]) -> None:
# Returning acks the message; raising retries it with backoff.
if msg.data.order_id in processed:
return
processed.add(msg.data.order_id)
print(f"receipt for {msg.data.order_id} (attempt {msg.delivery_attempt})")
receipts = pubsub.Subscription[OrderPlaced](
order_placed,
"send-receipt",
handler=send_receipt,
ack_deadline="30s",
max_retries=5,
)
app = FastAPI()
@app.post("/orders", status_code=202)
async def place_order(order: OrderPlaced) -> dict:
msg_id = await order_placed.publish(order)
return {"queued": msg_id}
if __name__ == "__main__":
# Serves the app where Encore assigns it and starts the subscriptions.
encore.run(app)
The payload is a Pydantic model, so serialization and validation happen on both ends, and the handler is async rather than a thread-pool callback. Retries, backoff and the ack deadline live on the Subscription, next to the handler they apply to. The service is registered with an encore.service.json file next to the code, and the SDK needs Python 3.9 or later.
Encore provisions an IAM role for the service, or a service account on GCP, and local traces follow each order from the endpoint into the handler. An agent adding a second consumer writes one more Subscription and watches its delivery in the local trace, with no gcloud commands or emulator setup scripts.
To run the same code, request a private-beta invite:
Create one pubsub_v1.PublisherClient per process, build the topic path with topic_path(project, topic), and call publish(topic_path, data, **attributes) with the payload as bytes. publish returns a future, and future.result() gives the server-assigned message ID once the batch is sent.
message.ack() tells Pub/Sub the message was handled so it is not delivered again. message.nack() releases it for redelivery right away instead of waiting for the ack deadline. A message that is neither acked nor nacked is redelivered after its deadline expires, which the client extends automatically while your callback runs.
Yes. Standard subscriptions deliver at least once, so a handler can see the same message twice after a timeout, a crash or a redelivery. Make the side effect idempotent, for example with a unique key in the database, or enable exactly-once delivery on the subscription and check the ack result.
Start the Pub/Sub emulator with gcloud beta emulators pubsub start and set PUBSUB_EMULATOR_HOST, and the Python client connects to it instead of Google Cloud. You still create topics and subscriptions in the emulator yourself. With Encore, encore run starts a local broker for the topics declared in code.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.