Nov 18, 20267 min read

The Transactional Outbox Pattern: Publishing Events Without Losing Them

Write the business row and the event in one Postgres transaction, then relay the event to your queue, with a hand-rolled relay and the Encore outbox package.

Illustration for The Transactional Outbox Pattern: Publishing Events Without Losing Them

A service that saves an order and then publishes order.created to a queue has two writes that can disagree. If the database commit succeeds and the process dies before the publish, the order exists and the warehouse never hears about it. If you publish first and the transaction rolls back, the warehouse ships something nobody bought.

The transactional outbox pattern fixes this by making the event part of the same transaction as the business change. You insert an outbox row next to the order, commit once, and let a relay move committed rows to the broker afterwards.

This guide builds the pattern twice in Go: once by hand with Postgres and a polling relay, and once with Encore's outbox package, which binds a Pub/Sub topic to a database transaction.

The outbox pattern at a glance

StepWhat happensFailure behaviour
WriteThe order row and an outbox row are inserted in one transactionBoth commit or neither does
RelayA poller selects committed outbox rows, publishes them, deletes themA crash after publish and before delete republishes the row
ConsumeSubscribers process the event and record its IDDuplicate deliveries are skipped by the ID check
GuaranteeEvent published if and only if the write committedAt least once, never zero times, never for a rolled-back write

The pattern is described in Chris Richardson's microservices.io catalog as the answer to the dual-write problem, and it is the usual companion to the saga pattern when several services take part in one business process.

Build the outbox by hand

The outbox is an ordinary table, kept narrow: an ID that orders rows, the destination, the payload and a timestamp.

CREATE TABLE orders ( id BIGSERIAL PRIMARY KEY, customer_id TEXT NOT NULL, total_cents BIGINT NOT NULL CHECK (total_cents >= 0), created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE outbox ( id BIGSERIAL PRIMARY KEY, topic TEXT NOT NULL, payload JSONB NOT NULL, inserted_at TIMESTAMPTZ NOT NULL DEFAULT now() );

The write path inserts both rows inside one transaction. With pgx, pgx.BeginFunc commits when the callback returns nil and rolls back otherwise.

func CreateOrder(ctx context.Context, pool *pgxpool.Pool, customerID string, totalCents int64) (int64, error) { var orderID int64 err := pgx.BeginFunc(ctx, pool, func(tx pgx.Tx) error { if err := tx.QueryRow(ctx, ` INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2) RETURNING id`, customerID, totalCents).Scan(&orderID); err != nil { return err } payload, err := json.Marshal(map[string]any{ "order_id": orderID, "customer_id": customerID, "total_cents": totalCents, }) if err != nil { return err } _, err = tx.Exec(ctx, ` INSERT INTO outbox (topic, payload) VALUES ('order.created', $1)`, payload) return err }) return orderID, err }

The relay runs in a loop, and FOR UPDATE SKIP LOCKED lets you run more than one relay instance without two of them claiming the same row, and deleting inside the same transaction means a row disappears only after its publish call returned.

func RelayOnce(ctx context.Context, pool *pgxpool.Pool, publish func(ctx context.Context, topic string, id int64, payload []byte) error) (int, error) { sent := 0 err := pgx.BeginFunc(ctx, pool, func(tx pgx.Tx) error { rows, err := tx.Query(ctx, ` SELECT id, topic, payload FROM outbox ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED`) if err != nil { return err } type row struct { id int64 topic string payload []byte } var batch []row for rows.Next() { var r row if err := rows.Scan(&r.id, &r.topic, &r.payload); err != nil { return err } batch = append(batch, r) } if err := rows.Err(); err != nil { return err } for _, r := range batch { if err := publish(ctx, r.topic, r.id, r.payload); err != nil { break // leave this row and the rest for the next poll } if _, err := tx.Exec(ctx, `DELETE FROM outbox WHERE id = $1`, r.id); err != nil { return err } sent++ } return nil }) return sent, err }

Pass the outbox id to the broker as a message attribute or deduplication key. If the relay crashes after publish and before the commit, the row survives and is published again, so the consumer needs that ID to recognise a repeat.

SKIP LOCKED trades strict ordering for parallelism: two relays can publish rows 101 and 102 in either order. If a consumer depends on per-entity order, run a single relay or use an ordered queue keyed by the entity ID, such as an SQS FIFO queue or Pub/Sub ordering keys.

Best fit: services that already own their broker client and want full control over polling, batching and ordering.

Make consumers idempotent

The outbox moves the failure from "event lost" to "event delivered twice", which is the failure you can handle in code. Record each processed event ID in the consumer's own database, in the same transaction as the side effect:

CREATE TABLE processed_events ( event_id BIGINT PRIMARY KEY, processed_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- Inside the consumer transaction: INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT (event_id) DO NOTHING; -- If zero rows were inserted, the event was already handled: commit and ack.

How do you use an outbox with Encore?

Encore

Encore is a backend framework with built-in infrastructure, and its Go SDK ships the outbox as the x.encore.dev/infra/pubsub/outbox package. You bind a Pub/Sub topic reference to a database transaction, so Publish inserts an outbox row instead of calling the broker, and a relay publishes committed rows to the broker topic. Locally encore run starts Postgres and an in-memory broker, and on deploy the same topic becomes SNS and SQS on AWS or Pub/Sub on GCP in your own account.

The package expects an outbox table with this shape, which you add as a normal migration:

-- orders/migrations/2_outbox.up.sql CREATE TABLE outbox ( id BIGSERIAL PRIMARY KEY, topic TEXT NOT NULL, data JSONB NOT NULL, inserted_at TIMESTAMPTZ NOT NULL ); CREATE INDEX outbox_topic_idx ON outbox (topic, id);

The service writes the order and publishes through the bound reference inside one sqldb transaction, and starts the relay in initService:

package orders import ( "context" "encore.dev/pubsub" "encore.dev/storage/sqldb" "x.encore.dev/infra/pubsub/outbox" ) type OrderCreated struct { OrderID int64 `json:"order_id"` CustomerID string `json:"customer_id"` TotalCents int64 `json:"total_cents"` } var OrderEvents = pubsub.NewTopic[*OrderCreated]("order-created", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var db = sqldb.NewDatabase("orders", sqldb.DatabaseConfig{Migrations: "./migrations"}) //encore:service type Service struct { events pubsub.Publisher[*OrderCreated] } func initService() (*Service, error) { ref := pubsub.TopicRef[pubsub.Publisher[*OrderCreated]](OrderEvents) relay := outbox.NewRelay(outbox.SQLDBStore(db)) outbox.RegisterTopic(relay, ref) go relay.PollForMessages(context.Background(), 100) return &Service{events: ref}, nil } type CreateParams struct { CustomerID string `json:"customer_id"` TotalCents int64 `json:"total_cents"` } type CreateResponse struct { OrderID int64 `json:"order_id"` } //encore:api public method=POST path=/orders func (s *Service) Create(ctx context.Context, p *CreateParams) (*CreateResponse, error) { tx, err := db.Begin(ctx) if err != nil { return nil, err } defer tx.Rollback() // no-op after a successful Commit var id int64 if err := tx.QueryRow(ctx, ` INSERT INTO orders (customer_id, total_cents) VALUES ($1, $2) RETURNING id`, p.CustomerID, p.TotalCents).Scan(&id); err != nil { return nil, err } bound := outbox.Bind(s.events, outbox.TxPersister(tx)) if _, err := bound.Publish(ctx, &OrderCreated{ OrderID: id, CustomerID: p.CustomerID, TotalCents: p.TotalCents, }); err != nil { return nil, err } if err := tx.Commit(); err != nil { return nil, err } return &CreateResponse{OrderID: id}, nil }

The relay in the package locks rows with FOR UPDATE and deletes each row in the same transaction after its publish succeeds, so the delivery guarantee matches the hand-rolled version: at least once. Subscribers should still check an event key, here order_id, before acting. The ID returned by a bound Publish refers to the outbox row rather than the broker message, as the outbox docs note.

An agent adding a new event can declare the topic, bind it in the write path and add the migration in one change, then create an order against the local stack and watch the message reach the subscriber in the trace before anything is deployed.

Best fit: Go services that want the outbox guarantee on Encore Pub/Sub with the topic, table and relay all living in the service code.

Change data capture instead of polling

Teams with heavy event volume sometimes replace the polling relay with change data capture: Debezium reads the Postgres write-ahead log and turns inserts into the outbox table into Kafka messages. It removes polling latency and load at the price of running Kafka Connect, a replication slot and its monitoring. A polling relay at a one-second interval is enough for most order, billing and notification flows.

How to choose

Write the outbox and relay by hand when you need custom ordering or a broker the Encore package does not publish to. Use Encore's outbox when your events already go through Encore Pub/Sub, and you want the topic, migration and relay declared in the same Go service that writes the data. Reach for CDC only when polling latency or database load becomes a measured problem.

Frequently asked questions

What is the transactional outbox pattern?

The service writes its business change and an outbox row describing the event in the same database transaction. A separate relay reads committed outbox rows, publishes them to the message broker and then deletes or marks them. The event is published if and only if the business change committed, at least once.

Does the outbox pattern give exactly-once delivery?

No. If the relay publishes a message and crashes before deleting the row, the message is published again on the next poll. Consumers still need to be idempotent, for example by recording processed event IDs in a table with a unique constraint.

Why not publish to the queue right after the database commit?

The process can crash, or the broker call can fail, between the commit and the publish. The row is saved but no event goes out, and nothing retries it. Publishing before the commit has the opposite failure: subscribers see an event for a change that was rolled back.

How do I use a transactional outbox with Encore?

Bind an Encore Pub/Sub topic reference to a sqldb transaction with outbox.Bind and outbox.TxPersister from x.encore.dev/infra/pubsub/outbox. Publish calls insert rows into an outbox table, and a relay created with outbox.NewRelay(outbox.SQLDBStore(db)) publishes committed rows to the broker topic.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders