Nov 20, 20265 min read

Change Data Capture with Postgres: Logical Decoding vs the Outbox

How change data capture works on Postgres with logical decoding, publications and replication slots, what it takes to run on RDS and Cloud SQL, the WAL retention risk of a stuck slot, and when a transactional outbox is the simpler way to publish events.

Illustration for Change Data Capture with Postgres: Logical Decoding vs the Outbox

Sooner or later another system needs to know when rows change: a search index, a data warehouse, a cache, a second service. Writing to the database and then publishing an event from application code works until a process dies between the two calls and the systems drift apart.

Change data capture solves it at the database level, and the transactional outbox solves it at the application level.

How logical decoding works

Every committed change in Postgres is written to the write-ahead log. Logical decoding turns those WAL records back into row-level changes and streams them to a client through a replication slot, which remembers how far the client has read. Three pieces are involved:

PieceRole
wal_level = logicalMakes the WAL carry enough information to decode rows
PublicationThe set of tables whose changes are streamed
Replication slotA durable cursor into the WAL for one consumer
CREATE PUBLICATION app_changes FOR TABLE orders, customers; SELECT pg_create_logical_replication_slot('search_indexer', 'pgoutput');

Consumers are rarely hand-written, and the Debezium Postgres connector reads a slot with the built-in pgoutput plugin, takes an initial snapshot and writes changes to Kafka or another sink, and managed services wrap the same mechanism.

Running it on RDS and Cloud SQL

Managed Postgres hides wal_level behind its own switch. On RDS, the logical replication docs describe setting the static parameter rds.logical_replication to 1 in a custom parameter group and rebooting, and the consumer's user needs the rds_replication role. On Cloud SQL the equivalent is the cloudsql.logical_decoding flag, set on the instance and also applied with a restart. Both increase WAL volume, so enable them only when a consumer exists.

The stuck slot problem

A replication slot holds every WAL segment its consumer has not confirmed. If the consumer stops, the WAL grows until storage runs out, and on managed databases storage autoscaling can hide it until the bill arrives. Two safeguards are standard:

-- Cap how much WAL a slot may retain (Postgres 13+); the slot is invalidated beyond this ALTER SYSTEM SET max_slot_wal_keep_size = '50GB'; -- on RDS, set it in the parameter group -- Alert on retained WAL per slot SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag FROM pg_replication_slots;

The replication settings reference documents max_slot_wal_keep_size. An invalidated slot means the consumer must resnapshot, which is better than a full disk. Slots also hold back vacuum's horizon for the system catalogs, another reason to drop unused slots promptly.

CDC or an outbox

CDC captures every change, including those made by migrations, admin scripts and other applications, without changing application code. That makes it the right tool for replicating tables into a warehouse or search index. Its events, though, are row images: an UPDATE orders SET status = 'shipped' arrives as a before and after row, not as an OrderShipped event with the fields a consumer cares about, and every schema change becomes a change to the event format.

A transactional outbox writes the event itself into an outbox table in the same transaction as the business change, and a relay publishes committed rows to a message broker. Events are designed for consumers, need no logical replication or slot monitoring, and can never be published for a rolled-back transaction.

Best fit: CDC for replicating tables into analytics, search and caches; an outbox for domain events between services.

Outbox events with Encore

Encore

Encore brings infrastructure from code to event publishing: the Pub/Sub topic is declared in Go and runs on SNS and SQS in your AWS account or Google Cloud Pub/Sub on GCP, and the transactional outbox package binds that topic to a database transaction so the event and the row commit together:

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

package users import ( "context" "encore.dev/pubsub" "encore.dev/storage/sqldb" "x.encore.dev/infra/pubsub/outbox" ) type UserCreated struct { UserID int64 `json:"user_id"` Email string `json:"email"` } var UserEvents = pubsub.NewTopic[*UserCreated]("user-created", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) // Migrations create the users table and the outbox table from the outbox package docs. var db = sqldb.NewDatabase("users", sqldb.DatabaseConfig{Migrations: "./migrations"}) //encore:service type Service struct{} func initService() (*Service, error) { relay := outbox.NewRelay(outbox.SQLDBStore(db)) outbox.RegisterTopic(relay, pubsub.TopicRef[pubsub.Publisher[*UserCreated]](UserEvents)) go relay.PollForMessage(context.Background(), -1) return &Service{}, nil } type SignupParams struct { Email string `json:"email"` } type SignupResponse struct { UserID int64 `json:"user_id"` } //encore:api public method=POST path=/signup func (s *Service) Signup(ctx context.Context, p *SignupParams) (*SignupResponse, error) { tx, err := db.Begin(ctx) if err != nil { return nil, err } defer tx.Rollback() var id int64 if err := tx.QueryRow(ctx, `INSERT INTO users (email) VALUES ($1) RETURNING id`, p.Email).Scan(&id); err != nil { return nil, err } // Publishing through the bound ref inserts an outbox row in this transaction. ref := outbox.Bind(pubsub.TopicRef[pubsub.Publisher[*UserCreated]](UserEvents), outbox.TxPersister(tx)) if _, err := ref.Publish(ctx, &UserCreated{UserID: id, Email: p.Email}); err != nil { return nil, err } return &SignupResponse{UserID: id}, tx.Commit() }

encore run starts Postgres and an in-process Pub/Sub locally, so you can sign up a user and watch the relay deliver the event in the local development dashboard. The same code deploys to RDS with SNS and SQS, or Cloud SQL with Google Cloud Pub/Sub, as listed in the infrastructure docs, and a table replication need can still use CDC on that RDS instance alongside it.

An AI coding agent adding a new event writes the struct, the topic and the publish call inside the existing transaction in one change, with no connector configuration or slot to set up outside the repository.

Best fit: services that publish domain events between services and want them consistent with the database without operating CDC.

How to approach it

Use CDC when you need every change to a table mirrored elsewhere, and budget for slot monitoring and max_slot_wal_keep_size from day one. Use an outbox when services need meaningful events, and keep the two separate: replicating tables is a data pipeline concern, publishing events is an application one.

Frequently asked questions

What is change data capture in Postgres?

Change data capture streams every committed insert, update and delete from the database to other systems. Postgres supports it through logical decoding, which reads the write-ahead log and turns it into row-level changes for a client attached to a replication slot.

How do I enable logical replication on Amazon RDS for PostgreSQL?

Set the static parameter rds.logical_replication to 1 in a custom DB parameter group and reboot the instance. That sets wal_level to logical, after which you can create publications and replication slots as a user with the rds_replication role.

Can a replication slot fill up the disk?

Yes. A slot keeps all WAL its consumer has not confirmed, so a stopped or slow consumer makes WAL accumulate until storage runs out. Set max_slot_wal_keep_size to cap retention and alert on the lag of every slot in pg_replication_slots.

How do I publish database changes as events in Encore without CDC?

Use the transactional outbox from x.encore.dev/infra/pubsub/outbox. Bind a topic reference to the transaction with outbox.Bind and outbox.TxPersister, publish as usual, and a relay delivers the committed rows to the Pub/Sub topic.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders