SQS and Google Cloud Pub/Sub deliver messages at least once, so every consumer eventually sees a duplicate. How to make consumers safe to run twice with dedupe tables, conditional updates and idempotency keys, and what FIFO queues and exactly-once delivery do and do not cover.

Amazon SQS standard queues and Google Cloud Pub/Sub both deliver messages at least once, and they mean it: a consumer that crashes after charging a card but before deleting the message, a handler that runs past the visibility timeout, or a network blip on the acknowledgement will hand the same message to a consumer again. For a log line that is harmless, and for "credit this wallet with 50 dollars" it is a support ticket.
The fix is a consumer that produces the same result no matter how many times it sees a message. This guide covers where duplicates come from, the three patterns that make handlers idempotent, what SQS FIFO and Pub/Sub exactly-once delivery guarantee, and a worked example in Go.
On SQS, a received message is hidden for the visibility timeout and must be deleted before it ends. If the consumer crashes, runs too long or fails to call DeleteMessage, the message becomes visible again and another consumer receives it. Standard queues can also, rarely, deliver an extra copy on their own, as the at-least-once delivery docs explain.
On Pub/Sub, a message not acknowledged before its ack deadline is redelivered, and push subscriptions retry when the endpoint returns an error or times out. Producers add their own duplicates: a publisher that times out waiting for a publish response and retries can publish the same event twice, with two different message IDs.
Give every event an ID chosen by the producer, such as an order ID plus an event type, and keep a processed_events table with that ID as the primary key. The handler inserts the ID and applies its change in one database transaction. If the insert conflicts, the event was already applied and the handler returns success, and if anything fails, the transaction rolls back both the record and the change, so the retry starts clean.
This is the most general pattern, and it only works when the side effect is in the same database as the dedupe table.
Many handlers move a row from one state to another: an order from paid to shipped, an invoice from pending to sent. Writing the update as UPDATE ... SET status = 'shipped' WHERE id = $1 AND status = 'paid' and checking the affected row count makes the second delivery a no-op without any extra table. Upserts with ON CONFLICT DO NOTHING and "set to value" updates are idempotent in the same way, while increments are not.
A payment provider or email API is outside your transaction, so a handler that crashes after the call and before recording it will call again. Providers such as Stripe accept an Idempotency-Key header and return the first result for a repeated key, so derive the key from the event ID. For APIs without that feature, record an intent row first, make the call, then record the result, and have a reconciliation job check intents with no result.
SQS FIFO queues drop a message whose deduplication ID matches one sent in the previous five minutes, which catches producer retries in that window. They do not stop a consumer from processing a message, crashing before the delete, and receiving it again.
Pub/Sub exactly-once delivery guarantees that a message successfully acknowledged is not redelivered, within a single region and for pull subscriptions. A handler that applies its change and then dies before acknowledging still sees the message again. Both features reduce duplicates, and neither removes the need for idempotent handlers.
With the AWS SDK, the consumer long-polls ReceiveMessage, parses the body, runs the transaction with the dedupe insert, and calls DeleteMessage only after the commit. The queue needs a redrive policy to a dead-letter queue so a message that always fails stops cycling, a visibility timeout longer than the slowest handler, and an IAM policy allowing receive and delete on that queue. Each new consumer repeats that setup.
Encore brings infrastructure from code to Go and TypeScript, so the topic, the queue per subscriber and its dead-letter queue come from a few declarations: on AWS they become SNS and SQS, and on GCP they become Pub/Sub topics and subscriptions with dead-letter topics, in your own account. Delivery is at least once by default, and the Pub/Sub docs state that subscribers should be idempotent, which is what the handler below does.
Install the Encore CLI with the command for your system, then create the app:
Run in your terminal to get started locally.
The migration creates the balance table and the dedupe table:
-- wallet/migrations/1_create_tables.up.sql
CREATE TABLE balances (
account_id TEXT PRIMARY KEY,
cents BIGINT NOT NULL DEFAULT 0
);
CREATE TABLE processed_events (
event_id TEXT PRIMARY KEY,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
The producer sets the event ID from the business key, so a retried publish carries the same ID, and the subscriber applies the credit and records the ID in one transaction:
// wallet/wallet.go
package wallet
import (
"context"
"fmt"
"time"
"encore.dev/pubsub"
"encore.dev/storage/sqldb"
)
var db = sqldb.NewDatabase("wallet", sqldb.DatabaseConfig{Migrations: "./migrations"})
type Credited struct {
EventID string // set by the producer, stable across retries
AccountID string
Cents int64
}
var Credits = pubsub.NewTopic[*Credited]("credits", pubsub.TopicConfig{
DeliveryGuarantee: pubsub.AtLeastOnce,
})
type RefundParams struct {
RefundID string `json:"refund_id"`
AccountID string `json:"account_id"`
Cents int64 `json:"cents"`
}
//encore:api public method=POST path=/refunds
func IssueRefund(ctx context.Context, p *RefundParams) error {
_, err := Credits.Publish(ctx, &Credited{
EventID: "refund:" + p.RefundID,
AccountID: p.AccountID,
Cents: p.Cents,
})
return err
}
var _ = pubsub.NewSubscription(Credits, "apply-credit", pubsub.SubscriptionConfig[*Credited]{
Handler: applyCredit,
RetryPolicy: &pubsub.RetryPolicy{
MinBackoff: 5 * time.Second,
MaxBackoff: 5 * time.Minute,
MaxRetries: 10,
},
})
func applyCredit(ctx context.Context, e *Credited) error {
tx, err := db.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback() // no-op after Commit
res, err := tx.Exec(ctx, `
INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING`, e.EventID)
if err != nil {
return err
}
if res.RowsAffected() == 0 {
return nil // already applied; acknowledge the duplicate
}
if _, err := tx.Exec(ctx, `
INSERT INTO balances (account_id, cents) VALUES ($1, $2)
ON CONFLICT (account_id) DO UPDATE SET cents = balances.cents + EXCLUDED.cents`,
e.AccountID, e.Cents); err != nil {
return fmt.Errorf("credit %s: %w", e.AccountID, err)
}
return tx.Commit()
}
Calling the endpoint twice with the same refund ID publishes two messages and credits the account once (in production this endpoint sits behind an auth handler), which you can see in the local trace for each delivery. A handler error rolls back the transaction, and the message is retried with backoff and then moved to the subscription's dead-letter queue. On deploy, Encore provisions the SNS topic, the SQS queue and its dead-letter queue, and RDS in your AWS account, and Encore provisions an IAM role for the service, or a service account on GCP. A coding agent adding a second consumer of Credits gets the same at-least-once semantics, so the pattern above, with its own dedupe table, is what to ask it to copy.
The database docs cover migrations, queries and transactions in more detail.
Best fit: Go and TypeScript teams that want queues with retries and dead-letter queues in their own AWS or GCP account and a place to write idempotent handlers without SQS or Pub/Sub plumbing.
An idempotent consumer is a message handler that produces the same result whether it processes a message once or several times. It usually records which messages it has handled, or applies changes that are naturally safe to repeat, so a redelivered message has no extra effect.
Standard SQS queues guarantee at-least-once delivery. A message can be delivered again when a consumer does not delete it before the visibility timeout ends, for example after a crash, a slow handler or a failed delete call, and on rare occasions SQS itself delivers a duplicate copy.
No. SQS FIFO deduplicates messages with the same deduplication ID sent within a 5-minute window, and Pub/Sub exactly-once delivery prevents redelivery after a successful acknowledgement within a region. Neither covers a handler that crashes after its side effect but before acknowledging, so the handler still needs to be safe to run twice.
Insert the message's event ID into a processed-events table with a unique constraint, in the same database transaction as the change itself. If the insert finds the ID already present, skip the change and acknowledge the message. If the transaction fails, neither the change nor the record is kept, so the retry starts clean.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.