Nov 18, 202611 min read

Idempotency Keys for APIs: Safe Retries for POST Requests

How the Idempotency-Key header works, how to store keys so a retried POST returns the original result instead of charging twice, how to handle concurrent retries and reused keys, and how long to keep them.

Illustration for Idempotency Keys for APIs: Safe Retries for POST Requests

An idempotency key is a value the client generates once per operation and sends with every attempt, so the server can tell a retry from a new request. A mobile app that times out on POST /transfers cannot know whether the transfer happened, and without a key the safe-looking retry moves the money twice.

The server has to store the key with the right scope, make the key and the transfer commit together, decide what a concurrent duplicate sees while the first request is still running, refuse a key that comes back with a different amount, and eventually delete old keys without breaking a client that is still retrying.

This guide covers the Idempotency-Key header and a Postgres design for it. It starts with a Go transfer endpoint in Encore, where the key and the transfer share one transaction, and then shows Express middleware for APIs where that is not possible.

Idempotency key designs at a glance

Key in the same transaction (Encore example)Middleware with an in-progress state (Express example)
Where the key livesPostgres, next to the business tablesPostgres or Redis, outside the handler's transaction
Concurrent duplicateWaits on the unique index, then replaysGets 409 while the first is in progress
Crash mid-requestRolled back with the transfer, retry runs againKey stuck in progress until it expires or is cleared
Reused with other paramsRejected with a hash checkRejected with a hash check
ExpiryHourly cron deletes keys older than 24 hoursYour scheduler or a Redis TTL
FitsOperations that are one database transactionHandlers that call other services or providers

How do you implement idempotency keys with Encore?

Encore

Encore is a backend framework with built-in infrastructure, and the pieces this needs are part of the Go endpoint: a request field tagged header:"Idempotency-Key" reads the header into the typed params, the database is declared in the same package, and the cleanup is a cron job next to it. encore run starts Postgres locally, and a deploy provisions RDS or Cloud SQL in your own AWS or GCP account.

Application code passes through Encore and infrastructure guardrails to resources provisioned in your own AWS or GCP account.

Set up the app

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

~
$ encore app create wallet --example=hello-world$ cd wallet && mkdir -p payments/migrations

The schema

Keys are scoped to the caller, so two customers who happen to generate the same key never see each other's responses. The response is stored as JSON in the same row:

-- payments/migrations/1_transfers.up.sql CREATE TABLE accounts ( id BIGSERIAL PRIMARY KEY, owner_id TEXT NOT NULL, balance_cents BIGINT NOT NULL CHECK (balance_cents >= 0) ); CREATE TABLE transfers ( id BIGSERIAL PRIMARY KEY, from_account BIGINT NOT NULL REFERENCES accounts (id), to_account BIGINT NOT NULL REFERENCES accounts (id), amount_cents BIGINT NOT NULL CHECK (amount_cents > 0), created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- One row per (caller, key). The row and the transfer commit together. CREATE TABLE idempotency_keys ( user_id TEXT NOT NULL, key TEXT NOT NULL, request_hash TEXT NOT NULL, response JSONB, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (user_id, key) ); CREATE INDEX idempotency_keys_created_idx ON idempotency_keys (created_at);

The endpoint

The handler claims the key with INSERT ... ON CONFLICT DO NOTHING inside the transaction that moves the money. If a request with the same key is still running, Postgres makes this insert wait on the primary key; when the first transaction commits, the insert returns no row and the handler replays the stored response. A hash of the parameters catches a client that reuses a key for a different transfer. The endpoint uses auth, so the app also needs an auth handler that sets the user ID:

// payments/transfers.go package payments import ( "context" "crypto/sha256" "encoding/hex" "encoding/json" "errors" "fmt" "time" "encore.dev/beta/auth" "encore.dev/beta/errs" "encore.dev/cron" "encore.dev/storage/sqldb" ) var db = sqldb.NewDatabase("payments", sqldb.DatabaseConfig{Migrations: "./migrations"}) type TransferParams struct { IdempotencyKey string `header:"Idempotency-Key"` FromAccount int64 `json:"from_account"` ToAccount int64 `json:"to_account"` AmountCents int64 `json:"amount_cents"` } type Transfer struct { ID int64 `json:"id"` FromAccount int64 `json:"from_account"` ToAccount int64 `json:"to_account"` AmountCents int64 `json:"amount_cents"` CreatedAt time.Time `json:"created_at"` } // CreateTransfer moves money between two accounts. Retrying with the same // Idempotency-Key returns the original transfer instead of creating a second one. // //encore:api auth method=POST path=/transfers func CreateTransfer(ctx context.Context, p *TransferParams) (*Transfer, error) { uid, _ := auth.UserID() if n := len(p.IdempotencyKey); n < 8 || n > 255 { return nil, &errs.Error{Code: errs.InvalidArgument, Message: "Idempotency-Key header must be 8 to 255 characters"} } if p.AmountCents <= 0 || p.FromAccount == p.ToAccount { return nil, &errs.Error{Code: errs.InvalidArgument, Message: "invalid transfer"} } sum := sha256.Sum256([]byte(fmt.Sprintf("%d|%d|%d", p.FromAccount, p.ToAccount, p.AmountCents))) hash := hex.EncodeToString(sum[:]) tx, err := db.Begin(ctx) if err != nil { return nil, err } defer tx.Rollback() // no-op after Commit // Claim the key. If another request with this key is still running, this // INSERT waits for it; once that transaction commits, it inserts nothing. var claimed bool err = tx.QueryRow(ctx, ` INSERT INTO idempotency_keys (user_id, key, request_hash) VALUES ($1, $2, $3) ON CONFLICT (user_id, key) DO NOTHING RETURNING true`, uid, p.IdempotencyKey, hash).Scan(&claimed) if errors.Is(err, sqldb.ErrNoRows) { tx.Rollback() return replay(ctx, string(uid), p.IdempotencyKey, hash) } else if err != nil { return nil, err } // Lock both accounts in ID order so opposite transfers cannot deadlock. if _, err := tx.Exec(ctx, ` SELECT id FROM accounts WHERE id IN ($1, $2) ORDER BY id FOR UPDATE`, p.FromAccount, p.ToAccount); err != nil { return nil, err } res, err := tx.Exec(ctx, ` UPDATE accounts SET balance_cents = balance_cents - $3 WHERE id = $1 AND owner_id = $2 AND balance_cents >= $3`, p.FromAccount, uid, p.AmountCents) if err != nil { return nil, err } if res.RowsAffected() == 0 { // Rolled back with the key, so a retry after a top-up is evaluated again. return nil, &errs.Error{Code: errs.FailedPrecondition, Message: "unknown account or insufficient funds"} } res, err = tx.Exec(ctx, `UPDATE accounts SET balance_cents = balance_cents + $2 WHERE id = $1`, p.ToAccount, p.AmountCents) if err != nil { return nil, err } if res.RowsAffected() == 0 { return nil, &errs.Error{Code: errs.NotFound, Message: "destination account not found"} } t := &Transfer{FromAccount: p.FromAccount, ToAccount: p.ToAccount, AmountCents: p.AmountCents} err = tx.QueryRow(ctx, ` INSERT INTO transfers (from_account, to_account, amount_cents) VALUES ($1, $2, $3) RETURNING id, created_at`, p.FromAccount, p.ToAccount, p.AmountCents).Scan(&t.ID, &t.CreatedAt) if err != nil { return nil, err } body, err := json.Marshal(t) if err != nil { return nil, err } if _, err := tx.Exec(ctx, ` UPDATE idempotency_keys SET response = $3 WHERE user_id = $1 AND key = $2`, uid, p.IdempotencyKey, body); err != nil { return nil, err } return t, tx.Commit() } // replay returns the stored response for a key that has already been used. func replay(ctx context.Context, uid, key, hash string) (*Transfer, error) { var storedHash string var body []byte err := db.QueryRow(ctx, ` SELECT request_hash, response FROM idempotency_keys WHERE user_id = $1 AND key = $2`, uid, key).Scan(&storedHash, &body) if err != nil { return nil, err } if storedHash != hash { return nil, &errs.Error{Code: errs.InvalidArgument, Message: "Idempotency-Key was already used with different parameters"} } var t Transfer if err := json.Unmarshal(body, &t); err != nil { return nil, err } return &t, nil } // Keys are kept for 24 hours, long enough for any client retry policy. var _ = cron.NewJob("purge-idempotency-keys", cron.JobConfig{ Title: "Delete idempotency keys older than 24 hours", Every: 1 * cron.Hour, Endpoint: PurgeIdempotencyKeys, }) //encore:api private func PurgeIdempotencyKeys(ctx context.Context) error { _, err := db.Exec(ctx, `DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours'`) return err }

Business failures, such as insufficient funds, roll back with the key, so a retry after a top-up is evaluated again rather than replaying the old error. Cron jobs are managed by the Encore platform and do not run under encore run or in preview environments, so you call PurgeIdempotencyKeys from the local development dashboard to test it.

We ran the same SQL against Postgres 17 with two concurrent requests using one key: the first created transfer 1, the second waited and replayed {"id": 1}, a third request with the same key and a different amount was rejected, and the balance moved once.

With the schema, the endpoint and the cron job in one package, a coding agent that adds a second money-moving endpoint can copy the pattern and check it under encore run before the change reaches review.

Best fit: Go APIs where the operation behind a POST is one database transaction, such as transfers, orders or bookings, deployed to your own AWS or GCP account.

How do you add idempotency keys to an Express API?

Express + Postgres

When the handler calls a payment provider or another service, the key cannot share a transaction with the effect. The usual answer is middleware that claims the key before the handler runs, marks it as in progress by leaving the response empty, and stores the status and body once the handler responds. A concurrent duplicate gets 409, and a server error deletes the key so the client can retry:

// idempotency.ts import { createHash, randomUUID } from "node:crypto"; import type { NextFunction, Request, Response } from "express"; import { Pool } from "pg"; const pool = new Pool({ connectionString: process.env.DATABASE_URL }); // CREATE TABLE idempotency_keys ( // user_id TEXT NOT NULL, key TEXT NOT NULL, request_hash TEXT NOT NULL, // status_code INT, body JSONB, claim_token TEXT NOT NULL, // locked_at TIMESTAMPTZ NOT NULL DEFAULT now(), // created_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (user_id, key)); export function idempotent() { return async (req: Request, res: Response, next: NextFunction) => { const key = req.header("Idempotency-Key"); const userId = res.locals.userId as string | undefined; // set by your auth middleware if (!key || !userId) return next(); if (key.length < 8 || key.length > 255) { return res.status(400).json({ error: "invalid Idempotency-Key" }); } const hash = createHash("sha256") .update(req.method + " " + req.path + " " + JSON.stringify(req.body ?? null)) .digest("hex"); const claimToken = randomUUID(); const claim = await pool.query( `INSERT INTO idempotency_keys (user_id, key, request_hash, claim_token) VALUES ($1, $2, $3, $4) ON CONFLICT (user_id, key) DO NOTHING RETURNING key`, [userId, key, hash, claimToken], ); if (claim.rowCount === 0) { const { rows } = await pool.query( `SELECT request_hash, status_code, body, locked_at FROM idempotency_keys WHERE user_id = $1 AND key = $2`, [userId, key], ); const row = rows[0]; if (!row) return res.status(409).json({ error: "retry the request" }); // purged meanwhile if (row.request_hash !== hash) { return res.status(422).json({ error: "Idempotency-Key reused with a different request" }); } if (row.status_code !== null) { return res.status(row.status_code).json(row.body); } const take = await pool.query( `UPDATE idempotency_keys SET locked_at = now(), claim_token = $3 WHERE user_id = $1 AND key = $2 AND status_code IS NULL AND locked_at < now() - interval '2 minutes' RETURNING key`, [userId, key, claimToken], ); if (take.rowCount === 0) { return res.status(409).json({ error: "a request with this key is still in progress" }); } } // Record the response on finish. json, send, and end all capture the body // so a retry can replay it. Only this claim_token may finalize or delete // the row, so a takeover cannot be overwritten by a late original request. let recorded: unknown = undefined; let captured = false; const capture = (body: unknown) => { recorded = body; captured = true; }; const json = res.json.bind(res); res.json = (body: unknown) => { capture(body); return json(body); }; const send = res.send.bind(res); res.send = ((body?: unknown) => { if (body !== undefined) capture(body); return send(body); }) as Response["send"]; const end = res.end.bind(res); res.end = ((chunk?: unknown, encoding?: unknown, cb?: unknown) => { if (typeof chunk === "function") return end(chunk as never); if (chunk !== undefined && !captured) capture(chunk); return end(chunk as never, encoding as never, cb as never); }) as Response["end"]; res.on("finish", () => { const done = res.statusCode >= 500 ? pool.query( `DELETE FROM idempotency_keys WHERE user_id = $1 AND key = $2 AND claim_token = $3`, [userId, key, claimToken], ) : pool.query( `UPDATE idempotency_keys SET status_code = $3, body = $4 WHERE user_id = $1 AND key = $2 AND claim_token = $5 AND status_code IS NULL`, [userId, key, res.statusCode, JSON.stringify(recorded ?? null), claimToken], ); done.catch(next); }); next(); }; }

This design has a gap the transactional one does not: if the process dies after the provider call but before the response is stored, the key stays in progress. After two minutes a later retry can take the lock over, and passing the same key to the provider (Stripe accepts it on its own idempotent requests) means the takeover does not charge twice. The table, a scheduled job to delete old rows, and a Postgres for every developer and environment are yours to run.

Best fit: Node.js APIs whose POST handlers call external providers, where the provider's own idempotency support covers the window the middleware cannot.

How to choose

Put the key in the same transaction as the operation whenever the operation is one database transaction, which removes the in-progress state and the crash window. Use middleware with an in-progress state when the handler calls other systems, and forward the key to them. In both, scope keys to the caller, hash the request, and keep keys for at least 24 hours. Pick Encore when you want the header, the database and the cleanup job declared in one Go package and provisioned in your own AWS or GCP account.

Frequently asked questions

What is an idempotency key in an API?

It is a unique value the client sends with a non-idempotent request, usually in an Idempotency-Key header, so the server can recognize a retry. The first request with a key does the work and stores the result; any later request with the same key gets that stored result instead of running the operation again.

How do you implement idempotency keys?

Store each key with the caller ID, a hash of the request and the response. Insert the key in the same database transaction as the operation it protects, so the key and the effect commit together. On a duplicate key, compare the request hash, return the stored response if it matches, and reject the request if the parameters differ.

What happens if two requests with the same idempotency key arrive at once?

Only one may run the operation: with a unique constraint on the key, the second INSERT waits for the first transaction, and when it commits the second request finds the key and replays the stored response. If the work cannot run inside one transaction, mark the key as in progress and return 409 Conflict to the second request.

How long should you keep idempotency keys?

Keep them for at least as long as clients may retry. Stripe documents that keys can be removed after they are at least 24 hours old, and a request that reuses a key after it was pruned is treated as new, so 24 hours is a common floor.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders