How to run A/B tests from the backend: deterministic variant assignment by hashing, logging exposures once, tying conversions to variants, checking for sample ratio mismatch and reading results, with a TypeScript example on Encore and Postgres.

A/B testing tools hide most of the backend work, which is fine until the test has to cover a pricing rule, a ranking algorithm or an email flow that the browser never sees. For those, the experiment has to run where the decision is made.
The backend part is small: assign a variant consistently, record who saw it, record what they did and count. This guide covers each step, the mistakes that invalidate results and a TypeScript service on Postgres.
Make assignment a pure function of the user ID and the experiment name. Hash the two together, take the result modulo 100 and compare it with the split:
import { createHash } from "node:crypto";
function bucket(userId: string, experiment: string): number {
const h = createHash("sha256").update(`${experiment}:${userId}`).digest();
return h.readUInt32BE(0) % 100;
}
The same user gets the same variant on every request and every instance, with nothing stored. Including the experiment name in the hash keeps experiments independent, so being in the treatment group of one test says nothing about another. Use a stable ID, such as the account ID, for logged-in experiments and an anonymous ID cookie otherwise.
Log an exposure the first time a user is shown the variant, once per user and experiment. If you log at assignment time, users who were assigned but never reached the page dilute both groups, and the effect shrinks toward zero. Make the insert idempotent with a unique constraint on user and experiment and ON CONFLICT DO NOTHING, so repeats are ignored.
Record conversion events with the user ID, the event name and a timestamp, and join them to exposures in SQL, counting only conversions that happened after the exposure. Decide the primary metric before the test starts, along with the duration or sample size, because choosing a metric after seeing results guarantees a winner somewhere.
Peeking is the common one: checking significance every day and stopping when it dips below 0.05 inflates false positives, as Evan Miller explains. Fix the sample size in advance, using a calculator such as Evan Miller's sample size calculator, and look at results when it is reached.
The second is sample ratio mismatch: compare exposure counts per variant with the planned split using a chi-squared test, and treat a failure as a bug in assignment or logging, not noise. Common causes are a variant that crashes before logging, bots filtered in one arm only and redirects that drop users.
Encore is a backend framework with built-in infrastructure, and the experiment service is an Encore service with a Postgres database declared in code. encore run starts Postgres locally, and on deploy Encore provisions RDS or Cloud SQL in your own cloud account.
Run in your terminal to get started locally.
-- experiments/migrations/1_create_events.up.sql
CREATE TABLE exposures (
experiment TEXT NOT NULL,
user_id TEXT NOT NULL,
variant TEXT NOT NULL,
at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (experiment, user_id)
);
CREATE TABLE conversions (
id BIGSERIAL PRIMARY KEY,
experiment TEXT NOT NULL,
user_id TEXT NOT NULL,
event TEXT NOT NULL,
at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX conversions_lookup ON conversions (experiment, user_id, event);
// experiments/experiments.ts
import { api } from "encore.dev/api";
import { SQLDatabase } from "encore.dev/storage/sqldb";
import { createHash } from "node:crypto";
const db = new SQLDatabase("experiments", { migrations: "./migrations" });
// Weights in percent, summing to 100.
const experiments: Record<string, { variants: [string, number][] }> = {
"new-checkout": { variants: [["control", 50], ["treatment", 50]] },
};
function assign(experiment: string, userId: string): string {
const exp = experiments[experiment];
const h = createHash("sha256").update(`${experiment}:${userId}`).digest();
const b = h.readUInt32BE(0) % 100;
let acc = 0;
for (const [name, weight] of exp.variants) {
acc += weight;
if (b < acc) return name;
}
return exp.variants[0][0];
}
export const variant = api(
{ method: "POST", path: "/experiments/:experiment/exposure", expose: true },
async ({ experiment, userId }: { experiment: string; userId: string }): Promise<{ variant: string }> => {
const v = assign(experiment, userId);
await db.exec`
INSERT INTO exposures (experiment, user_id, variant) VALUES (${experiment}, ${userId}, ${v})
ON CONFLICT DO NOTHING`;
return { variant: v };
},
);
export const convert = api(
{ method: "POST", path: "/experiments/:experiment/conversion", expose: true },
async ({ experiment, userId, event }: { experiment: string; userId: string; event: string }): Promise<void> => {
await db.exec`INSERT INTO conversions (experiment, user_id, event) VALUES (${experiment}, ${userId}, ${event})`;
},
);
export const results = api(
{ method: "GET", path: "/experiments/:experiment/results", expose: false },
async ({ experiment }: { experiment: string }): Promise<{ rows: { variant: string; exposed: number; converted: number }[] }> => {
const rows: { variant: string; exposed: number; converted: number }[] = [];
for await (const r of db.query<{ variant: string; exposed: string; converted: string }>`
SELECT e.variant,
count(*) AS exposed,
count(*) FILTER (WHERE EXISTS (
SELECT 1 FROM conversions c
WHERE c.experiment = e.experiment AND c.user_id = e.user_id
AND c.event = 'purchase' AND c.at >= e.at)) AS converted
FROM exposures e WHERE e.experiment = ${experiment} GROUP BY e.variant`) {
rows.push({ variant: r.variant, exposed: Number(r.exposed), converted: Number(r.converted) });
}
return { rows };
},
);
The exposure endpoint is idempotent through ON CONFLICT DO NOTHING, and the results query counts a user as converted only if the purchase came after exposure. Run a chi-squared test on the exposed column against the planned split before reading conversion rates. The results endpoint is not exposed publicly, so it is reachable from other services and the local development dashboard. Add an auth handler so the user ID comes from the session instead of the request body, and each request appears as a trace with its SQL.
A coding agent can add a new experiment entry and the code path for each variant in one change, then test both arms locally by calling the exposure endpoint with different user IDs.
Best fit: TypeScript teams that need experiments on server-side decisions and want assignment, events and results in their own Postgres database.
Hash the user ID together with the experiment key and map the hash to a bucket from 0 to 99, then compare it to the traffic split. The same user always gets the same variant without storing an assignment, and different experiments split users independently.
Log it when the user sees or receives the variant, once per user and experiment, not when the assignment is computed. Counting people who were assigned but never reached the feature dilutes the effect and makes results harder to interpret.
It means the observed split between variants differs from the intended one more than chance explains, for example 50/50 planned and 52/48 observed on a large sample. It usually points to a bug in assignment or logging, and results from such a test should not be trusted until it is fixed.
Write the assignment function, an exposure endpoint and a conversion endpoint in an Encore service backed by Postgres. Assignment is a pure function of the user ID and experiment key, so it behaves the same under encore run and in your AWS or GCP environments.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.