Horizontal Scaling for Backends: Stateless Services, Sessions and Databases

Horizontal scaling means running more copies of the same service and spreading requests across them. Vertical scaling means giving one process more CPU and RAM. Cloud backends lean on the horizontal model because instances fail, traffic spikes, and a single box has a finite size.

The hard part is rarely starting a second container but making sure any replica can handle any request, that sessions and uploads do not live only in local memory, and that the database and queues survive the extra load. This guide covers those constraints and how Encore (infrastructure from code) structures services and infrastructure for that model in your own cloud.

Horizontal scaling at a glance

Stateless services and sticky sessions

A service is effectively stateless when any healthy replica can serve the next request. Request-scoped data lives in the request; durable data lives in a database, cache or object store that every replica can reach. That shared-nothing stance is what lets a load balancer use round-robin or least-outstanding-requests without caring which instance handled the previous call.

Sticky sessions (session affinity) pin a client to one target, usually with a cookie. On an Application Load Balancer that is a target-group stickiness setting. They buy time when session data still sits in process memory, and they create uneven load and painful deploys: drain that instance and those users lose their session unless you also replicate memory. Prefer a shared session store such as Redis, or move to JWTs / opaque tokens validated against a central store, then turn stickiness off.

Local disk is the same class of problem. A multipart upload written to /tmp on one task is invisible to the next task. Put blobs in object storage and keep only keys in the database.

The database as the bottleneck

Adding app replicas multiplies database connections and write pressure. Ten tasks with a pool of twenty connections each are two hundred sessions on Postgres before you count migrations and admin tools. Connection pooling (on the app, or via a proxy such as RDS Proxy) and a hard cap per service matter as much as instance size.

Reads and writes scale differently. Read replicas absorb reporting and fan-out reads with replication lag as the tradeoff. Writes still go to one primary in typical relational setups, so hot rows, missing indexes and long transactions show up as soon as the API tier grows. Queues help by moving slow work off the request path: the API enqueues a job, returns quickly, and workers scale on their own.

Autoscaling concepts follow from metrics. Application Auto Scaling target tracking on CPU, memory or request count per target raises or lowers desired count between a minimum and maximum for services such as ECS. Scale-out without a matching plan for the database moves the bottleneck to the database. RDS Proxy is one way to keep connection counts under control when many replicas open pools.

How does autoscaling add replicas on Kubernetes?

On Kubernetes, the Horizontal Pod Autoscaler changes the replica count of a Deployment to keep a metric near a target. The autoscaling/v2 API supports CPU and memory utilization relative to the pod's resource requests, plus custom and external metrics such as queue depth:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Utilization targets only work when the pods declare CPU requests, and the metrics server has to be installed in the cluster. Keep minReplicas at two or more for anything user-facing, so a single pod restart does not take the service down, and pair scale-out with a readiness probe so new pods receive traffic only after they are ready.

Scale-in deserves the same care as scale-out. A replica that is removed while it still holds in-flight requests or an unacknowledged queue message causes retries, so handlers should finish or hand back work on SIGTERM within the termination grace period, which defaults to 30 seconds.

How Encore services scale in your cloud

Encore combines an open source infrastructure SDK for TypeScript and Go with a platform that provisions and deploys into your own AWS or GCP account. Each service you declare becomes a deployable unit on Fargate or Kubernetes. Databases, Pub/Sub, object storage, caches, cron jobs and secrets are declared next to the code and provisioned as managed primitives that every replica of that service shares, as described in the AWS infrastructure docs and infrastructure configuration.

Process allocation decides how services map to containers. Running several services in one process cuts cost and latency between them. Deploying each service as its own process improves scalability and limits the blast radius when something goes wrong, which the infrastructure configuration docs recommend only for production environments.

On AWS, Postgres is RDS by default, Pub/Sub maps to SNS and SQS, object storage to S3, and cache to ElastiCache for Redis. Encore provisions an IAM role for the service. Deploys are rolling updates, and the load balancer routes traffic only to healthy targets.

Set up the app

A service that accepts a request, writes to Postgres and publishes to a topic stays safe under horizontal scale: any replica can run the handler, the database holds durable state, and workers compete on the subscription. The service also needs a notify/encore.service.ts file exporting new Service("notify") and a migration that creates the users table:

// notify/events.ts
import { api } from "encore.dev/api";
import { Topic } from "encore.dev/pubsub";
import { SQLDatabase } from "encore.dev/storage/sqldb";

const db = new SQLDatabase("notify", { migrations: "./migrations" });

export interface SignupEvent {
  userId: string;
  email: string;
}

// SNS topic + SQS subscriptions on AWS; in-process delivery locally.
export const signups = new Topic<SignupEvent>("signups", {
  deliveryGuarantee: "at-least-once",
});

export const register = api(
  { expose: true, method: "POST", path: "/signup" },
  async ({ email }: { email: string }): Promise<{ userId: string }> => {
    const row = await db.queryRow<{ id: string }>`
      INSERT INTO users (email) VALUES (${email}) RETURNING id::text AS id`;
    const userId = row!.id;
    await signups.publish({ userId, email });
    return { userId };
  },
);

encore run starts the API, local Postgres and a local Pub/Sub implementation. On deploy, Encore creates the managed resources in your account. The TypeScript Pub/Sub docs and database docs cover the APIs. Connect AWS or GCP with the own-cloud guide.

Best fit: Teams who want services that scale as containers in their own AWS or GCP account, with databases, queues and object storage as shared managed resources.

Good to know

• Cap database connections per service before raising replica counts.
• Keep containers disposable: no unique local state that a restart would lose.
• Separate request handlers from workers when background load grows faster than HTTP traffic.

Related guides

• AWS load balancer guide
• What is Amazon RDS
• Database connection pooling
• ECS vs EC2
• Deploy microservices to AWS