Kubernetes HPA: Horizontal Pod Autoscaling for Backends

The Kubernetes Horizontal Pod Autoscaler (HPA) changes how many pods a Deployment runs so capacity tracks demand. For backends that already sit on a cluster, it is the usual way to grow and shrink the API and worker tiers from live metrics.

Getting the YAML right still leaves the harder part: metrics have to match how the service actually saturates, min and max replicas have to leave room for rollouts and cold starts, and the database has to survive the extra connections when scale-out succeeds. This guide covers those pieces and how Encore scales services in your own AWS or GCP account from infrastructure declared in application code.

HPA at a glance

How HPA chooses a replica count

HPA runs as a control loop in the control plane, by default about every fifteen seconds. It reads the metrics listed on the HorizontalPodAutoscaler, computes a desired replica count for each metric, and picks the largest of those recommendations, then clamps the result between minReplicas and maxReplicas. The core ratio is current metric value over desired metric value, so if average CPU is twice the target, the controller aims to roughly double the pods.

Resource utilization metrics need container resource requests on the pods. Without requests, CPU utilization is undefined and HPA will not act on that metric. The autoscaling/v2 API is the stable surface for memory, multiple metrics and configurable behavior such as scale-up policies and a downscale stabilization window.

For backends, CPU alone is often a poor signal because a service blocked on Postgres or an HTTP client can look idle on CPU while latency climbs. Custom or external metrics (request rate, queue depth, p95 latency) fit those cases once a metrics pipeline exposes them to the custom or external metrics APIs.

Min and max replicas in practice

minReplicas is the count you keep running when traffic is quiet. Setting it to 1 saves money and leaves no spare capacity during a rolling update or a bad deploy. Many production APIs keep a floor of two or three across zones so one pod failure or one surge does not take the service dark. Scaling all the way to zero is a separate path: with recent Kubernetes it is supported for object or external metrics, not for CPU or memory resource metrics, because those can only be measured on running pods.

maxReplicas is the safety rail. Without a ceiling, a bad metric or a retry storm can schedule more pods than the cluster or the database can feed. The max should reflect node capacity, Pod Disruption Budgets and the hard limits of shared dependencies, not only the budget for compute.

When HPA fails for backends

Cold start and readiness delay are the first failure mode. New pods spend time pulling images, running init containers and passing readiness probes before they accept traffic. HPA also treats recently started pods carefully when scaling on CPU, with an initialization period that defaults to five minutes, which dampens scale-up while pods are still warming. A flash crowd can exhaust the Ready pods before the new ones join.

Database and connection limits are the second failure mode: each new pod opens its own pool. Ten pods with twenty connections each are two hundred sessions on Postgres before migrations and admin tools. Scale-out that ignores max_connections, RDS Proxy or a similar pooler turns a healthy HPA event into connection refused errors. Write-heavy paths still hit one primary, so missing indexes and hot rows show up as soon as the API tier grows.

Wrong metrics and thrashing are the third failure mode, where scaling on CPU while the bottleneck is a downstream API, or oscillating around a noisy metric without a stabilization window, burns churn without fixing latency. HPA also cannot place pods if the cluster is out of CPU, memory or IP addresses; that needs a cluster autoscaler or more node capacity beside the workload autoscaler.

How Encore scales services in your cloud

Services you declare in Encore are infrastructure from code, and the platform provisions and deploys them into your own AWS or GCP account. You declare services and the databases, Pub/Sub, object storage, caches, cron jobs and secrets they use in code. On deploy, Encore creates the matching managed resources and runs the services on Fargate or Kubernetes in that account, as described in the AWS infrastructure docs, the Kubernetes deploy docs and infrastructure configuration.

Encore is not a Kubernetes distribution: when you choose Kubernetes for an environment it deploys each service as a Deployment on GKE or EKS Fargate, and when you choose Fargate or Cloud Run it runs containers without you maintaining cluster manifests. Process allocation decides whether services share one process or each get their own, so a hot API can scale independently of a quieter worker. You review compute and database settings per environment in the dashboard, and Encore provisions an IAM role for the service.

Set up the app

A service that keeps durable state in Postgres and offloads slow work to a topic stays safe when replica counts rise, because any healthy instance can handle the next request:

// checkout/encore.service.ts
import { Service } from "encore.dev/service";
export default new Service("checkout");

// checkout/checkout.ts
import { api } from "encore.dev/api";
import { Topic } from "encore.dev/pubsub";
import { SQLDatabase } from "encore.dev/storage/sqldb";

const db = new SQLDatabase("checkout", { migrations: "./migrations" });

export interface OrderPlaced {
  orderId: string;
}

export const orders = new Topic<OrderPlaced>("orders", {
  deliveryGuarantee: "at-least-once",
});

export const place = api(
  { expose: true, method: "POST", path: "/orders" },
  async ({ sku }: { sku: string }): Promise<{ orderId: string }> => {
    const row = await db.queryRow<{ id: string }>`
      INSERT INTO orders (sku) VALUES (${sku}) RETURNING id::text AS id`;
    const orderId = row!.id;
    await orders.publish({ orderId });
    return { orderId };
  },
);

-- checkout/migrations/1_create_orders.up.sql
CREATE TABLE orders (
  id BIGSERIAL PRIMARY KEY,
  sku TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

encore run starts Postgres locally and runs the service against a local Pub/Sub implementation. After you connect your cloud account, a push deploys the same code with managed Postgres and messaging in your account. The TypeScript database docs and Pub/Sub docs cover the APIs.

An AI coding agent can add a second service or a subscription in the same change, run it with encore run, and leave replica and database sizing to the environment configuration rather than authoring HorizontalPodAutoscaler manifests next to the handler.

Best fit: Teams who want backends that scale as services in their own AWS or GCP account, with infrastructure from code and shared managed databases, queues and storage.

Good to know

• Set resource requests before trusting CPU or memory utilization targets.
• Raise database connection budgets and pooling before raising maxReplicas.
• Prefer request, queue or latency metrics when the bottleneck is I/O rather than CPU.

Related guides

• Do you need Kubernetes?
• ECS vs EKS
• What is AWS Fargate?
• Deploy to AWS without Kubernetes
• Deploy microservices to AWS