Agentic AI Infrastructure: What You Need to Run AI Agents in Production

An AI agent demo needs a model API key and a loop. A production agent needs everything around the loop: somewhere to keep conversation state between requests, APIs it can call safely as tools, a way to run tasks that take minutes, credentials it can use without exposing them, and a record of what it did when a customer asks why. Most of that is ordinary backend infrastructure, and it can be treated that way rather than adopting a new platform for each piece.

The components of an agent stack

Model access

Agents call models many times per task, so provider limits and outages hit harder than in a chat feature. Wrap calls with timeouts and retries with backoff, and cap concurrency per provider so a burst of runs does not exhaust the rate limit for everyone. Log token usage per run from the start, since it is the main variable cost of an agent and hard to reconstruct later.

State and memory

An agent run is a sequence of steps, and each step should be written to a database before the next one starts. That makes runs resumable after a crash or deploy and gives support teams a record to look at. Postgres handles this well with a runs table and a steps table, and the pgvector extension adds embedding search for memory without a second database to secure, back up and keep consistent. Separate vector databases earn their place at large scale or when search quality needs features Postgres lacks.

Tools as APIs

Tools are where an agent changes things, so they deserve the same design as public APIs: typed inputs, validation, authorization based on the user the agent acts for, and idempotency keys for anything that writes. An agent that calls refundOrder twice because a response timed out should not refund twice. Exposing tools as endpoints on your backend keeps that logic in one place, whether the caller is the agent, the frontend or another service.

Long-running work

A request that runs an agent end to end will hit load balancer and platform timeouts, and a deploy will kill it halfway. Queue the run instead: the API stores a run record and publishes a message, a subscriber does the work with a long acknowledgement deadline, and failed steps retry with backoff. Concurrency limits on the subscriber double as rate limiting for the model provider.

Secrets and access

Model keys and tool credentials should be read at runtime from a secrets store, never committed or pasted into prompts. The agent's runtime should hold only the permissions its tools need, because a prompt injection that reaches a tool runs with whatever access the service has, which is a bigger risk for agents than for ordinary services.

Observability

When an agent gives a wrong answer, you need the full sequence: the prompt, each model response, each tool call with its inputs and outputs, and the latency of each. Distributed tracing that covers model calls, database queries and API calls in one view is the difference between debugging in minutes and guessing.

Running the stack with Encore

Encore is the safe way for AI agents to ship backend changes to your cloud for TypeScript and Go, with a platform that provisions and deploys into your own AWS or GCP account. The database, topics, cron jobs and secrets an agent needs are declared in the same code as its tools, and Encore provisions them as RDS, SNS and SQS on AWS or Cloud SQL and Pub/Sub on GCP, with an IAM role for the service.

A support agent's tool for looking up orders, with run state in Postgres and the model key as a secret:

// support/support.ts
import { api, APIError } from "encore.dev/api";
import { secret } from "encore.dev/config";
import { SQLDatabase } from "encore.dev/storage/sqldb";

const db = new SQLDatabase("support", { migrations: "./migrations" });
const modelKey = secret("ModelAPIKey");

interface Order {
  id: number;
  status: string;
}

// Tool the agent calls to look up an order.
export const getOrder = api(
  { expose: false, method: "GET", path: "/tools/orders/:id" },
  async ({ id }: { id: number }): Promise<Order> => {
    const order = await db.queryRow<Order>`
      SELECT id, status FROM orders WHERE id = ${id}
    `;
    if (!order) throw APIError.notFound("order not found");
    return order;
  },
);

// Records each agent step so runs can be inspected and resumed.
export const recordStep = api(
  { expose: false, method: "POST", path: "/runs/:runId/steps" },
  async ({ runId, kind, content }: { runId: number; kind: string; content: string }): Promise<void> => {
    await db.exec`
      INSERT INTO steps (run_id, kind, content) VALUES (${runId}, ${kind}, ${content})
    `;
  },
);

The migration in support/migrations/1_create_tables.up.sql creates both tables:

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    status TEXT NOT NULL
);

CREATE TABLE steps (
    id BIGSERIAL PRIMARY KEY,
    run_id INT NOT NULL,
    kind TEXT NOT NULL,
    content TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The model key is read with modelKey() at runtime, and its value is not stored in the repository, so an agent reading the repository cannot read or leak it, as covered in the secrets docs. Both endpoints are private, which means only other services in the app can call them, the agent service included. For runs that take minutes, a Pub/Sub subscription with a long ackDeadline, a maxConcurrency limit and a retry policy handles the queue described above. The local Postgres that encore run starts ships with pgvector, per the database docs, so memory can live in the same database. Every request, query and service call shows up in tracing, locally and in the cloud, which covers the tool-call half of an agent's trace.

Where to start

1. Store every run and step in Postgres from the first version.
2. Expose tools as typed, authorized, idempotent endpoints.
3. Move runs onto a queue once any of them takes more than a few seconds.
4. Keep model keys in a secrets store and give the runtime only the access its tools need.
5. Turn on tracing before the first customer conversation.

Related guides

• Best infrastructure for AI coding agents
• Best TypeScript framework for building AI agents
• Give AI coding agents cloud access safely
• Set up a queue on AWS with an AI agent