Most developers have seen a trace viewer, the waterfall chart with colored bars and nested spans. But if you asked ten developers to explain what a trace actually contains, what data is captured at each stage and how spans relate to each other across services, you'd get ten different answers. That's partly because tracing tools show you the finished product without explaining how it was built. And partly because in most systems, what a trace captures depends on what someone remembered to instrument, so the structure is different every time.

We've previously written about Tracing Reimagined, how distributed tracing should be built into the runtime rather than being an afterthought. This post goes deeper, walking through what a complete distributed trace actually looks like when every operation is instrumented automatically, walked through one step at a time.

The request

A user sends a POST request to shorten a URL. The request hits a url service, which checks authentication, queries the database, writes to a cache, calls an analytics service, and publishes a message to a Pub/Sub topic. The analytics service writes to its own database and makes an outbound HTTP call to Mixpanel. Three services, ten operations, one trace.

In Encore, all of these operations are traced automatically by the runtime. There's no SDK to configure, no spans to create manually, and no context to propagate by hand. The trace below is what you get by default.

Each colored bar in the waterfall represents a span, a unit of work with a start time, an end time, and metadata about what happened. Spans nest: the root span is the HTTP request itself, every operation that happens during that request becomes a child span, and operations within those operations become grandchildren. A trace ID ties them all together.

What each span type captures

The runtime captures different data depending on the operation.

API calls record the HTTP method, path, request headers, request body, response status, and response body. When one service calls another, both sides get their own span. The caller gets an outbound RPC span, and the callee gets a new root-level span linked to the same trace. Trace context is propagated automatically in the request headers.

Database queries capture the full SQL text with bound parameters. When using Encore's SQL databases, you see SELECT * FROM urls WHERE id = 'x7k9m2' in the trace, not SELECT * FROM urls WHERE id = ?. That's the difference between knowing a database query happened and knowing exactly what it did. Row counts and execution times are included.

Pub/Sub operations capture the topic name, message ID, and payload on publish. When using Encore's Pub/Sub, if a subscriber processes that message later, potentially minutes later in a different service, its trace links back to the original publish span. You can follow a message from publish through to the subscriber's processing, across async boundaries.

Cache operations capture the key, operation type (GET, SET, DELETE), the result (hit, miss, error), and the TTL. Most teams don't bother instrumenting cache calls manually, which makes them invisible when debugging cache-related latency issues.

Outbound HTTP calls capture the URL, method, status code, and timing of requests to third-party APIs. When Stripe or Mixpanel is slow, you see it in the trace.

Auth handlers capture the authentication result and attach user identity to the trace. When using Encore's Auth handlers, every subsequent span becomes searchable by user ID, which is useful when you need to debug what happened for a specific user.

How spans nest

The nesting structure is what gives distributed tracing its value. A flat list of events ("database query took 14ms, HTTP call took 64ms") tells you what happened. The tree structure tells you the causal chain.

When the url service calls analytics.track, the runtime creates a child span under the root. Inside the analytics service, the database query and the Mixpanel HTTP call become children of that service call span:

POST /shorten (187ms)
├─ auth.verify (17ms)
├─ SELECT url FROM urls... (7ms)
├─ INSERT INTO urls... (14ms)
├─ cache.set url:x7k9m2 (3ms)
├─ analytics.track (86ms)
│  ├─ INSERT INTO events... (13ms)
│  └─ POST api.mixpanel.com/track (64ms)
└─ url-events → publish (7ms)

The total request took 187ms. The analytics call accounts for 86ms of that. Within the analytics call, 64ms is the Mixpanel HTTP request. If this endpoint's latency increases, the trace tree shows exactly where the time is going.

From individual traces to patterns

A single trace tells you what happened to one request. Aggregating traces across thousands of requests tells you how your system behaves.

If your p50 is 42ms and your p99 is 480ms, something is making 1% of your requests ten times slower. With complete traces, you can filter to those slow requests and compare their span trees to the fast ones. The answer is usually a specific database query, a third-party API call, or a cache miss that triggers a cold code path.

In Encore Cloud, the new Trace Explorer lets you filter by latency percentile, compare across time ranges, correlate spikes with deploys, and group by service or endpoint. It's built on ClickHouse, so the queries are fast even across millions of spans.

The same trace, local and production

These aren't production-only features. When you run encore run during development, the same tracing runs locally at localhost:9400. You get the same waterfall, the same span details, the same query capture.

Most debugging happens during development, and most development environments have no tracing at all. When every local request is fully traced, the workflow changes. You look at what your code actually does across service boundaries instead of reading source code and guessing.

What completeness enables

When every database query, every service call, and every cache operation is captured without anyone having to remember to instrument it, traces become a reliable source of truth about system behavior.

You can answer "why was this request slow?" without first asking "was the slow part instrumented?" You can search for all requests by a specific user across all services. You can compare latency distributions before and after a deploy and see which endpoint regressed. You can follow a pub/sub message from publish through subscribe to processing.

None of that works when traces have gaps. With manual instrumentation, they always do.

Encore is an open-source backend framework where infrastructure is declared in TypeScript or Go and provisioned from the code. Tracing is built into the runtime. GitHub.