Tracing

Encore.rs records distributed traces of what your app does with its infrastructure. Since Encore.rs doesn't own your HTTP server, your request handling is traced with your own instrumentation, which Encore.rs supports through OpenTelemetry. This page shows how to connect the two, so a request's trace shows both the request and the queries, messages, and other calls it made.

What Encore traces automatically

Encore records a span for each:

• Postgres query and transaction
• Redis command
• Object storage operation
• Pub/Sub publish, and each message delivered to a subscription

Requests routed through Encore's API gateway also get a span from the gateway, and reach your service with a W3C traceparent header so your own request span can continue the trace.

Each infrastructure span is parented on the current OpenTelemetry context, encore::opentelemetry::Context::current(). Inside a Pub/Sub subscription handler, Encore sets that context to the message delivery's span for you. Everywhere else, including your HTTP handlers, the context is whatever your code has attached. If nothing is attached, each query or publish starts a trace of its own.

Log records written with encore::log carry the IDs of the current context's span too.

The OpenTelemetry API

Encore.rs records spans through its own OpenTelemetry tracer provider, and re-exports the opentelemetry crate (version 0.29) as encore::opentelemetry, so you use the same version as Encore. The encore::telemetry module gives you:

Encore doesn't install its provider as the OpenTelemetry global. To use libraries that record through the global tracer provider, install it at startup:

encore::opentelemetry::global::set_tracer_provider(encore::telemetry::tracer_provider());

Recording a span directly

With the OpenTelemetry API, start a span, put it in a context, and run the work under that context:

use encore::opentelemetry::trace::{FutureExt, Tracer as _};
use encore::opentelemetry::Context;

async fn charge(order: &Order) -> Result<(), encore::Error> {
    let tracer = encore::telemetry::tracer("billing");
    let cx = Context::current_with_span(tracer.start("charge order"));
    // The query's span is a child of "charge order", which ends when `cx` is dropped.
    record_charge(order).with_context(cx).await
}

Tracing an axum app

Most Rust web frameworks are instrumented with the tracing crate. tracing-opentelemetry turns tracing spans into OpenTelemetry spans, and with Encore's tracer they're recorded into Encore's traces. The setup below uses axum and tower-http's TraceLayer, and works the same way for other frameworks built on tracing.

Dependencies

tracing-opentelemetry 0.30 is the release built on opentelemetry 0.29, the version Encore.rs uses:

Cargo.toml
[dependencies]
encore-dev = { path = "/path/to/encore-dev" }
axum = "0.7"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net"] }
tower-http = { version = "0.6", features = ["trace"] }
tracing = "0.1"
tracing-opentelemetry = { version = "0.30", default-features = false }
tracing-subscriber = { version = "0.3", default-features = false, features = ["registry", "std"] }
rand = "0.8"

A tracer for tracing-opentelemetry

tracing-opentelemetry needs a tracer that also implements its PreSampledTracer trait, which reports a span's context before the span is built. Wrap Encore's tracer in one. Tracer::span_context does the work: it fills in the span's IDs and applies Encore's sampling decision.

src/otel.rs
use std::borrow::Cow;

use encore::opentelemetry::trace::{SpanBuilder, SpanId, TraceContextExt, TraceId};
use encore::opentelemetry::Context;
use tracing_opentelemetry::{OtelData, PreSampledTracer};

/// Encore's tracer, with the `PreSampledTracer` tracing-opentelemetry needs.
#[derive(Debug, Clone)]
pub struct Tracer(encore::telemetry::Tracer);

/// A `Tracer` with instrumentation scope `name`.
pub fn tracer(name: impl Into<Cow<'static, str>>) -> Tracer {
    Tracer(encore::telemetry::tracer(name))
}

impl encore::opentelemetry::trace::Tracer for Tracer {
    type Span = <encore::telemetry::Tracer as encore::opentelemetry::trace::Tracer>::Span;

    fn build_with_context(&self, builder: SpanBuilder, parent_cx: &Context) -> Self::Span {
        self.0.build_with_context(builder, parent_cx)
    }
}

impl PreSampledTracer for Tracer {
    fn sampled_context(&self, data: &mut OtelData) -> Context {
        let cx = self.0.span_context(&mut data.builder, &data.parent_cx);
        data.parent_cx.with_remote_span_context(cx)
    }

    fn new_trace_id(&self) -> TraceId {
        loop {
            let id = TraceId::from_bytes(rand::random());
            if id != TraceId::INVALID {
                return id;
            }
        }
    }

    fn new_span_id(&self) -> SpanId {
        loop {
            let id = SpanId::from_bytes(rand::random());
            if id != SpanId::INVALID {
                return id;
            }
        }
    }
}

Installing the subscriber

In main, install a tracing subscriber with the OpenTelemetry layer and the tracer above. Filter it to your own crate's spans, so spans from libraries such as hyper and h2 stay out of your traces:

src/main.rs
mod otel;

use tracing::Level;
use tracing_subscriber::filter::Targets;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::Layer;

#[tokio::main]
async fn main() {
    encore::initialize();
    encore::log::init();

    tracing_subscriber::registry()
        .with(
            tracing_opentelemetry::layer()
                .with_tracer(otel::tracer("my_app"))
                // Leave out the thread and busy/idle time attributes.
                .with_threads(false)
                .with_tracked_inactivity(false)
                .with_filter(Targets::new().with_target("my_app", Level::INFO)),
        )
        .init();

    // ... build the router and serve, below.
}

Replace my_app with your crate's name as it appears in module paths (with _ for -).

A span for each request

tower-http's TraceLayer creates a span for each request. Make it a server span named after the route, with the HTTP semantic-convention attributes, and continue the caller's trace from its traceparent header:

use std::time::Duration;

use axum::extract::{MatchedPath, Request};
use axum::http::{header, HeaderMap};
use axum::response::Response;
use encore::opentelemetry::trace::TraceContextExt;
use encore::opentelemetry::Context;
use tracing::{field, Span};
use tracing_opentelemetry::OpenTelemetrySpanExt;

/// The server span for a request, continuing the caller's trace.
fn request_span(req: &Request) -> Span {
    let method = req.method().as_str();
    let route = req.extensions().get::<MatchedPath>().map(MatchedPath::as_str);
    let user_agent = req
        .headers()
        .get(header::USER_AGENT)
        .and_then(|v| v.to_str().ok());
    let span = tracing::info_span!(
        "request",
        otel.name = format!("{method} {}", route.unwrap_or_default()).trim_end(),
        otel.kind = "server",
        otel.status_code = field::Empty,
        http.request.method = method,
        http.route = route,
        url.path = req.uri().path(),
        user_agent.original = user_agent,
        http.response.status_code = field::Empty,
        "error.type" = field::Empty,
    );
    if let Some(parent) = remote_parent(req.headers()) {
        span.set_parent(parent);
    }
    span
}

/// The caller's span, from the W3C `traceparent` and `tracestate` headers.
fn remote_parent(headers: &HeaderMap) -> Option<Context> {
    let header = |name: &str| headers.get(name).and_then(|v| v.to_str().ok());
    let cx = encore::telemetry::parse_traceparent(header("traceparent")?, header("tracestate"))?;
    Some(Context::new().with_remote_span_context(cx))
}

/// Records the response's status, and marks server errors.
fn record_response<B>(res: &Response<B>, _latency: Duration, span: &Span) {
    let status = res.status();
    span.record("http.response.status_code", status.as_u16());
    if status.is_server_error() {
        span.record("otel.status_code", "ERROR");
        span.record("error.type", status.as_str());
    }
}

Parenting on your request span

tracing-opentelemetry doesn't change the current OpenTelemetry context when a tracing span is entered. So inside your handler, Context::current() isn't the request span, and Encore's query and publish spans wouldn't be its children. Fix that with a middleware that runs each handler under the request span's context, using FutureExt::with_context:

use axum::middleware::Next;
use encore::opentelemetry::trace::FutureExt;

/// Runs the handler under the request span's OpenTelemetry context, so
/// infrastructure calls and log lines are parented on it.
async fn otel_context(req: Request, next: Next) -> Response {
    let cx = Span::current().context();
    next.run(req).with_context(cx).await
}

Then put it together. The middleware sits inside TraceLayer, so Span::current() in it is the request span:

use axum::middleware;
use axum::routing::get;
use axum::Router;
use tower_http::trace::TraceLayer;

let app = Router::new()
    .route("/", get(hello))
    .layer(middleware::from_fn(otel_context))
    .layer(
        TraceLayer::new_for_http()
            .make_span_with(request_span)
            .on_request(())
            .on_response(record_response)
            .on_failure(()),
    );

let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await.unwrap();
axum::serve(listener, app)
    .with_graceful_shutdown(encore::shutdown::signal())
    .await
    .expect("serve");
std::process::exit(encore::shutdown::drained().await.code());

Awaiting encore::shutdown::drained() before exiting also makes sure the last spans are exported. See Graceful shutdown.

Your own spans

A span you create in a handler needs the same treatment: attach its context to the future that makes the infrastructure calls, as well as instrumenting the future with the tracing span:

use encore::opentelemetry::trace::FutureExt;
use tracing::Instrument;
use tracing_opentelemetry::OpenTelemetrySpanExt;

async fn hello() -> String {
    let span = tracing::info_span!("load greeting");
    let cx = span.context();
    let name = latest_user()
        .with_context(cx)
        .instrument(span)
        .await
        .unwrap_or_else(|| "World".to_string());
    format!("Hello, {name}!\n")
}

The query in latest_user is then recorded as a child of load greeting, which is a child of the request span.

Viewing traces

Locally, list the most recent traces and print one with the encore trace command:

$ encore trace list
$ encore trace get <trace-id>

encore trace get takes the 32-character trace ID shown by encore trace list. For deployed environments, traces are shown in the Encore Cloud dashboard. See Distributed Tracing for more about tracing in Encore.