Nov 20, 20265 min read

ECS Fargate Logs in CloudWatch: Setup, Structure and Queries

How to send ECS and Fargate container logs to CloudWatch with the awslogs driver, the IAM and log group settings that commonly break, non-blocking mode, retention, structured JSON logs, Logs Insights queries and FireLens, plus logs that arrive structured and organized by service.

Illustration for ECS Fargate Logs in CloudWatch: Setup, Structure and Queries

On ECS and Fargate, container output goes to CloudWatch Logs through the awslogs driver in the task definition. The first lines appear easily, and then production adds details: a log group per service, retention that defaults to forever and plain text that Logs Insights cannot filter on.

This guide covers the task definition, the common failures, structured logs and queries, and a setup that handles the configuration for you.

How Do You Configure the awslogs Driver?

Each container definition gets a logConfiguration. The ECS CloudWatch logging guide documents every option, and this is a reasonable starting point:

{ "family": "orders-api", "executionRoleArn": "arn:aws:iam::111122223333:role/ordersTaskExecutionRole", "containerDefinitions": [ { "name": "api", "image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/orders-api:1.4.2", "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/orders-api", "awslogs-region": "eu-west-1", "awslogs-stream-prefix": "api", "awslogs-create-group": "true", "mode": "non-blocking", "max-buffer-size": "25m" } } } ] }

Each task then writes to a stream named api/api/TASK_ID, so a log line can be traced back to the exact task that produced it. In blocking mode, a slow or throttled CloudWatch endpoint makes writes to stdout block, which can freeze request handling; non-blocking with a buffer keeps the app running and drops lines only when the buffer fills, which is the better trade for most APIs.

Why Do ECS Logs Fail to Appear?

When a task starts and no stream shows up, check these in order:

  • The task execution role is missing permissions. The awslogs driver runs as the execution role, not the task role, and needs logs:CreateLogStream and logs:PutLogEvents, which the managed AmazonECSTaskExecutionRolePolicy includes. awslogs-create-group additionally needs logs:CreateLogGroup, which that policy does not.
  • The log group does not exist. Without awslogs-create-group, the group must be created before the first task starts, and the task fails to start with a ResourceInitializationError otherwise.
  • The task cannot reach CloudWatch. Tasks in private subnets need a NAT gateway or a logs interface VPC endpoint.
  • The app logs to a file. The driver only captures stdout and stderr, so a logger writing to /var/log/app.log inside the container sends nothing.

How Do You Control Retention and Cost?

New log groups keep events forever, and CloudWatch charges both to ingest and to store logs, as the CloudWatch pricing page lists. Set retention on every group when you create it:

~
$ aws logs put-retention-policy --log-group-name /ecs/orders-api --retention-in-days 30

With awslogs-create-group, ECS creates the group without retention, so either create groups up front in Terraform with retention_in_days, or run a periodic job that sets a policy on groups that have none. Ingestion usually costs more than storage, so the bigger saving is logging less: drop per-request debug lines in production and keep health check requests out of access logs.

How Do You Make ECS Logs Queryable?

Write one JSON object per line, and CloudWatch Logs Insights discovers fields in JSON log events automatically, so a line such as {"level":"error","msg":"charge failed","order_id":"o-1001","duration_ms":812} can be filtered by order_id and aggregated by level without parsing rules:

fields @timestamp, msg, order_id, duration_ms | filter level = "error" | stats count(*) as errors by bin(5m)

Include a request or trace ID on every line so all the lines from one request can be pulled together, and keep field names consistent across services so one query works over several log groups. When some logs need to go elsewhere, such as a third-party log platform or S3, FireLens runs Fluent Bit as a sidecar and routes them, at the cost of one more container per task to configure.

How Does Encore Handle ECS Logs?

Encore

Encore brings infrastructure from code, and when it deploys an app to Fargate in your own AWS account, it sets up CloudWatch logging for every service, with logs structured and organized by service name and retention policies configured. The application side is a structured logger that also attaches each line to the request's trace.

Set up the app

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

~
$ encore app create payments-api --example=hello-world$ cd payments-api && mkdir -p payments

The service logs with rlog, passing fields as key-value pairs rather than formatting them into the message:

// payments/payments.go package payments import ( "context" "time" "encore.dev/beta/errs" "encore.dev/rlog" ) type ChargeParams struct { OrderID string `json:"order_id"` AmountCents int64 `json:"amount_cents"` } type ChargeResponse struct { Status string `json:"status"` } //encore:api public method=POST path=/charges func Charge(ctx context.Context, p *ChargeParams) (*ChargeResponse, error) { log := rlog.With("order_id", p.OrderID) if p.AmountCents <= 0 { log.Warn("charge rejected", "reason", "non_positive_amount", "amount_cents", p.AmountCents) return nil, &errs.Error{Code: errs.InvalidArgument, Message: "amount must be positive"} } start := time.Now() status, err := chargeProvider(ctx, p) if err != nil { log.Error("charge failed", "err", err, "duration_ms", time.Since(start).Milliseconds()) return nil, err } log.Info("charge completed", "status", status, "duration_ms", time.Since(start).Milliseconds()) return &ChargeResponse{Status: status}, nil } func chargeProvider(ctx context.Context, p *ChargeParams) (string, error) { // Call the payment provider here. return "succeeded", nil }
~/payments-api
$ encore run$ curl -X POST localhost:4000/charges -d '{"order_id":"o-1001","amount_cents":4200}'$ encore logs --env=prod

Locally, each line appears in the terminal and in the request's trace in the local development dashboard, next to the API call itself. In production, the same lines land in CloudWatch Logs in your account, where the Logs Insights queries above work on fields like order_id and duration_ms, and encore logs --env=prod streams them to your terminal. The Go logging docs cover log levels and field conventions, and the AWS infrastructure docs describe the CloudWatch setup per service.

A coding agent investigating a failing charge can run the request locally and read the trace with its log lines, or stream production logs from the CLI, without needing someone to grant it CloudWatch console access.

Best fit: services on ECS Fargate in your own AWS account that need structured, per-service CloudWatch logs with retention set, from the first deploy.

ECS CloudWatch Logs Checklist

  • Configure awslogs with a group per service and a stream prefix per container.
  • Use non-blocking mode with a max-buffer-size.
  • Give the execution role logs:CreateLogGroup if ECS creates the group.
  • Add a logs VPC endpoint or NAT for tasks in private subnets.
  • Set retention on every log group.
  • Log one JSON object per line with a request ID.
  • Use FireLens only when logs need a destination other than CloudWatch.

Frequently asked questions

How do I send ECS Fargate logs to CloudWatch?

Add a logConfiguration with logDriver awslogs to each container definition, with awslogs-group, awslogs-region and awslogs-stream-prefix options. The task execution role needs logs:CreateLogStream and logs:PutLogEvents, plus logs:CreateLogGroup if you set awslogs-create-group to true.

Why are my ECS task logs not showing up in CloudWatch?

The usual causes are a log group that does not exist without awslogs-create-group, a task execution role missing the CloudWatch Logs permissions, a task in a private subnet with no NAT gateway or CloudWatch Logs VPC endpoint, or an app that writes to a file instead of stdout and stderr.

How long does CloudWatch keep ECS logs?

By default a log group keeps events forever, and you pay for the storage. Set a retention policy on each log group, for example 30 days with aws logs put-retention-policy, and export or archive anything you need for longer.

How do I search ECS logs in CloudWatch?

Use CloudWatch Logs Insights, select the service's log group and run a query such as fields @timestamp, @message | filter level = "error" | sort @timestamp desc | limit 50. JSON log lines are parsed automatically, so their fields can be filtered and aggregated directly.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders