Nov 17, 20264 min read

Amazon EventBridge Pipes: Connect Sources to Targets Without a Lambda

Build a point-to-point pipe from SQS, Kinesis, DynamoDB Streams or Kafka to a target, with optional filtering and enrichment, and when an Encore subscription is a better fit.

Illustration for Amazon EventBridge Pipes: Connect Sources to Targets Without a Lambda

A common AWS glue job is "read this queue or stream and put the payload on that other service," and the default way to do it used to be a small Lambda that polls and forwards.

EventBridge Pipes is the managed version of that glue: one source, an optional filter, an optional enrichment, and one target, with order preserved when the source itself is ordered. You are charged for events that match the filter, not for every record the source emits.

This guide walks through creating a pipe from SQS to a target, then shows the Encore path for events that start in your application code, where a topic and subscription replace the pipe.

EventBridge Pipes at a glance

PieceWhat it doesNotes
SourceSQS, Kinesis, DynamoDB Streams, Amazon MQ, MSK, Apache KafkaThe source cannot be changed after the pipe is created
FilterEvent pattern on the payloadOnly matching events are billed and forwarded
EnrichmentLambda, Express Step Functions, API Gateway, API destinationInvoked synchronously, 6 MB response limit
TargetOne destination per pipeSeveral pipes can share a target; use an event bus to fan out
OrderingPreserved when the source is orderedKept through filter and enrichment, including batches

Create a pipe from SQS

Pick the queue, add a filter if you only want some messages, optionally attach an enrichment, and choose a target, as described in Creating a pipe. Before running the command, create or verify the order-paid-pipe role has a trust policy allowing pipes.amazonaws.com to assume it, and least-privilege permissions for sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the queue plus states:StartExecution on the state machine (and any required KMS permissions for encrypted resources). EventBridge validates the source permissions when the pipe is created. The CLI command below assumes this role already exists. Enrichments and targets can be changed later; the source cannot.

A typical SQS-to-Step-Functions pipe looks like this in the AWS CLI:

~
$ aws pipes create-pipe --name order-paid-to-sfn \ --source arn:aws:sqs:eu-west-1:123456789012:order-paid \ --target arn:aws:states:eu-west-1:123456789012:stateMachine:OrderPaid \ --role-arn arn:aws:iam::123456789012:role/order-paid-pipe \ --source-parameters '{"SqsQueueParameters":{"BatchSize":1}}'

Filters use an event pattern: for an SQS message whose body is JSON, match its fields under body, for example { "body": [{ "orderStatus": ["PAID"] }] }, as shown in the pipes filtering documentation. Use data only after an enrichment explicitly produces that field. Enrichment can also drop an event by returning {}, [] or an empty string; return [{}] if the target should be invoked with an empty payload.

Batch size is validated against the target: if the target does not support batches, the pipe cannot batch, and API Gateway or API destination enrichments only support a batch size of 1.

Best fit: a single queue or stream that should feed a single AWS target, with a filter and maybe a small enrichment, and no desire to run a polling function.

When a Lambda is still the right glue

Write a Lambda when the mapping needs branches, retries against your own database, or a side effect that a pipe cannot express. Pipes do not replace a workflow engine: long-running, branching processes still belong in Step Functions or in application code. They also do not replace a topic with several independent consumers; use SNS or an event bus when one event must reach many targets.

How do you move application events with Encore?

Encore

Encore provisions topics from code: when the event is produced by your API, declare a topic and a subscription in Go. On AWS, a deploy provisions SNS and SQS in your account; on GCP that is Pub/Sub. Encore does not create EventBridge Pipes. Encore provisions an IAM role for the service, or a service account on GCP.

A coding agent can add the topic, the publish call and the subscriber in the same change, then use encore run to publish a message and watch the subscriber run on the laptop.

var Paid = pubsub.NewTopic[*OrderPaid]("order-paid", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var _ = pubsub.NewSubscription(Paid, "start-fulfillment", pubsub.SubscriptionConfig[*OrderPaid]{ Handler: fulfillOrder, RetryPolicy: &pubsub.RetryPolicy{MaxRetries: 8}, }, )

If the messages already live in an SQS queue or a Kinesis stream that Encore does not own, keep a pipe (or a consumer you write) in front of that source, and publish into an Encore topic only once the payload has entered your application.

Best fit: events that start in your API and should be processed by code in the same repository, with local delivery under encore run.

How to choose

Pick EventBridge Pipes when an existing SQS queue or stream should feed one AWS target without a polling Lambda. Pick a Lambda when the transformation needs application logic. Pick Encore when your service produces the event and you want the subscriber declared next to the publisher in Go.

Frequently asked questions

What is Amazon EventBridge Pipes?

EventBridge Pipes is a point-to-point integration that reads from a single source, optionally filters and enriches the event, and sends it to a single target. Supported sources include SQS, Kinesis, DynamoDB Streams, Amazon MQ, MSK and self-managed Kafka. You pay for events that match the filter.

Do I need a Lambda to move SQS messages to another AWS service?

Not always. A pipe can read an SQS queue and send matching messages to a target such as Step Functions, Kinesis, a Lambda function or an API destination. Use a Lambda anyway when the transformation needs custom code that a pipe filter and enrichment cannot express.

Does Encore provision EventBridge Pipes?

No. Encore provisions SNS and SQS or GCP Pub/Sub for topics and subscriptions declared in your code, and it does not create pipes. When the event starts in your application, publish to an Encore topic and handle it in a subscription. When the event already lives in SQS or a stream you do not own, a pipe can move it without a polling function.

How does enrichment in a pipe work?

The pipe can call a Lambda function, an Express Step Functions workflow, API Gateway or an API destination synchronously, then pass the response to the target. Enrichment responses are limited to 6 MB. Returning an empty payload such as {}, [] or an empty string skips the target.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders