Ivan CernjaService-to-Service Communication Patterns
Synchronous calls versus events, the tradeoffs of each, and the reliability work that makes either one hold up
A checkout request lands on an orders service. It calls the payments service to charge the card, waits for that to return, then needs three more things to happen: the warehouse gets notified to ship, the customer gets a confirmation email, and the analytics pipeline records the sale. If the orders service calls all five services in a row and waits for each, the customer stares at a spinner while the email sends, and a slow analytics database can fail an order that already took the money. If it publishes an event instead and lets those services react on their own, the checkout returns the moment payment clears, and the email and analytics catch up a moment later.
The decision running through that example is whether the caller needs an answer before it can continue. Some interactions do, so they call and wait, and others are side effects that can happen after the response goes out, so they fire an event and move on. Most distributed-system pain traces back to picking the wrong one, either a chain of synchronous calls that fail together, or an event flow used for something the caller needed the result of.
Two ways the same checkout can wire its downstream work: a synchronous chain where the orders service calls payment, shipping, email, and analytics in sequence and waits for each, versus an event-driven flow where it calls payment inline and then publishes an order-placed event that shipping, email, and analytics consume independently.
The two families
Everything services do to talk to each other reduces to two families, and the difference between them is whether the caller waits.
Synchronous means the caller sends a request and blocks until it gets a response. REST over HTTP is the common form, readable and easy to debug with curl, and it fits anything a browser or third party will also call. gRPC is the other common form, using binary Protocol Buffers over HTTP/2 with generated typed clients, which cuts latency and serialization work on high-volume internal traffic and supports streaming in both directions. The tradeoff for the immediate answer is coupling in time: the callee has to be up and reachable right now, and its latency is added to yours.
Asynchronous means the sender hands a message off and returns without waiting for it to be processed. A message queue delivers each message to one consumer, which is how you distribute work like resizing an image or generating a report across a pool of workers. Pub/Sub delivers each published message to every subscriber, which is how one event, an order being placed, fans out to shipping, email, and analytics without the publisher knowing they exist. The sender and receiver are decoupled in time, so a spike gets buffered and a briefly-down consumer catches up when it recovers. The cost is that there is no return value, and a single logical operation now spans a producer, a broker, and one or more consumers, which is harder to trace and reason about. The queue-versus-Pub/Sub choice within this family has its own tradeoffs, covered in Message Queues vs Pub/Sub.
Which pattern for which job
The deciding question is whether the caller needs the result to continue.
| Pattern | Caller waits? | Use it for | Watch out for |
|---|---|---|---|
| REST | Yes | External APIs, browser-facing calls, reads across services | Latency adds up in call chains |
| gRPC | Yes | High-volume internal calls, streaming, typed contracts | Not browser-native; extra tooling |
| Message queue | No | Work distributed across workers, deferred processing | Ordering and duplicate delivery |
| Pub/Sub | No | One event, many reactions; decoupling side effects | No back-channel for results |
A read that a page render depends on, fetching a user profile before showing a dashboard, is synchronous by nature, and forcing it through a queue only adds latency and complexity. A side effect that the caller does not need to see finish, updating a search index or sending a receipt, is a good fit for an event, and doing it synchronously couples the caller's success to a service it should not depend on.
The failure mode to watch for is a long synchronous chain: service A calls B, which calls C, which calls D, all blocking. The end-to-end latency is the sum of every hop, and if any one is down or slow, the whole chain fails or hangs. When you find one, look for hops that could be events instead, and for reads that could be cached or fetched in parallel rather than in sequence.
The reliability work either way
Whichever pattern you pick, the network will drop, delay, and duplicate messages, and a few concerns show up in every distributed system regardless of transport.
- Timeouts. Every synchronous call needs a deadline. Without one, a stuck downstream call holds the caller's thread or connection open, and under load those pile up until the caller runs out and falls over too. Set timeouts shorter as you go down a call chain so an inner call cannot outlast the outer request that is waiting on it.
- Retries. A failed call or an unacknowledged message often just needs to be tried again, but naive retries make an overloaded service worse. Use exponential backoff with jitter so a thousand clients do not retry in lockstep, and cap the attempts so a permanently broken dependency does not get hammered forever.
- Idempotency. Retries and at-least-once delivery mean a receiver will sometimes get the same request twice. Any operation with a side effect, charging a card, creating an order, needs to be safe to run more than once. The usual mechanism is an idempotency key that the caller sends and the receiver records, so a repeat with the same key returns the stored result instead of doing the work again.
- Backpressure. A fast producer will outrun a slow consumer, and the system needs a way to signal it. In synchronous chains that signal is rejected requests and circuit breakers that stop calling a failing dependency for a while. In asynchronous systems the queue is the buffer, holding the spike while the consumer drains it, though the queue depth then needs watching so it does not grow without bound.
Async buys resilience through the buffer, but it moves the hard part rather than removing it: you now have delivery guarantees to reason about, consumers that must be idempotent because delivery is at-least-once, and an operation whose success is spread across services you have to stitch back together in tracing.
Where Encore fits
The two families map onto two things Encore gives you directly, so the choice stays about the interaction rather than about wiring.
For synchronous calls between your own services, you import the callee and call it as a function. There is no HTTP client to configure, no service URL to thread through config, and no hand-written serialization, because the call is type-checked against the callee's signature at compile time. Rename or change a field on the callee and the caller stops compiling, which catches the class of break that HTTP boundaries let through until runtime.
// orders/orders.ts
import { api } from "encore.dev/api";
import { Topic } from "encore.dev/pubsub";
import { payments } from "~encore/clients";
// A typed Pub/Sub topic. Subscribers in other services react to this.
export const orderPlaced = new Topic<OrderPlaced>("order-placed", {
deliveryGuarantee: "at-least-once",
});
export const create = api(
{ method: "POST", path: "/orders", expose: true },
async (p: CreateParams): Promise<Order> => {
// Synchronous, because we need the charge to succeed before confirming.
const charge = await payments.charge({ amount: p.amount, card: p.card });
// Asynchronous: shipping, email, and analytics subscribe and react.
const order = await placeOrder(p, charge.id);
await orderPlaced.publish({ orderId: order.id });
return order;
},
);
For asynchronous work you declare a Pub/Sub topic as a typed object with a delivery guarantee, and subscribers in other services react to it. The at-least-once guarantee is stated in the declaration, which is a reminder that your subscribers need to be idempotent. Because both the calls and the topics are declared in application code, the distributed tracing that stitches a synchronous chain and its downstream events back into one timeline is wired up without extra work, which is the part of async debugging that usually hurts. Encore provisions the queues and topics into your own AWS or GCP account, using SNS and SQS or GCP Pub/Sub, so the delivery semantics are the cloud's, not a layer on top.
Related reading
Frequently asked questions
What is service-to-service communication?
Service-to-service communication is how one backend service calls or notifies another to get work done. It falls into two families. Synchronous calls over REST or gRPC have the caller send a request and wait for a response, while asynchronous messaging over events or a queue has the sender publish a message and move on while the receiver processes it later. Most real systems use both, picking per interaction based on whether the caller needs an answer right away.
What is the difference between synchronous and asynchronous communication in microservices?
Synchronous communication blocks the caller until the callee responds, so a request to fetch a user's profile returns the data inline. Asynchronous communication hands a message to a queue or event stream and returns immediately, so the receiver processes it on its own schedule. Synchronous is simpler to reason about and gives an immediate answer. Asynchronous decouples the two services in time, absorbs load spikes, and survives a receiver being briefly down, at the cost of harder end-to-end tracing.
When should you use REST versus gRPC between services?
Use REST when the endpoint is also consumed by browsers or third parties, when human readability matters, or when the ecosystem around your language favors it. Use gRPC for high-volume internal traffic where its binary encoding, streaming, and generated typed clients cut latency and remove hand-written serialization. Many systems expose REST at the edge for external clients and use gRPC or typed calls between internal services where both ends are yours.
What is idempotency and why does it matter for service calls?
An idempotent operation produces the same result whether it runs once or many times. It matters because retries and at-least-once message delivery mean a service will sometimes receive the same request twice. If charging a card is idempotent, a duplicate delivery does not double-charge the customer. The common approach is an idempotency key the caller sends and the receiver records, so a repeat with the same key returns the first result instead of running the work again.
What is backpressure in a distributed system?
Backpressure is how a system signals that it cannot keep up with incoming work so upstream senders slow down. In synchronous chains it shows up as timeouts, rejected requests, or circuit breakers tripping. In asynchronous systems the queue itself is the buffer, absorbing a spike while the consumer drains it at its own rate. Without backpressure a fast producer overwhelms a slow consumer, and the failure cascades back through the chain of callers.
Should microservices communicate directly or through a message broker?
Direct synchronous calls fit interactions where the caller needs an answer to continue, like reading data before rendering a page. A message broker fits work that can happen after the response is sent, like sending a confirmation email or updating a search index, and it keeps the two services from failing together. Systems usually mix the two, and frameworks like Encore let you make a direct call a type-checked function and publish an event through a typed Pub/Sub topic in the same codebase.