Nov 19, 20267 min read

The Bulkhead Pattern for Backend Services

How bulkheads stop one slow dependency from taking down a whole service: concurrency limits per dependency, separate connection pools, isolated worker queues, and a tested Go implementation.

Illustration for The Bulkhead Pattern for Backend Services

A healthy service often goes down because of a dependency: one downstream slows, and every request waiting on it piles up, holding goroutines, connections and memory until healthy endpoints start timing out too.

The bulkhead pattern caps how much of the service any one dependency can take. When the recommender hangs, product pages lose recommendations and checkout keeps working.

The bulkhead pattern at a glance

CompartmentWhat it isolatesTypical implementation
Concurrency per dependencyIn-flight calls to one downstreamSemaphore with a short wait
Connection pool per dependencySockets to one hostSeparate http.Client and Transport
Database pool per workloadConnections for reporting vs requestsSeparate pools or read replicas
Queue per job typeWorker capacity for one job typeSeparate topics and subscriptions with their own concurrency
Deployment per serviceCPU and memorySeparate processes or containers

Why does one slow dependency take down a service?

Little's law says the number of requests in flight equals arrival rate times latency. At 200 requests per second and 50 ms of latency to a recommender, about 10 calls are in flight. When that dependency slows to 5 seconds, the same traffic means 1,000 in-flight calls, each holding a goroutine, a socket and a request context. Go handles a thousand goroutines easily, but the connection pool, file descriptors, downstream rate limits and upstream timeouts do not, and requests that never touch the recommender queue behind the ones that do.

Michael Nygard named this in Release It! as one of the main stability patterns, and the Azure Architecture Center describes the same pattern for cloud services. The fix is to decide in advance how much capacity each dependency may use, then fail fast or degrade when it asks for more.

How do you build a bulkhead in Go?

A buffered channel works as a counting semaphore, so the bulkhead waits briefly for a slot, then returns ErrFull instead of queueing forever:

package bulkhead import ( "context" "errors" "time" ) // ErrFull is returned when the bulkhead has no free slot within the wait time. var ErrFull = errors.New("bulkhead full") // Bulkhead caps how many calls to one dependency can be in flight at once. type Bulkhead struct { slots chan struct{} maxWait time.Duration } func New(maxConcurrent int, maxWait time.Duration) *Bulkhead { return &Bulkhead{slots: make(chan struct{}, maxConcurrent), maxWait: maxWait} } // Do runs fn if a slot frees up within maxWait, and fails fast otherwise. func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error { timer := time.NewTimer(b.maxWait) defer timer.Stop() select { case b.slots <- struct{}{}: case <-timer.C: return ErrFull case <-ctx.Done(): return ctx.Err() } defer func() { <-b.slots }() return fn(ctx) }

The golang.org/x/sync/semaphore package offers the same with weighted slots, if some calls cost more than others. The handler wraps the non-critical call and degrades when the bulkhead is full or the call fails. Each dependency also gets its own http.Client, since a shared Transport shares its connection pool:

// Each dependency gets its own HTTP client and connection cap, so a slow // recommender cannot hold connections the payment client needs. var ( recsClient = &http.Client{ Timeout: 800 * time.Millisecond, Transport: &http.Transport{MaxConnsPerHost: 20}, } recsBulkhead = New(20, 50*time.Millisecond) ) func ProductPage(w http.ResponseWriter, r *http.Request) { p := Product{ID: r.PathValue("id")} err := recsBulkhead.Do(r.Context(), func(ctx context.Context) error { var err error p.Recommendations, err = fetchRecs(ctx, p.ID) return err }) if err != nil { p.Recommendations = []string{} // degrade: the page renders without them } json.NewEncoder(w).Encode(p) }

A test makes the behavior concrete with ten concurrent calls to a recommender that hangs for 200 ms against a bulkhead of two: two run, eight are rejected within 10 ms, and a call through the separate payments bulkhead completes immediately:

func TestSlowDependencyDoesNotStarveOthers(t *testing.T) { recs := New(2, 10*time.Millisecond) // slow, non-critical dependency payments := New(2, 10*time.Millisecond) // critical dependency var full atomic.Int32 var wg sync.WaitGroup for i := 0; i < 10; i++ { // 10 concurrent calls to a recommender that hangs wg.Add(1) go func() { defer wg.Done() err := recs.Do(context.Background(), func(ctx context.Context) error { time.Sleep(200 * time.Millisecond) return nil }) if errors.Is(err, ErrFull) { full.Add(1) } }() } time.Sleep(20 * time.Millisecond) start := time.Now() err := payments.Do(context.Background(), func(ctx context.Context) error { return nil }) if err != nil || time.Since(start) > 5*time.Millisecond { t.Fatalf("payments blocked: %v after %v", err, time.Since(start)) } wg.Wait() if full.Load() != 8 { t.Fatalf("expected 8 rejected recommender calls, got %d", full.Load()) } }

Where else do bulkheads belong?

Database pools. A reporting query that holds connections for 30 seconds can drain a pool shared with request handlers. Give reports their own small pool or send them to a read replica.

Worker queues. If thumbnails and receipts share one queue and one worker pool, an upload burst delays every receipt. Separate queues with separate concurrency limits keep each job type in its own lane.

Incoming traffic. A per-endpoint or per-tenant concurrency limit keeps one noisy caller from using all the capacity, which is closely related to load shedding, where a server rejects work early rather than accepting more than it can finish.

Libraries. JVM teams get the same compartments from Resilience4j's bulkhead module, with semaphore and thread-pool variants.

How do you size a bulkhead?

Use Little's law again: normal requests per second times p99 latency in seconds gives the usual in-flight count, and the limit sits at two to three times that. Keep the wait short, tens of milliseconds, because a long wait turns the bulkhead back into a queue. Export rejections and in-flight counts as metrics, alert on sustained rejections, and pair the bulkhead with a timeout on every call, since a slot held by a call that never returns is a slot lost.

Bulkheads for async work with Encore

Encore

Encore is a backend framework with built-in infrastructure, and for background work the compartments are part of the declarations. Each Pub/Sub subscription has its own queue and its own concurrency limit, so a burst on one topic cannot use the workers of another.

Application code passes through Encore and infrastructure guardrails to resources provisioned in your own AWS or GCP account.
Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

~
$ encore app create shop --example=hello-world$ cd shop && mkdir media
// media/media.go package media import ( "context" "time" "encore.dev/pubsub" ) type UploadEvent struct { ImageKey string `json:"image_key"` } type OrderEvent struct { OrderID string `json:"order_id"` } var ImageUploaded = pubsub.NewTopic[*UploadEvent]("image-uploaded", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var OrderPlaced = pubsub.NewTopic[*OrderEvent]("order-placed", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) // CPU-heavy work gets a small compartment: at most 4 thumbnails at once // per instance, however many uploads arrive. var _ = pubsub.NewSubscription(ImageUploaded, "make-thumbnails", pubsub.SubscriptionConfig[*UploadEvent]{ Handler: makeThumbnail, MaxConcurrency: 4, AckDeadline: 2 * time.Minute, }, ) // Receipts have their own compartment, so a thumbnail backlog never // delays them. var _ = pubsub.NewSubscription(OrderPlaced, "send-receipt", pubsub.SubscriptionConfig[*OrderEvent]{ Handler: sendReceipt, MaxConcurrency: 50, RetryPolicy: &pubsub.RetryPolicy{MinBackoff: time.Second, MaxRetries: 10}, }, ) func makeThumbnail(ctx context.Context, e *UploadEvent) error { // Download the image, resize it and upload the thumbnail. return nil } func sendReceipt(ctx context.Context, e *OrderEvent) error { // Call the email provider. return nil }

MaxConcurrency applies per instance of the service. On GCP with Cloud Run, at-least-once subscriptions are configured as push subscriptions with adaptive concurrency, where the limit does not apply. Retries back off per subscription under its own retry policy, so a failing thumbnail job retries without touching the receipt queue. On the synchronous side, each service declares its own database, so a slow report in one service does not drain the connection pool of another.

encore run runs both subscriptions locally, and the local trace for each message shows how long it waited and ran. An agent adding a third job type, such as video transcoding, declares a new topic and subscription with its own limit, so the compartment comes with the feature instead of as a later tuning task.

Best fit: Go and TypeScript teams that want each background workload isolated by default, with queues and concurrency limits declared next to the handlers.

Frequently asked questions

What is the bulkhead pattern?

The bulkhead pattern splits a service's resources into isolated compartments, one per dependency or workload, so exhausting one compartment does not affect the others. The name comes from ship hulls, where watertight bulkheads keep a breach in one section from flooding the whole ship.

What is the difference between a bulkhead and a circuit breaker?

A bulkhead limits how much of your capacity one dependency can consume at a time, while a circuit breaker stops calling a dependency after it has failed repeatedly. They complement each other: the bulkhead protects you while a dependency is slow, and the breaker stops the calls once it is clearly down.

How do I implement a bulkhead in Go?

Use a buffered channel or golang.org/x/sync/semaphore as a counting semaphore per dependency, acquire a slot with a short wait before each call, and return an error immediately when no slot frees up. Pair it with a separate http.Client and connection limit per dependency, and a fallback for non-critical calls.

How big should a bulkhead be?

Start from the dependency's normal concurrency, which is roughly requests per second times p99 latency in seconds, and add headroom. Too small rejects healthy traffic, too large lets a slow dependency hold most of your goroutines and connections again. Track rejections as a metric and adjust.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders