How bulkheads stop one slow dependency from taking down a whole service: concurrency limits per dependency, separate connection pools, isolated worker queues, and a tested Go implementation.

A healthy service often goes down because of a dependency: one downstream slows, and every request waiting on it piles up, holding goroutines, connections and memory until healthy endpoints start timing out too.
The bulkhead pattern caps how much of the service any one dependency can take. When the recommender hangs, product pages lose recommendations and checkout keeps working.
Little's law says the number of requests in flight equals arrival rate times latency. At 200 requests per second and 50 ms of latency to a recommender, about 10 calls are in flight. When that dependency slows to 5 seconds, the same traffic means 1,000 in-flight calls, each holding a goroutine, a socket and a request context. Go handles a thousand goroutines easily, but the connection pool, file descriptors, downstream rate limits and upstream timeouts do not, and requests that never touch the recommender queue behind the ones that do.
Michael Nygard named this in Release It! as one of the main stability patterns, and the Azure Architecture Center describes the same pattern for cloud services. The fix is to decide in advance how much capacity each dependency may use, then fail fast or degrade when it asks for more.
A buffered channel works as a counting semaphore, so the bulkhead waits briefly for a slot, then returns ErrFull instead of queueing forever:
package bulkhead
import (
"context"
"errors"
"time"
)
// ErrFull is returned when the bulkhead has no free slot within the wait time.
var ErrFull = errors.New("bulkhead full")
// Bulkhead caps how many calls to one dependency can be in flight at once.
type Bulkhead struct {
slots chan struct{}
maxWait time.Duration
}
func New(maxConcurrent int, maxWait time.Duration) *Bulkhead {
return &Bulkhead{slots: make(chan struct{}, maxConcurrent), maxWait: maxWait}
}
// Do runs fn if a slot frees up within maxWait, and fails fast otherwise.
func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error {
timer := time.NewTimer(b.maxWait)
defer timer.Stop()
select {
case b.slots <- struct{}{}:
case <-timer.C:
return ErrFull
case <-ctx.Done():
return ctx.Err()
}
defer func() { <-b.slots }()
return fn(ctx)
}
The golang.org/x/sync/semaphore package offers the same with weighted slots, if some calls cost more than others. The handler wraps the non-critical call and degrades when the bulkhead is full or the call fails. Each dependency also gets its own http.Client, since a shared Transport shares its connection pool:
// Each dependency gets its own HTTP client and connection cap, so a slow
// recommender cannot hold connections the payment client needs.
var (
recsClient = &http.Client{
Timeout: 800 * time.Millisecond,
Transport: &http.Transport{MaxConnsPerHost: 20},
}
recsBulkhead = New(20, 50*time.Millisecond)
)
func ProductPage(w http.ResponseWriter, r *http.Request) {
p := Product{ID: r.PathValue("id")}
err := recsBulkhead.Do(r.Context(), func(ctx context.Context) error {
var err error
p.Recommendations, err = fetchRecs(ctx, p.ID)
return err
})
if err != nil {
p.Recommendations = []string{} // degrade: the page renders without them
}
json.NewEncoder(w).Encode(p)
}
A test makes the behavior concrete with ten concurrent calls to a recommender that hangs for 200 ms against a bulkhead of two: two run, eight are rejected within 10 ms, and a call through the separate payments bulkhead completes immediately:
func TestSlowDependencyDoesNotStarveOthers(t *testing.T) {
recs := New(2, 10*time.Millisecond) // slow, non-critical dependency
payments := New(2, 10*time.Millisecond) // critical dependency
var full atomic.Int32
var wg sync.WaitGroup
for i := 0; i < 10; i++ { // 10 concurrent calls to a recommender that hangs
wg.Add(1)
go func() {
defer wg.Done()
err := recs.Do(context.Background(), func(ctx context.Context) error {
time.Sleep(200 * time.Millisecond)
return nil
})
if errors.Is(err, ErrFull) {
full.Add(1)
}
}()
}
time.Sleep(20 * time.Millisecond)
start := time.Now()
err := payments.Do(context.Background(), func(ctx context.Context) error { return nil })
if err != nil || time.Since(start) > 5*time.Millisecond {
t.Fatalf("payments blocked: %v after %v", err, time.Since(start))
}
wg.Wait()
if full.Load() != 8 {
t.Fatalf("expected 8 rejected recommender calls, got %d", full.Load())
}
}
Database pools. A reporting query that holds connections for 30 seconds can drain a pool shared with request handlers. Give reports their own small pool or send them to a read replica.
Worker queues. If thumbnails and receipts share one queue and one worker pool, an upload burst delays every receipt. Separate queues with separate concurrency limits keep each job type in its own lane.
Incoming traffic. A per-endpoint or per-tenant concurrency limit keeps one noisy caller from using all the capacity, which is closely related to load shedding, where a server rejects work early rather than accepting more than it can finish.
Libraries. JVM teams get the same compartments from Resilience4j's bulkhead module, with semaphore and thread-pool variants.
Use Little's law again: normal requests per second times p99 latency in seconds gives the usual in-flight count, and the limit sits at two to three times that. Keep the wait short, tens of milliseconds, because a long wait turns the bulkhead back into a queue. Export rejections and in-flight counts as metrics, alert on sustained rejections, and pair the bulkhead with a timeout on every call, since a slot held by a call that never returns is a slot lost.
Encore is a backend framework with built-in infrastructure, and for background work the compartments are part of the declarations. Each Pub/Sub subscription has its own queue and its own concurrency limit, so a burst on one topic cannot use the workers of another.

Run in your terminal to get started locally.
// media/media.go
package media
import (
"context"
"time"
"encore.dev/pubsub"
)
type UploadEvent struct {
ImageKey string `json:"image_key"`
}
type OrderEvent struct {
OrderID string `json:"order_id"`
}
var ImageUploaded = pubsub.NewTopic[*UploadEvent]("image-uploaded", pubsub.TopicConfig{
DeliveryGuarantee: pubsub.AtLeastOnce,
})
var OrderPlaced = pubsub.NewTopic[*OrderEvent]("order-placed", pubsub.TopicConfig{
DeliveryGuarantee: pubsub.AtLeastOnce,
})
// CPU-heavy work gets a small compartment: at most 4 thumbnails at once
// per instance, however many uploads arrive.
var _ = pubsub.NewSubscription(ImageUploaded, "make-thumbnails",
pubsub.SubscriptionConfig[*UploadEvent]{
Handler: makeThumbnail,
MaxConcurrency: 4,
AckDeadline: 2 * time.Minute,
},
)
// Receipts have their own compartment, so a thumbnail backlog never
// delays them.
var _ = pubsub.NewSubscription(OrderPlaced, "send-receipt",
pubsub.SubscriptionConfig[*OrderEvent]{
Handler: sendReceipt,
MaxConcurrency: 50,
RetryPolicy: &pubsub.RetryPolicy{MinBackoff: time.Second, MaxRetries: 10},
},
)
func makeThumbnail(ctx context.Context, e *UploadEvent) error {
// Download the image, resize it and upload the thumbnail.
return nil
}
func sendReceipt(ctx context.Context, e *OrderEvent) error {
// Call the email provider.
return nil
}
MaxConcurrency applies per instance of the service. On GCP with Cloud Run, at-least-once subscriptions are configured as push subscriptions with adaptive concurrency, where the limit does not apply. Retries back off per subscription under its own retry policy, so a failing thumbnail job retries without touching the receipt queue. On the synchronous side, each service declares its own database, so a slow report in one service does not drain the connection pool of another.
encore run runs both subscriptions locally, and the local trace for each message shows how long it waited and ran. An agent adding a third job type, such as video transcoding, declares a new topic and subscription with its own limit, so the compartment comes with the feature instead of as a later tuning task.
Best fit: Go and TypeScript teams that want each background workload isolated by default, with queues and concurrency limits declared next to the handlers.
The bulkhead pattern splits a service's resources into isolated compartments, one per dependency or workload, so exhausting one compartment does not affect the others. The name comes from ship hulls, where watertight bulkheads keep a breach in one section from flooding the whole ship.
A bulkhead limits how much of your capacity one dependency can consume at a time, while a circuit breaker stops calling a dependency after it has failed repeatedly. They complement each other: the bulkhead protects you while a dependency is slow, and the breaker stops the calls once it is clearly down.
Use a buffered channel or golang.org/x/sync/semaphore as a counting semaphore per dependency, acquire a slot with a short wait before each call, and return an error immediately when no slot frees up. Pair it with a separate http.Client and connection limit per dependency, and a fallback for non-critical calls.
Start from the dependency's normal concurrency, which is roughly requests per second times p99 latency in seconds, and add headroom. Too small rejects healthy traffic, too large lets a slow dependency hold most of your goroutines and connections again. Track rejections as a metric and adjust.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.