The AWS Rust SDK will let a single SQS receive hang forever, and it does so by default. If you long-poll a queue in a loop, which is the normal way to consume SQS, a connection that dies quietly leaves one receive waiting for a response that never comes, and nothing in the default configuration ends the wait. The consumer keeps running so nothing restarts it, and the only sign of trouble is the queue backing up behind a worker that still looks healthy. AWS knows about this and has decided the default stays. We hit it in Encore's Pub/Sub integration, and the fix came down to one timeout, placed carefully.

How the subscriber reads the queue

On Encore, Pub/Sub is a declared primitive that you write as a topic and a subscription in your code, and the runtime provisions the queue and runs the consumer for you, so on AWS the consumer pulling messages is Encore's rather than something each service writes. That consumer reads work by long-polling SQS, calling ReceiveMessage in a loop where each call waits up to twenty seconds for a message so an empty queue does not spin:

// runtimes/core/src/pubsub/sqs_sns/sub.rs
let result = client
    .receive_message()
    .queue_url(queue_url)
    .visibility_timeout(ack_deadline.as_secs() as i32)
    .wait_time_seconds(LONG_POLL_WAIT_SECS as i32) // 20s long poll
    .max_number_of_messages(max_items as i32)
    .send()
    .await;

The client making that call was built with the SDK's default configuration, which set a region and nothing else:

// before: region only, no timeouts on the request itself
aws_config::defaults(BehaviorVersion::latest())
    .region(provider)
    .load()
    .await

Those defaults bound how long it takes to open a connection, but nothing bounds a request once the connection is open, so a single ReceiveMessage can wait for a response for as long as the socket stays up.

Why the whole subscription stalls

When a load balancer or a NAT gateway quietly drops the idle TCP connection that the client still treats as open, the next send() goes out on a socket that will never answer, and with no timeout on the request the future for that call never resolves.

The fetch loop is sequential and waits for each receive to finish before starting the next, so one stuck call parks the whole loop and the subscription stops fetching at all. The retry and backoff that would normally cover a failed receive never run, because they only trigger when a receive comes back with an error, and this call never comes back with anything. From the outside the process looks healthy, so nothing restarts it and a health check stays green while the queue backs up behind it, its oldest message getting steadily older until someone notices.

The timeout that fixes it

The fix (#2469) was to give the client a per-attempt timeout:

// runtimes/core/src/pubsub/sqs_sns/mod.rs
// Must exceed the 20s long-poll wait, so an empty poll isn't cut off. Bounds a
// stalled request so it fails and the SDK retries on a fresh connection instead
// of hanging forever on a silently-dropped one.
const OPERATION_ATTEMPT_TIMEOUT: Duration =
    Duration::from_secs(LONG_POLL_WAIT_SECS + 10);

let timeout_config = TimeoutConfig::builder()
    .operation_attempt_timeout(OPERATION_ATTEMPT_TIMEOUT)
    .build();

An operation_attempt_timeout bounds each individual HTTP attempt, so a request that stalls is abandoned after thirty seconds instead of waiting forever. An abandoned attempt counts as a retryable failure, so the SDK retries the operation and the fetch loop keeps moving instead of parking on a dead socket, with nothing downstream having to notice.

Why it has to be a per-attempt timeout

The SDK also offers an operation timeout, but that one covers a whole operation including its retries, so to leave room for a twenty-second long poll and a retry or two it would have to be generous, which means you still wait a long time before it fires, and when it does it gives up on the operation rather than recovering it. The per-attempt timeout bounds each try instead, so a single stall costs one attempt and the operation retries rather than giving up.

The timeout also has to clear the long poll, since the receive call waits up to twenty seconds for a message and anything shorter would cut off ordinary empty polls as though they had stalled, which is why the value is the long-poll wait plus ten seconds of headroom and why the two constants sit next to each other in the code.

The same behavior was reported against the AWS Rust SDK by a Vector user, whose S3 source polls SQS in the same kind of loop and saw the occasional receive_message hang forever. The recommendation on that issue was to set an operation_attempt_timeout a little above the long-poll wait, alongside an admission that the SDK ought to ship better default timeouts and a decision not to change them for now. The default that causes the hang is known and not changing, so any service long-polling SQS with this SDK is one dropped connection away from the same failure until it sets the timeout itself.

Any long-polling client

None of this is specific to SQS, since any client that holds a connection open with no limit on a single attempt can hang indefinitely the moment that connection fails without warning, and any worker that handles one item at a time turns that single hang into a full stop. A per-attempt timeout, set above the longest wait you expect, lets the system clear itself instead of waiting on a restart.

The fix is in the shared runtime, so every Encore app that consumes a queue got it by upgrading without anyone having to know the AWS default existed, while a consumer you wrote yourself needs that timeout set by hand.

The whole change is seventeen lines, including a comment explaining why the timeout has to be there.