Background Jobs in Go: Queues, Workers and Scheduled Tasks

Background jobs in Go start out as a goroutine launched from an HTTP handler, and that holds until the first deploy restarts the container halfway through sending an email or building an invoice. Moving the work onto a queue fixes that, and adds what a goroutine never had: retries with backoff, a dead-letter queue for jobs that keep failing, a limit on how many run at once, and scheduled jobs that fire once per cluster instead of once per replica.

This guide covers the options in the order a Go backend usually meets them, with a working example of a report job that is enqueued from an API, retried on failure and scheduled nightly.

Background job options in Go at a glance

How do you run background jobs in Go with Encore?

Encore combines an open source infrastructure SDK with a platform that provisions and deploys into your own AWS or GCP account. In Go, a background job is a Pub/Sub topic for the queue, a subscription for the worker and a cron job for anything on a schedule, all declared as package variables in the service that owns the work.

Set up the app

The service below enqueues a report job from an API, builds the report in a worker that retries with backoff and dead-letters after five attempts, writes the result to an object storage bucket, and schedules the nightly fan-out:

// reports/reports.go
package reports

import (
	"context"
	"fmt"
	"strings"
	"time"

	"encore.dev/cron"
	"encore.dev/pubsub"
	"encore.dev/rlog"
	"encore.dev/storage/objects"
)

// ReportRequested is the job payload. Keep it small: IDs, not whole records.
type ReportRequested struct {
	AccountID string `json:"account_id"`
	Day       string `json:"day"` // YYYY-MM-DD
}

// The queue: SNS and SQS on AWS, Pub/Sub on GCP, in-process with `encore run`.
var ReportJobs = pubsub.NewTopic[*ReportRequested]("report-jobs", pubsub.TopicConfig{
	DeliveryGuarantee: pubsub.AtLeastOnce,
})

// Finished reports: S3 on AWS, Cloud Storage on GCP, a local bucket with `encore run`.
var Reports = objects.NewBucket("reports", objects.BucketConfig{})

// The worker. Failed jobs are retried with backoff and moved to a
// dead-letter queue after 5 attempts.
var _ = pubsub.NewSubscription(ReportJobs, "build-report",
	pubsub.SubscriptionConfig[*ReportRequested]{
		Handler:        buildReport,
		MaxConcurrency: 10,
		AckDeadline:    2 * time.Minute,
		RetryPolicy: &pubsub.RetryPolicy{
			MinBackoff: 5 * time.Second,
			MaxBackoff: 5 * time.Minute,
			MaxRetries: 5,
		},
	},
)

func buildReport(ctx context.Context, job *ReportRequested) error {
	// The object key is derived from the job, so a redelivered message
	// overwrites the same file instead of creating a duplicate.
	key := fmt.Sprintf("%s/%s.csv", job.AccountID, job.Day)
	csv := "account_id,day,generated_at\n" +
		strings.Join([]string{job.AccountID, job.Day, time.Now().UTC().Format(time.RFC3339)}, ",") + "\n"

	w := Reports.Upload(ctx, key)
	if _, err := w.Write([]byte(csv)); err != nil {
		w.Abort(err)
		return err // returning an error triggers a retry
	}
	if err := w.Close(); err != nil {
		return err
	}
	rlog.Info("report built", "key", key)
	return nil
}

type EnqueueParams struct {
	AccountID string `json:"account_id"`
	Day       string `json:"day"`
}

type EnqueueResponse struct {
	MessageID string `json:"message_id"`
}

// Enqueue puts a report job on the queue and returns immediately.
//
//encore:api public method=POST path=/reports
func Enqueue(ctx context.Context, p *EnqueueParams) (*EnqueueResponse, error) {
	id, err := ReportJobs.Publish(ctx, &ReportRequested{AccountID: p.AccountID, Day: p.Day})
	if err != nil {
		return nil, err
	}
	return &EnqueueResponse{MessageID: id}, nil
}

type ReportStatus struct {
	Ready bool  `json:"ready"`
	Size  int64 `json:"size"`
}

//encore:api public method=GET path=/reports/:accountID/:day
func Status(ctx context.Context, accountID, day string) (*ReportStatus, error) {
	attrs, err := Reports.Attrs(ctx, fmt.Sprintf("%s/%s.csv", accountID, day))
	if err != nil {
		return &ReportStatus{Ready: false}, nil
	}
	return &ReportStatus{Ready: true, Size: attrs.Size}, nil
}

// Nightly fan-out at 02:00 UTC. Cron jobs run in deployed environments,
// not under `encore run`, so call EnqueueNightly by hand to test it locally.
var _ = cron.NewJob("nightly-reports", cron.JobConfig{
	Title:    "Enqueue yesterday's reports",
	Schedule: "0 2 * * *",
	Endpoint: EnqueueNightly,
})

//encore:api private
func EnqueueNightly(ctx context.Context) error {
	day := time.Now().UTC().AddDate(0, 0, -1).Format("2006-01-02")
	for _, account := range []string{"acme", "globex"} { // load from your database
		if _, err := ReportJobs.Publish(ctx, &ReportRequested{AccountID: account, Day: day}); err != nil {
			return err
		}
	}
	return nil
}

With encore run, POST /reports returns a message ID straight away, the worker builds the file a moment later and GET /reports/acme/2026-11-08 reports it as ready. Publishing the same job twice rewrites the same object, which is what makes at-least-once delivery safe here, and the local development dashboard shows each publish and handler run as a trace.

On deploy, Encore provisions SNS and SQS with a dead-letter queue on AWS, or Pub/Sub topics and subscriptions with dead-letter topics on GCP, plus the S3 or Cloud Storage bucket, and it provisions an IAM role for the service on AWS or a service account on GCP. The cron job is registered with the Encore platform, which calls EnqueueNightly on schedule and needs no scheduler in your account. A worker that falls behind can be split into its own service and scaled on its own without touching the publisher. An AI coding agent asked to add a second job type adds a topic and a subscription in the same file and can run the whole flow locally before the change is reviewed.

Best fit: Go backends on AWS or GCP that want durable jobs, retries, dead letters and scheduled work declared next to the code that uses them.

Goroutines

A goroutine is the right tool for work that is cheap to lose. The two mistakes are using the request's context, which is cancelled when the response is written, and not waiting for the goroutines on shutdown:

var jobs sync.WaitGroup

func handler(w http.ResponseWriter, r *http.Request) {
	// context.WithoutCancel keeps request values but survives the response.
	ctx := context.WithoutCancel(r.Context())
	jobs.Add(1)
	go func() {
		defer jobs.Done()
		warmCache(ctx)
	}()
	w.WriteHeader(http.StatusAccepted)
}

// In main, after srv.Shutdown(ctx): jobs.Wait()

ECS sends SIGTERM and waits 30 seconds by default before SIGKILL, configurable with the container's stopTimeout, so anything longer than that is lost on every deploy. There is no retry, no record of failure and no limit on how many goroutines a traffic spike starts. A time.Ticker for scheduled work runs once per replica, so three replicas send three nightly emails.

Best fit: best-effort work such as cache warming, metrics flushing and fire-and-forget notifications.

A Postgres-backed queue (River)

River is an open source Go job queue that stores jobs in Postgres through pgx and claims them with SELECT ... FOR UPDATE SKIP LOCKED, a Postgres feature designed for queue-like tables. Its main advantage is transactional enqueueing: a job inserted in the same transaction as the order row is only visible if the order commits, which removes a whole class of "the email went out but the order rolled back" bugs. It includes retries with backoff, unique jobs, periodic jobs and a web UI.

Every enqueue, claim and completion is a write on the primary database, and a busy queue shares vacuum, connections and IOPS with the application tables. You also run the worker processes and decide how they scale.

Best fit: Go apps on Postgres that want jobs enqueued atomically with the data they depend on, at moderate volume.

A Redis-backed queue (Asynq)

Asynq is a Go task queue on Redis with retries, scheduled and periodic tasks, priority queues, unique tasks and a monitoring UI. It is fast and familiar to anyone who has used Sidekiq or Celery. Durability depends on the Redis configuration: with eviction or without persistence, queued jobs can disappear, so the Redis instance behind it needs noeviction and AOF or a managed Redis that persists. Periodic tasks need a single scheduler process, which is one more thing to keep running exactly once.

Best fit: high-throughput job queues for teams already running Redis and comfortable operating it.

SQS with the AWS SDK

Amazon SQS is the managed queue on AWS, where a worker long-polls with ReceiveMessage, processes the message and calls DeleteMessage; anything not deleted becomes visible again after the visibility timeout and is retried, and a redrive policy moves it to a dead-letter queue after a set number of receives. The polling loop, concurrency limit, graceful shutdown and visibility extension for long jobs are yours to write, and the queue, DLQ, IAM permissions and EventBridge Scheduler rules for periodic jobs are separate Terraform. Locally it means a dev queue in AWS or an emulator.

Best fit: AWS-only Go services where a platform team already manages SQS and wants every setting in infrastructure code.

How to choose

Pick Encore when you want durable jobs with retries, dead letters and cron declared in Go, running locally under encore run and provisioned as SNS and SQS or Pub/Sub in your own cloud account. Pick River when jobs must be enqueued in the same Postgres transaction as your data, Asynq when you already run Redis and need high throughput, and raw SQS when a platform team owns the queues in Terraform. Keep goroutines for work that is fine to lose.

Related guides

• How to add Pub/Sub to a Go API
• How to add cron jobs to a Go API
• Background jobs in Node.js
• What is Amazon SQS?
• Durable workflows