Nov 16, 20268 min read

How to Add Pub/Sub to a Fiber App

Publish events from a Go Fiber v3 handler and process them in idempotent subscribers with retries, using Encore on SNS and SQS or GCP Pub/Sub, or the AWS SDK by hand.

Illustration for How to Add Pub/Sub to a Fiber App

A Fiber handler should answer quickly, so work like sending emails, revoking sessions or calling a CRM usually moves out of the request and into an event that other code processes. Starting a goroutine looks like the shortcut, but it has no retry and dies with the process on the next deploy, and the durable version on AWS means an SNS topic, an SQS queue per consumer, a dead-letter queue for each, IAM for every publisher and consumer, and a local stand-in for all of it.

This guide adds Pub/Sub to a Fiber v3 API that handles account deletion requests by fanning one event out to two consumers, one that revokes sessions and one that sends a confirmation email exactly once. It starts with Encore, which declares the topic and subscriptions in Go and runs them locally and in your own AWS or GCP account, then shows the SNS and SQS version with the AWS SDK.

Pub/Sub options for a Fiber app at a glance

EncoreSNS + SQS by handGoroutine in the handler
Declared inGo code (pubsub.NewTopic)Terraform or the AWS consoleNothing to declare
Local developmentRuns with encore runLocalStack or a real AWS accountRuns in process
Fan-outOne subscription per consumerOne SQS queue per consumerCall each function yourself
Retries and DLQRetryPolicy, DLQ per subscriptionVisibility timeout and redrive policyNone
Survives a deployYesYesNo
ProductionSNS and SQS on AWS, Pub/Sub on GCP, in your accountSNS and SQS you provisionYour process

How do you add Pub/Sub to Fiber with Encore?

Encore

Encore works alongside Fiber for this, so the Fiber handler stays the place that accepts the request while the topic and its subscribers are Go declarations in the same package. The app is mounted with Fiber's adaptor.FiberApp in one fallback route, and the Pub/Sub API gives each consumer its own subscription with its own retries.

Set up the app

Get started
brew install encoredev/tap/encore &&
encore app create
Copy

Run in your terminal to get started locally.

~
$ encore app create account-api --example=hello-world$ cd account-api && go get github.com/gofiber/fiber/v3@v3.7.0$ mkdir account

The Fiber handler publishes AccountDeleted and returns 202 with the message ID. revoke-sessions deletes the user's session set, which is safe to repeat, and send-deletion-email claims the send with SetIfNotExists before calling the email provider and releases the claim if the call fails, so a redelivery after a failure sends the email and a redelivery after a success does nothing:

// account/account.go package account import ( "context" "errors" "net/http" "strings" "time" "encore.dev/pubsub" "encore.dev/rlog" "encore.dev/storage/cache" "github.com/gofiber/fiber/v3" "github.com/gofiber/fiber/v3/middleware/adaptor" ) type AccountDeleted struct { UserID string `json:"user_id"` Email string `json:"email"` RequestedAt time.Time `json:"requested_at"` } // Every subscription gets its own copy of each event. var Deletions = pubsub.NewTopic[*AccountDeleted]("account-deleted", pubsub.TopicConfig{ DeliveryGuarantee: pubsub.AtLeastOnce, }) var cluster = cache.NewCluster("account", cache.ClusterConfig{ EvictionPolicy: cache.NoEviction, }) // Session tokens per user (written by the login flow, not shown). var sessions = cache.NewSetKeyspace[string, string](cluster, cache.KeyspaceConfig{ KeyPattern: "sessions/:key", }) // Marks a confirmation email as sent so a redelivery does not send it twice. var sent = cache.NewStringKeyspace[string](cluster, cache.KeyspaceConfig{ KeyPattern: "deletion-email/:key", DefaultExpiry: cache.ExpireIn(7 * 24 * time.Hour), }) //encore:service type Service struct { handler http.HandlerFunc } func initService() (*Service, error) { app := fiber.New() app.Post("/account/delete", requestDeletion) return &Service{handler: adaptor.FiberApp(app)}, nil } //encore:api public raw path=/!fallback func (s *Service) Fallback(w http.ResponseWriter, req *http.Request) { s.handler(w, req) } func requestDeletion(c fiber.Ctx) error { // Stand-in for your auth middleware. userID := c.Get("X-User-ID") email := c.Get("X-User-Email") if userID == "" || !strings.Contains(email, "@") { return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{"error": "not signed in"}) } id, err := Deletions.Publish(c.Context(), &AccountDeleted{ UserID: userID, Email: email, RequestedAt: time.Now().UTC(), }) if err != nil { return err } return c.Status(fiber.StatusAccepted).JSON(fiber.Map{"status": "scheduled", "message_id": id}) } // Deleting the session set is idempotent, so a redelivery is harmless. var _ = pubsub.NewSubscription(Deletions, "revoke-sessions", pubsub.SubscriptionConfig[*AccountDeleted]{ Handler: func(ctx context.Context, ev *AccountDeleted) error { if _, err := sessions.Delete(ctx, ev.UserID); err != nil { return err } rlog.Info("sessions revoked", "user", ev.UserID) return nil }, }, ) var _ = pubsub.NewSubscription(Deletions, "send-deletion-email", pubsub.SubscriptionConfig[*AccountDeleted]{ Handler: sendDeletionEmail, AckDeadline: time.Minute, RetryPolicy: &pubsub.RetryPolicy{ MinBackoff: 30 * time.Second, MaxBackoff: 10 * time.Minute, MaxRetries: 10, }, }, ) func sendDeletionEmail(ctx context.Context, ev *AccountDeleted) error { // Claim the send first; only one delivery of this event gets past here. if err := sent.SetIfNotExists(ctx, ev.UserID, ev.RequestedAt.Format(time.RFC3339)); errors.Is(err, cache.KeyExists) { return nil } else if err != nil { return err } if err := emailProvider(ctx, ev.Email); err != nil { // Release the claim so the retry can send it. _, _ = sent.Delete(ctx, ev.UserID) rlog.Warn("deletion email failed, will retry", "user", ev.UserID, "err", err) return err } rlog.Info("deletion email sent", "user", ev.UserID) return nil } // emailProvider stands in for your email API client. func emailProvider(ctx context.Context, to string) error { if strings.HasSuffix(to, "@bounce.test") { return errors.New("email provider unavailable") } return nil }

Running this with encore run, a request with the two headers returns 202 and both subscribers log within a second, and a request without them returns 401 before anything is published. For an address on bounce.test the session revocation succeeds while the email subscriber logs the failure and returns an error, and Encore delivers the message to that subscription again while the other subscription is unaffected. After MaxRetries failures the message moves to the subscription's dead-letter queue instead of blocking newer events.

AckDeadline is the time a handler has before the message is considered lost and redelivered, and the handler's ctx is cancelled when it passes. Encore uses it as the SQS visibility timeout on AWS, and it defaults to 30 seconds. On deploy, Encore provisions SNS and SQS on AWS or Pub/Sub on GCP in your own account, with a dead-letter queue per subscription and publish and consume permissions only for the services that use them.

When an agent adds a third consumer, for example one that deletes uploaded files, the change is one more pubsub.NewSubscription in Go, and encore run delivers real events to it locally before the pull request is opened.

Best fit: Fiber APIs that hand work to background consumers and want retries, dead-lettering and fan-out running locally and on SNS/SQS or Pub/Sub in their own cloud.

SNS and SQS with aws-sdk-go-v2

The hand-built version publishes to an SNS topic from the Fiber handler with aws-sdk-go-v2 and reads one subscription queue in a long-polling loop. A message is only deleted after the work succeeds, so a failure leaves it to reappear when the visibility timeout expires:

package main import ( "context" "encoding/json" "log" "os" "time" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/sns" "github.com/aws/aws-sdk-go-v2/service/sqs" "github.com/gofiber/fiber/v3" ) type AccountDeleted struct { UserID string `json:"user_id"` Email string `json:"email"` RequestedAt time.Time `json:"requested_at"` } func main() { ctx := context.Background() cfg, err := config.LoadDefaultConfig(ctx) // task role credentials on ECS if err != nil { log.Fatal(err) } snsClient, sqsClient := sns.NewFromConfig(cfg), sqs.NewFromConfig(cfg) go consume(ctx, sqsClient, os.Getenv("EMAIL_QUEUE_URL")) app := fiber.New() app.Post("/account/delete", func(c fiber.Ctx) error { body, _ := json.Marshal(AccountDeleted{UserID: c.Get("X-User-ID"), Email: c.Get("X-User-Email"), RequestedAt: time.Now().UTC()}) _, err := snsClient.Publish(c.Context(), &sns.PublishInput{ TopicArn: aws.String(os.Getenv("ACCOUNT_DELETED_TOPIC_ARN")), Message: aws.String(string(body)), }) if err != nil { return err } return c.SendStatus(fiber.StatusAccepted) }) log.Fatal(app.Listen(":8080")) } // consume long-polls one subscription queue (raw message delivery enabled). func consume(ctx context.Context, client *sqs.Client, queueURL string) { for { out, err := client.ReceiveMessage(ctx, &sqs.ReceiveMessageInput{ QueueUrl: aws.String(queueURL), MaxNumberOfMessages: 10, WaitTimeSeconds: 20, VisibilityTimeout: 60, }) if err != nil { log.Println("receive:", err) time.Sleep(time.Second) continue } for _, m := range out.Messages { var ev AccountDeleted if err := json.Unmarshal([]byte(aws.ToString(m.Body)), &ev); err != nil { continue // left for the redrive policy to move to the DLQ } if err := sendDeletionEmail(ctx, ev); err != nil { continue // becomes visible again after the visibility timeout } client.DeleteMessage(ctx, &sqs.DeleteMessageInput{QueueUrl: aws.String(queueURL), ReceiptHandle: m.ReceiptHandle}) } } } func sendDeletionEmail(ctx context.Context, ev AccountDeleted) error { return nil }

Each consumer gets its own SQS queue subscribed to the topic with raw message delivery so the body is the event JSON, a queue policy that lets the topic send to it, a dead-letter queue with a maxReceiveCount in the redrive policy, and a task role that can sns:Publish on the topic and receive and delete on its queue. The idempotency claim from the Encore version still applies, since SQS standard queues deliver at least once. Running the consumer inside the API process, as here, couples its scaling to the web tier, and splitting it out means a second ECS service with its own task definition.

Best fit: teams that already manage SNS and SQS in Terraform and want full control over queue settings and the consumer loop.

Goroutine in the handler

go sendDeletionEmail(ev) inside the Fiber handler returns fast and needs no infrastructure. The work is lost if the process restarts before it finishes, there is no retry when the email provider fails, and with several replicas there is no record of which deletions were confirmed. It is fine for best-effort work such as warming a cache, and a poor fit for anything a user or a regulation expects to happen.

Best fit: best-effort side work that can be lost without consequences.

How to choose

Pick Encore when the Fiber app needs durable events with several consumers, retries and dead-letter queues, running locally with encore run and on SNS/SQS or Pub/Sub in your own account. Pick SNS and SQS by hand when the queues are already defined in Terraform and you want every setting under your control. Keep goroutines for work that does not need to survive a restart.

Frequently asked questions

How do I add Pub/Sub to a Fiber app?

Publish from the Fiber handler to a topic, return 202, and process the event in a separate consumer with retries and a dead-letter queue. On AWS that is usually an SNS topic with one SQS queue per consumer, published with the aws-sdk-go-v2 SNS client and read by a long-polling SQS loop.

How do I add Pub/Sub to a Fiber app with Encore?

Declare a topic with pubsub.NewTopic and one pubsub.NewSubscription per consumer, mount the Fiber app in a raw endpoint with path=/!fallback through adaptor.FiberApp, and call Topic.Publish from the Fiber handler. encore run runs the topic and subscribers locally, and deploys provision SNS and SQS on AWS or Pub/Sub on GCP in your own account.

Should I process background work in a goroutine from a Fiber handler?

Only for work you can afford to lose. A goroutine started from a handler dies with the process during a deploy or crash and has no retry, so jobs like emails and data deletion belong on a queue with retries and a dead-letter queue.

How do I make a Pub/Sub subscriber idempotent?

Assume every message can arrive more than once. Make the handler's effect naturally repeatable, like deleting a key, or claim the work first with an atomic write such as SET NX keyed by the event, and release the claim if the side effect fails so the retry can run it.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders