How to Add Object Storage to a Go API

Most Go APIs need to store files at some point, such as profile pictures or invoice PDFs. Putting the bytes in Postgres works for a while, until the table gets large and backups slow down. An S3 bucket is the usual next step, and it comes with an SDK client, credentials, a bucket policy and a separate setup for local development.

This guide adds avatar uploads to a Go API. The upload endpoint streams the body into a bucket with a size limit and a content type check, and a second endpoint returns a signed download link that expires after 15 minutes. It is shown with Encore and with the AWS SDK for Go v2 against S3.

Object storage options for a Go API at a glance

Encore: a bucket declared in the service

Encore combines an open source infrastructure SDK for Go and TypeScript with a platform that deploys into your own AWS or GCP account. A bucket is a package-level variable from the object storage package. The upload uses a raw endpoint so the body can be streamed into the bucket instead of read into memory:

// Service avatars stores profile pictures in object storage.
package avatars

import (
	"context"
	"io"
	"net/http"
	"time"

	"encore.dev"
	"encore.dev/beta/errs"
	"encore.dev/storage/objects"
)

// An S3 bucket on AWS, a Cloud Storage bucket on GCP.
var avatars = objects.NewBucket("avatars", objects.BucketConfig{})

const maxSize = 5 << 20 // 5 MB

// Upload streams the request body into the bucket.
//
//encore:api public raw method=PUT path=/avatars/:userID
func Upload(w http.ResponseWriter, req *http.Request) {
	userID := encore.CurrentRequest().PathParams.Get("userID")
	ctype := req.Header.Get("Content-Type")
	if ctype != "image/png" && ctype != "image/jpeg" {
		http.Error(w, "only png and jpeg", http.StatusUnsupportedMediaType)
		return
	}

	up := avatars.Upload(req.Context(), "users/"+userID, objects.WithUploadAttrs(objects.UploadAttrs{
		ContentType: ctype,
	}))
	_, err := io.Copy(up, http.MaxBytesReader(w, req.Body, maxSize))
	if err != nil {
		up.Abort(err)
		http.Error(w, "upload failed", http.StatusBadRequest)
		return
	}
	if err := up.Close(); err != nil {
		http.Error(w, "upload failed", http.StatusInternalServerError)
		return
	}
	w.WriteHeader(http.StatusNoContent)
}

type URLResponse struct {
	URL string `json:"url"`
}

// URL returns a short-lived download link for a user's avatar.
//
//encore:api public method=GET path=/avatars/:userID/url
func URL(ctx context.Context, userID string) (*URLResponse, error) {
	key := "users/" + userID
	ok, err := avatars.Exists(ctx, key)
	if err != nil {
		return nil, err
	}
	if !ok {
		return nil, &errs.Error{Code: errs.NotFound, Message: "no avatar"}
	}
	u, err := avatars.SignedDownloadURL(ctx, key, objects.WithTTL(15*time.Minute))
	if err != nil {
		return nil, err
	}
	return &URLResponse{URL: u.URL}, nil
}

Abort discards a partial upload when the body is too large or the connection drops, so a half-written object never replaces a good one. In a real app the userID would come from the authenticated user rather than the path. Under encore run the bucket is backed by a local object store:

On deploy, Encore creates the bucket in your account and provisions an IAM role for the service, and the signed URLs come from S3 or Cloud Storage. For large files, SignedUploadURL gives the client a URL to upload to directly, and BucketConfig{Public: true} with PublicURL serves files such as images through a public bucket. Each environment gets its own bucket, so preview environments never write into production data.

A coding agent that adds a bucket writes the declaration and the upload handler in one change, and can curl the endpoint under encore run before anything touches AWS.

aws-sdk-go-v2 and S3

With the AWS SDK, the service creates an S3 client from the default credential chain and streams the body with the upload manager, which handles multipart uploads for larger files:

cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
	log.Fatal(err)
}
client := s3.NewFromConfig(cfg)
uploader := manager.NewUploader(client)
presigner := s3.NewPresignClient(client)

// In the upload handler:
_, err = uploader.Upload(r.Context(), &s3.PutObjectInput{
	Bucket:      aws.String(os.Getenv("AVATARS_BUCKET")),
	Key:         aws.String("users/" + userID),
	Body:        http.MaxBytesReader(w, r.Body, maxSize),
	ContentType: aws.String(ctype),
})

// In the download handler:
req, err := presigner.PresignGetObject(r.Context(), &s3.GetObjectInput{
	Bucket: aws.String(os.Getenv("AVATARS_BUCKET")),
	Key:    aws.String("users/" + userID),
}, s3.WithPresignExpires(15*time.Minute))

The bucket itself comes from the console, the CLI or Terraform, with Block Public Access left on. The ECS task role or Lambda execution role needs s3:PutObject and s3:GetObject on that bucket, and a presigned URL only works while the credentials that signed it are valid, so URLs signed with temporary role credentials can expire before their own TTL.

Good to know

For local development, MinIO speaks the S3 API. The client needs o.BaseEndpoint set to the MinIO address and o.UsePathStyle = true, which is configuration that only exists for local runs. Checking HeadObject before signing a URL avoids handing out links to objects that do not exist.

How to choose

Pick Encore when you want the bucket, its IAM role and signed URLs to come from the Go code, with a working local object store under encore run and the same code on S3 or Cloud Storage in your own account. The service can add a database for file metadata alongside the bucket.

Pick the AWS SDK directly when the bucket already exists under Terraform, or when you need S3 features such as lifecycle rules or replication configured line by line.

Related guides

• How to add an S3 bucket to AWS with an AI agent
• How to add file uploads to a TypeScript API
• How to add a database to a Go API
• How to deploy a Go API to AWS