Object Storage

Object Storage is a simple and scalable way to store files and unstructured data. The most common implementation is Amazon S3, and its semantics are supported by every major cloud provider.

Encore.rs gives you a cloud-agnostic API for object storage. Encore provisions the buckets your app declares: Amazon S3 or Google Cloud Storage in the cloud, and a local emulator under encore run. You also get:

• Tracing of every object storage operation
• Local development without any setup
• A bucket of its own for each test under encore test

Creating a bucket

The core of object storage is the bucket, a collection of objects (files) addressed by key. Declare a bucket as a module-level static:

src/storage.rs
use encore::objects::Bucket;

pub static PROFILE_PICTURES: Bucket = Bucket::new("profile-pictures");

Encore reads the declaration from your source code to provision the bucket, so:

• It must be a module-level static, not a local variable or a value built inside a function.
• The name must be a string literal: lowercase letters, digits and dashes, starting with a letter.

Options follow the name, each a method of its own:

static AVATARS: Bucket = Bucket::new("avatars").public().versioned();

The options are part of the declaration: they take the bucket by value, so they can't be called on the static later.

Uploading objects

upload stores bytes under a key and returns the stored object's attributes:

let attrs = PROFILE_PICTURES.upload("user-123.jpg", image_bytes).await?;
println!("stored {} bytes", attrs.size);

upload_with takes UploadOptions as well, to set the content type or a precondition:

use encore::objects::{UploadOptions, UploadPreconditions};

PROFILE_PICTURES
    .upload_with(
        "user-123.jpg",
        image_bytes,
        UploadOptions {
            content_type: Some("image/jpeg".into()),
            // Fail rather than overwrite an existing object.
            preconditions: Some(UploadPreconditions { not_exists: Some(true) }),
        },
    )
    .await?;

If the precondition doesn't hold, the upload fails with ErrCode::FailedPrecondition.

Streaming uploads

To store an object without buffering it in memory, upload from a reader or a stream:

• upload_stream(key, reader, options) takes anything that is tokio::io::AsyncRead + Unpin + Send + Sync + 'static, such as a tokio::fs::File.
• upload_chunks(key, chunks, options) takes a Stream of Result<B, E> chunks, where a chunk is anything that converts into Bytes (Bytes, Vec<u8>, String, …).

For example, to store the body of an axum request as it arrives:

use axum::body::Body;
use axum::extract::Path;
use axum::http::StatusCode;
use encore::objects::{UploadOptions, UploadPreconditions};

async fn upload(Path(name): Path<String>, body: Body) -> StatusCode {
    let options = UploadOptions {
        // Fail rather than overwrite an existing object.
        preconditions: Some(UploadPreconditions { not_exists: Some(true) }),
        ..Default::default()
    };
    let stored = PROFILE_PICTURES
        .upload_chunks(&name, body.into_data_stream(), options)
        .await;
    match stored {
        Ok(_) => StatusCode::CREATED,
        Err(_) => StatusCode::INTERNAL_SERVER_ERROR,
    }
}

If the stream fails partway, the upload fails and nothing is stored.

This handler uses the key from the path as it is. In a real service, authenticate the caller, check they may write that key (or generate the key on the server), and limit the body size before uploading.

Downloading objects

download returns an object's full contents as a Vec<u8>:

let data = PROFILE_PICTURES.download("user-123.jpg").await?;

download_with takes DownloadOptions: a version of a versioned bucket's object, or a ByteRange to read only part of the object:

use encore::objects::{ByteRange, DownloadOptions};

// The 256 bytes starting at offset 1024.
let part = PROFILE_PICTURES
    .download_with(
        "big.bin",
        DownloadOptions {
            range: Some(ByteRange { offset: 1024, length: Some(256) }),
            ..Default::default()
        },
    )
    .await?;

A ByteRange with length: None reads to the end of the object. A zero-length range is an InvalidArgument error.

Streaming downloads

download_stream(key, options) returns a Download, which hands you the object as it arrives. Take it:

• chunk by chunk, with download.next().await, which returns Option<Result<Bytes, Error>>;
• as a futures_core::Stream of the same items;
• all at once, with download.bytes().await;
• as a tokio::io::AsyncRead, with download.into_async_read(), for whatever takes a reader.

use encore::objects::DownloadOptions;

let mut download = PROFILE_PICTURES
    .download_stream("big.bin", DownloadOptions::default())
    .await?;
while let Some(chunk) = download.next().await {
    hasher.update(&chunk?);
}

// Or straight into a file.
let mut reader = PROFILE_PICTURES
    .download_stream("big.bin", DownloadOptions::default())
    .await?
    .into_async_read();
tokio::io::copy(&mut reader, &mut file).await?;

Since a Download is a Stream, it can be the body of an axum response:

use axum::body::Body;

let download = PROFILE_PICTURES
    .download_stream(&name, DownloadOptions::default())
    .await?;
let body = Body::from_stream(download);

Object attributes

attrs returns an object's ObjectAttrs: its name, size, content_type, etag, and version (in a versioned bucket). exists says whether there is an object at the key:

use encore::ErrCode;

match PROFILE_PICTURES.attrs("user-123.jpg").await {
    Ok(attrs) => println!("{} bytes, {:?}", attrs.size, attrs.content_type),
    Err(err) if err.code == ErrCode::NotFound => println!("no such picture"),
    Err(err) => return Err(err),
}

if PROFILE_PICTURES.exists("user-123.jpg").await? {
    // …
}

attrs_with and exists_with take AttrsOptions and ExistsOptions, whose version names a specific version.

Listing objects

list takes ListOptions — a key prefix and a limit, both optional — and returns a ListIterator of ListEntrys, each with the object's name, size and etag:

use encore::objects::ListOptions;

let mut entries = PROFILE_PICTURES
    .list(ListOptions {
        prefix: Some("user-".into()),
        ..Default::default()
    })
    .await?;
while let Some(entry) = entries.next().await {
    println!("{}", entry?.name);
}

A ListIterator is also a Stream, so it works with StreamExt combinators.

Removing objects

remove deletes an object. remove_with takes RemoveOptions, whose version deletes a specific version:

match PROFILE_PICTURES.remove("user-123.jpg").await {
    Ok(()) => {}
    Err(err) if err.code == ErrCode::NotFound => {}
    Err(err) => return Err(err),
}

Versioned buckets

In a bucket declared .versioned(), overwriting or removing an object keeps its earlier versions. ObjectAttrs::version identifies the version an upload stored, and every operation that reads or removes an object has a _with form whose options take a version:

use encore::objects::{Bucket, DownloadOptions};

static DOCUMENTS: Bucket = Bucket::new("documents").versioned();

let first = DOCUMENTS.upload("contract.pdf", draft).await?;
DOCUMENTS.upload("contract.pdf", signed).await?;

// The draft is still there.
let draft = DOCUMENTS
    .download_with(
        "contract.pdf",
        DownloadOptions {
            version: first.version.clone(),
            ..Default::default()
        },
    )
    .await?;

Public buckets

A bucket declared .public() serves its objects to anyone over HTTP(S), without authentication. This suits static assets like images and videos.

static PUBLIC_ASSETS: Bucket = Bucket::new("public-assets").public();

let url = PUBLIC_ASSETS.public_url("images/logo.png")?;

public_url is not async: it builds the URL without a request. On a bucket that isn't public it fails with ErrCode::FailedPrecondition.

When you deploy with the Encore platform, Encore makes the bucket publicly readable and configures a CDN in front of it. When self-hosting, see how to configure public buckets in the infrastructure configuration docs.

Signed URLs

A signed URL lets someone without credentials upload or download one object directly, for a limited time. Anyone holding the URL can use it.

use std::time::Duration;
use encore::objects::SignedUrlOptions;

// Valid for an hour.
let upload = PROFILE_PICTURES.signed_upload_url("user-123.jpg").await?;

// Valid for five minutes.
let download = PROFILE_PICTURES
    .signed_download_url_with(
        "user-123.jpg",
        SignedUrlOptions {
            ttl: Some(Duration::from_secs(5 * 60)),
        },
    )
    .await?;

println!("{} {}", upload.url, download.url);

A URL is valid for one hour unless ttl says otherwise, and for at most seven days. The client uploads with a PUT of the object's contents:

curl -X PUT --data-binary @dog-wizard.jpeg "https://storage.googleapis.com/profile-pictures/user-123.jpg?x-goog-signature=b7a1<...>"

Signed URLs keep large or long-running transfers out of your service: the content flows between the client and the bucket, and only keys and metadata go through your code. For content that isn't private, prefer a public bucket: its URLs are served through a CDN.

Errors

Every operation returns Result<_, encore::Error>. Match on its code (an encore::ErrCode) to tell failures apart:

Passing a bucket to other code

Encore reads which operations each service calls on a bucket — upload, download, list, … — and grants the service exactly those permissions. To make that possible, a Bucket can only be used by calling its methods on the static directly. It can't be stored, returned, or passed to a function.

To hand a bucket to other code, make a BucketRef with reference, naming the permissions it needs:

use encore::objects::{perms, Bucket, Uploader};

static UPLOADS: Bucket = Bucket::new("uploads");

// Library code asks for the permissions it needs, as traits.
async fn save(bucket: &impl Uploader, key: &str) -> Result<(), encore::Error> {
    bucket.upload(key, b"hello".to_vec()).await?;
    Ok(())
}

async fn greet() -> Result<(), encore::Error> {
    save(&UPLOADS.reference::<perms::Uploader>(), "greeting.txt").await
}

The service is granted every permission the reference names, whether or not the code it's passed to uses them. The permissions must be written as reference's type argument: they're never inferred, not even from the type the reference is assigned to.

The permissions are the types in encore::objects::perms. Each has a trait of the same name in encore::objects with the methods it grants:

Ask for several with a tuple. A BucketRef is Copy, and reference is const, so a reference can also initialize a static:

use encore::objects::{perms, BucketRef, Lister, Uploader};

static AVATARS: BucketRef<(perms::Uploader, perms::Lister)> =
    UPLOADS.reference::<(perms::Uploader, perms::Lister)>();

async fn sync(bucket: &(impl Uploader + Lister)) -> Result<(), encore::Error> {
    let mut entries = bucket.list(Default::default()).await?;
    // …
    Ok(())
}

A BucketRef has the bucket's methods for the permissions it grants, so calling them doesn't need the traits in scope. Library code that takes the traits instead, as save and sync do above, works with any reference that grants them, and can be tested against a fake that implements them.

Using a bucket from another service

A bucket can be used from any service. In a service other than the one that declares it, bind it by name with Bucket::named:

use encore::objects::Bucket;

static UPLOADS: Bucket = Bucket::named("uploads");

The name must be a string literal. A named bucket takes no options: it's configured where it's declared, which may be in a service written in another language.

Testing

Under encore test, each test gets a bucket of its own, empty unless a fixture fills it: what one test uploads, no other test sees. See Testing.