# Object Storage

> Simple and scalable storage APIs for files and unstructured data

Object Storage is a simple and scalable way to store files and unstructured data. The most common implementation is Amazon S3, and its semantics are supported by every major cloud provider.

Encore.rs gives you a cloud-agnostic API for object storage. Encore provisions the buckets your app declares: Amazon S3 or Google Cloud Storage in the cloud, and a local emulator under `encore run`. You also get:

* Tracing of every object storage operation
* Local development without any setup
* A bucket of its own for each test under `encore test`

## Creating a bucket

The core of object storage is the **bucket**, a collection of objects (files) addressed by key. Declare a bucket as a module-level `static`:

```rust
-- src/storage.rs --
use encore::objects::Bucket;

pub static PROFILE_PICTURES: Bucket = Bucket::new("profile-pictures");
```

Encore reads the declaration from your source code to provision the bucket, so:

* It must be a module-level `static`, not a local variable or a value built inside a function.
* The name must be a string literal: lowercase letters, digits and dashes, starting with a letter.

Options follow the name, each a method of its own:

```rust
static AVATARS: Bucket = Bucket::new("avatars").public().versioned();
```

| Option | Effect |
| --- | --- |
| `.public()` | Objects are readable by anyone, at the URL [`public_url`](#public-buckets) gives. |
| `.versioned()` | An object keeps its earlier versions, which the `_with` operations can name. |

The options are part of the declaration: they take the bucket by value, so they can't be called on the `static` later.

## Uploading objects

`upload` stores bytes under a key and returns the stored object's attributes:

```rust
let attrs = PROFILE_PICTURES.upload("user-123.jpg", image_bytes).await?;
println!("stored {} bytes", attrs.size);
```

`upload_with` takes `UploadOptions` as well, to set the content type or a precondition:

```rust
use encore::objects::{UploadOptions, UploadPreconditions};

PROFILE_PICTURES
    .upload_with(
        "user-123.jpg",
        image_bytes,
        UploadOptions {
            content_type: Some("image/jpeg".into()),
            // Fail rather than overwrite an existing object.
            preconditions: Some(UploadPreconditions { not_exists: Some(true) }),
        },
    )
    .await?;
```

If the precondition doesn't hold, the upload fails with `ErrCode::FailedPrecondition`.

### Streaming uploads

To store an object without buffering it in memory, upload from a reader or a stream:

* `upload_stream(key, reader, options)` takes anything that is `tokio::io::AsyncRead + Unpin + Send + Sync + 'static`, such as a `tokio::fs::File`.
* `upload_chunks(key, chunks, options)` takes a `Stream` of `Result<B, E>` chunks, where a chunk is anything that converts into `Bytes` (`Bytes`, `Vec<u8>`, `String`, …).

For example, to store the body of an axum request as it arrives:

```rust
use axum::body::Body;
use axum::extract::Path;
use axum::http::StatusCode;
use encore::objects::{UploadOptions, UploadPreconditions};

async fn upload(Path(name): Path<String>, body: Body) -> StatusCode {
    let options = UploadOptions {
        // Fail rather than overwrite an existing object.
        preconditions: Some(UploadPreconditions { not_exists: Some(true) }),
        ..Default::default()
    };
    let stored = PROFILE_PICTURES
        .upload_chunks(&name, body.into_data_stream(), options)
        .await;
    match stored {
        Ok(_) => StatusCode::CREATED,
        Err(_) => StatusCode::INTERNAL_SERVER_ERROR,
    }
}
```

If the stream fails partway, the upload fails and nothing is stored.

This handler uses the key from the path as it is. In a real service, authenticate the caller, check they may write that key (or generate the key on the server), and limit the body size before uploading.

## Downloading objects

`download` returns an object's full contents as a `Vec<u8>`:

```rust
let data = PROFILE_PICTURES.download("user-123.jpg").await?;
```

`download_with` takes `DownloadOptions`: a `version` of a versioned bucket's object, or a `ByteRange` to read only part of the object:

```rust
use encore::objects::{ByteRange, DownloadOptions};

// The 256 bytes starting at offset 1024.
let part = PROFILE_PICTURES
    .download_with(
        "big.bin",
        DownloadOptions {
            range: Some(ByteRange { offset: 1024, length: Some(256) }),
            ..Default::default()
        },
    )
    .await?;
```

A `ByteRange` with `length: None` reads to the end of the object. A zero-length range is an `InvalidArgument` error.

### Streaming downloads

`download_stream(key, options)` returns a `Download`, which hands you the object as it arrives. Take it:

* chunk by chunk, with `download.next().await`, which returns `Option<Result<Bytes, Error>>`;
* as a `futures_core::Stream` of the same items;
* all at once, with `download.bytes().await`;
* as a `tokio::io::AsyncRead`, with `download.into_async_read()`, for whatever takes a reader.

```rust
use encore::objects::DownloadOptions;

let mut download = PROFILE_PICTURES
    .download_stream("big.bin", DownloadOptions::default())
    .await?;
while let Some(chunk) = download.next().await {
    hasher.update(&chunk?);
}

// Or straight into a file.
let mut reader = PROFILE_PICTURES
    .download_stream("big.bin", DownloadOptions::default())
    .await?
    .into_async_read();
tokio::io::copy(&mut reader, &mut file).await?;
```

Since a `Download` is a `Stream`, it can be the body of an axum response:

```rust
use axum::body::Body;

let download = PROFILE_PICTURES
    .download_stream(&name, DownloadOptions::default())
    .await?;
let body = Body::from_stream(download);
```

## Object attributes

`attrs` returns an object's `ObjectAttrs`: its `name`, `size`, `content_type`, `etag`, and `version` (in a versioned bucket). `exists` says whether there is an object at the key:

```rust
use encore::ErrCode;

match PROFILE_PICTURES.attrs("user-123.jpg").await {
    Ok(attrs) => println!("{} bytes, {:?}", attrs.size, attrs.content_type),
    Err(err) if err.code == ErrCode::NotFound => println!("no such picture"),
    Err(err) => return Err(err),
}

if PROFILE_PICTURES.exists("user-123.jpg").await? {
    // …
}
```

`attrs_with` and `exists_with` take `AttrsOptions` and `ExistsOptions`, whose `version` names a specific version.

## Listing objects

`list` takes `ListOptions` — a key `prefix` and a `limit`, both optional — and returns a `ListIterator` of `ListEntry`s, each with the object's `name`, `size` and `etag`:

```rust
use encore::objects::ListOptions;

let mut entries = PROFILE_PICTURES
    .list(ListOptions {
        prefix: Some("user-".into()),
        ..Default::default()
    })
    .await?;
while let Some(entry) = entries.next().await {
    println!("{}", entry?.name);
}
```

A `ListIterator` is also a `Stream`, so it works with `StreamExt` combinators.

## Removing objects

`remove` deletes an object. `remove_with` takes `RemoveOptions`, whose `version` deletes a specific version:

```rust
match PROFILE_PICTURES.remove("user-123.jpg").await {
    Ok(()) => {}
    Err(err) if err.code == ErrCode::NotFound => {}
    Err(err) => return Err(err),
}
```

## Versioned buckets

In a bucket declared `.versioned()`, overwriting or removing an object keeps its earlier versions. `ObjectAttrs::version` identifies the version an upload stored, and every operation that reads or removes an object has a `_with` form whose options take a `version`:

```rust
use encore::objects::{Bucket, DownloadOptions};

static DOCUMENTS: Bucket = Bucket::new("documents").versioned();

let first = DOCUMENTS.upload("contract.pdf", draft).await?;
DOCUMENTS.upload("contract.pdf", signed).await?;

// The draft is still there.
let draft = DOCUMENTS
    .download_with(
        "contract.pdf",
        DownloadOptions {
            version: first.version.clone(),
            ..Default::default()
        },
    )
    .await?;
```

## Public buckets

A bucket declared `.public()` serves its objects to anyone over HTTP(S), without authentication. This suits static assets like images and videos.

```rust
static PUBLIC_ASSETS: Bucket = Bucket::new("public-assets").public();

let url = PUBLIC_ASSETS.public_url("images/logo.png")?;
```

`public_url` is not `async`: it builds the URL without a request. On a bucket that isn't public it fails with `ErrCode::FailedPrecondition`.

When you deploy with the Encore platform, Encore makes the bucket publicly readable and [configures a CDN](/docs/platform/infrastructure/infra#production-infrastructure) in front of it. When self-hosting, see how to configure public buckets in the [infrastructure configuration docs](/docs/self-host/configure-infra).

## Signed URLs

A signed URL lets someone without credentials upload or download one object directly, for a limited time. Anyone holding the URL can use it.

```rust
use std::time::Duration;
use encore::objects::SignedUrlOptions;

// Valid for an hour.
let upload = PROFILE_PICTURES.signed_upload_url("user-123.jpg").await?;

// Valid for five minutes.
let download = PROFILE_PICTURES
    .signed_download_url_with(
        "user-123.jpg",
        SignedUrlOptions {
            ttl: Some(Duration::from_secs(5 * 60)),
        },
    )
    .await?;

println!("{} {}", upload.url, download.url);
```

A URL is valid for one hour unless `ttl` says otherwise, and for at most seven days. The client uploads with a `PUT` of the object's contents:

```bash
curl -X PUT --data-binary @dog-wizard.jpeg "https://storage.googleapis.com/profile-pictures/user-123.jpg?x-goog-signature=b7a1<...>"
```

Signed URLs keep large or long-running transfers out of your service: the content flows between the client and the bucket, and only keys and metadata go through your code. For content that isn't private, prefer a [public bucket](#public-buckets): its URLs are served through a CDN.

## Errors

Every operation returns `Result<_, encore::Error>`. Match on its `code` (an `encore::ErrCode`) to tell failures apart:

| Code | When |
| --- | --- |
| `NotFound` | There is no object at the key (or no such version). |
| `FailedPrecondition` | An upload precondition didn't hold, or `public_url` was called on a private bucket. |
| `InvalidArgument` | The arguments were wrong: an empty byte range, a malformed version. |
| `Unavailable` | The storage backend couldn't be reached or failed. |
| `Internal` | Something else went wrong. |

## Passing a bucket to other code

Encore reads which operations each service calls on a bucket — `upload`, `download`, `list`, … — and grants the service exactly those permissions. To make that possible, a `Bucket` can only be used by calling its methods on the `static` directly. It can't be stored, returned, or passed to a function.

To hand a bucket to other code, make a `BucketRef` with `reference`, naming the permissions it needs:

```rust
use encore::objects::{perms, Bucket, Uploader};

static UPLOADS: Bucket = Bucket::new("uploads");

// Library code asks for the permissions it needs, as traits.
async fn save(bucket: &impl Uploader, key: &str) -> Result<(), encore::Error> {
    bucket.upload(key, b"hello".to_vec()).await?;
    Ok(())
}

async fn greet() -> Result<(), encore::Error> {
    save(&UPLOADS.reference::<perms::Uploader>(), "greeting.txt").await
}
```

The service is granted every permission the reference names, whether or not the code it's passed to uses them. The permissions must be written as `reference`'s type argument: they're never inferred, not even from the type the reference is assigned to.

The permissions are the types in `encore::objects::perms`. Each has a trait of the same name in `encore::objects` with the methods it grants:

| Permission | Grants |
| --- | --- |
| `perms::Uploader` | `upload`, `upload_with`, `upload_stream`, `upload_chunks` |
| `perms::Downloader` | `download`, `download_with`, `download_stream` |
| `perms::Attrser` | `attrs`, `attrs_with`, `exists`, `exists_with` |
| `perms::Lister` | `list` |
| `perms::Remover` | `remove`, `remove_with` |
| `perms::SignedUploader` | `signed_upload_url`, `signed_upload_url_with` |
| `perms::SignedDownloader` | `signed_download_url`, `signed_download_url_with` |
| `perms::PublicUrler` | `public_url` |
| `perms::ReadWriter` | Every permission above except `PublicUrler` |

Ask for several with a tuple. A `BucketRef` is `Copy`, and `reference` is `const`, so a reference can also initialize a `static`:

```rust
use encore::objects::{perms, BucketRef, Lister, Uploader};

static AVATARS: BucketRef<(perms::Uploader, perms::Lister)> =
    UPLOADS.reference::<(perms::Uploader, perms::Lister)>();

async fn sync(bucket: &(impl Uploader + Lister)) -> Result<(), encore::Error> {
    let mut entries = bucket.list(Default::default()).await?;
    // …
    Ok(())
}
```

A `BucketRef` has the bucket's methods for the permissions it grants, so calling them doesn't need the traits in scope. Library code that takes the traits instead, as `save` and `sync` do above, works with any reference that grants them, and can be tested against a fake that implements them.

## Using a bucket from another service

A bucket can be used from any service. In a service other than the one that declares it, bind it by name with `Bucket::named`:

```rust
use encore::objects::Bucket;

static UPLOADS: Bucket = Bucket::named("uploads");
```

The name must be a string literal. A named bucket takes no options: it's configured where it's declared, which may be in a service written in another language.

## Testing

Under `encore test`, each test gets a bucket of its own, empty unless a fixture fills it: what one test uploads, no other test sees. See [Testing](/docs/rs/develop/testing).