Use the google-cloud-storage library to upload, download and stream objects, avoid overwrites with preconditions, and generate V4 signed URLs on Cloud Run without a key file, plus buckets declared in Python.

Google Cloud Storage holds the files a Python backend produces and receives: invoices, uploads, exports and backups. The google-cloud-storage library covers all of it, but a few defaults surprise people, such as uploads that overwrite silently and signed URLs that fail on Cloud Run.
This guide walks through uploads, downloads, streaming, preconditions and signed URLs with version 3.x of the library.
Install the library with pip install google-cloud-storage and create one client per process. It picks up Application Default Credentials, which on Cloud Run or GKE means the runtime service account. This module covers the common operations:
from datetime import timedelta
from typing import Optional
import google.auth
from google.auth.transport import requests as grequests
from google.api_core.exceptions import NotFound, PreconditionFailed
from google.cloud import storage
client = storage.Client() # one client per process; picks up ADC
bucket = client.bucket("acme-invoices")
def upload_invoice(path: str, key: str) -> bool:
blob = bucket.blob(key)
try:
# Files over 8 MiB switch to a resumable upload automatically.
# if_generation_match=0 means "only create": an existing key fails.
blob.upload_from_filename(path, content_type="application/pdf", if_generation_match=0)
except PreconditionFailed:
return False # an invoice with this key already exists
return True
def read_invoice(key: str) -> Optional[bytes]:
try:
return bucket.blob(key).download_as_bytes()
except NotFound:
return None
def stream_lines(key: str) -> int:
# blob.open streams in chunks instead of loading the object into memory.
count = 0
with bucket.blob(key).open("rt") as f:
for _ in f:
count += 1
return count
client.bucket() only builds a reference without a network call, while client.get_bucket() fetches the bucket and needs storage.buckets.get, a permission a writer-only service account often lacks. Uploads above 8 MiB use resumable uploads, which retry interrupted chunks instead of restarting.
GCS replaces an object on upload unless you say otherwise. Request preconditions make writes conditional: if_generation_match=0 means "create only", and passing the generation you read earlier makes an update fail if someone else wrote in between. Preconditions also make retries safe, which is why the library only retries some mutating calls automatically when a precondition is set.
Signed URLs let a browser download or upload one object for a limited time. Locally with a service account key file, generate_signed_url signs with the private key. On Cloud Run and GKE there is no key, so pass the service account email and an access token, and the library signs through the IAM Credentials API:
def download_url(key: str) -> str:
# On Cloud Run and GKE there is no private key file, so sign through the
# IAM Credentials API with the runtime service account's access token.
credentials, _ = google.auth.default()
credentials.refresh(grequests.Request())
return bucket.blob(key).generate_signed_url(
version="v4",
expiration=timedelta(minutes=15),
method="GET",
service_account_email=credentials.service_account_email,
access_token=credentials.token,
)
The service account needs roles/iam.serviceAccountTokenCreator on itself for the signing call, and V4 URLs expire after at most seven days.
fake-gcs-server implements the JSON API well enough for the calls above. Start it, create the bucket directory, and point the client at it:
With the emulator, create the client with AnonymousCredentials() and a project ID. Uploads, downloads, preconditions and streaming behave as they do against GCS, while signed URLs and IAM checks are not emulated.
Encore works alongside your Python framework, here FastAPI, as an SDK that declares infrastructure in code. Encore for Python is in private beta, and you can request an invite. A bucket is a module-level variable, encore run serves it from local storage, and a deploy creates a Cloud Storage bucket in your own GCP project, or an S3 bucket on AWS.
import encore
from encore.storage import objects
from fastapi import FastAPI, HTTPException, Request
# Cloud Storage on GCP, S3 on AWS, local files under `encore run`.
invoices = objects.Bucket("invoices", versioned=False)
app = FastAPI()
@app.put("/invoices/{invoice_id}", status_code=201)
async def upload_invoice(invoice_id: str, request: Request) -> dict:
data = await request.body()
try:
attrs = await invoices.upload(
f"{invoice_id}.pdf",
data,
content_type="application/pdf",
if_not_exists=True, # never overwrite an issued invoice
)
except objects.PreconditionFailed:
raise HTTPException(status_code=409, detail="invoice already uploaded")
return {"key": attrs.name, "size": attrs.size}
@app.get("/invoices/{invoice_id}/link")
async def invoice_link(invoice_id: str) -> dict:
key = f"{invoice_id}.pdf"
if not await invoices.exists(key):
raise HTTPException(status_code=404, detail="no such invoice")
# Encore signs the URL; there is no key file or signing setup in your code.
url = await invoices.signed_download_url(key, ttl=900)
return {"url": url}
if __name__ == "__main__":
encore.run(app)
The bucket methods are async, so they fit FastAPI handlers without a thread pool, and the create-only upload maps to the same precondition as if_generation_match=0. The service is registered with an encore.service.json file next to the code, and the SDK needs Python 3.9 or later.
On deploy, Encore provisions an IAM role for the service, or a service account on GCP. A coding agent adding a thumbnails bucket declares it next to the handler, tests it under encore run, and gets the bucket on the next deploy.
Request a private-beta invite to try this setup:
Create a storage.Client(), get the bucket with client.bucket(name), then call bucket.blob(key).upload_from_filename(path, content_type=...). For data already in memory use upload_from_string, and for an open file use upload_from_file. Files over 8 MiB are sent as resumable uploads automatically.
Pass if_generation_match=0 to the upload, and GCS then only creates the object if no live version exists, and the call raises PreconditionFailed when the key is taken. The same parameter with a specific generation number makes an update conditional on nobody else having changed the object.
Cloud Run credentials have no private key, so the default signing path raises an error saying the credentials need a private key. Pass service_account_email and access_token from the refreshed default credentials, which makes the library sign through the IAM Credentials API. The service account also needs the Service Account Token Creator role on itself.
Run fake-gcs-server and set STORAGE_EMULATOR_HOST, and the Python client sends requests to it with anonymous credentials. Encore takes another route, where encore run serves declared buckets from local storage with no emulator to manage.
Deploy to your cloud on AWS and Google Cloud
without managing infrastructure
Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.