Nov 19, 20265 min read

Upload and Download Files with Google Cloud Storage in Python

Use the google-cloud-storage library to upload, download and stream objects, avoid overwrites with preconditions, and generate V4 signed URLs on Cloud Run without a key file, plus buckets declared in Python.

Illustration for Upload and Download Files with Google Cloud Storage in Python

Google Cloud Storage holds the files a Python backend produces and receives: invoices, uploads, exports and backups. The google-cloud-storage library covers all of it, but a few defaults surprise people, such as uploads that overwrite silently and signed URLs that fail on Cloud Run.

This guide walks through uploads, downloads, streaming, preconditions and signed URLs with version 3.x of the library.

Cloud Storage in Python at a glance

TaskMethodWatch out for
Upload a fileblob.upload_from_filename(path)Overwrites by default
Upload bytesblob.upload_from_string(data)Set content_type explicitly
Downloadblob.download_as_bytes()Loads the whole object into memory
Streamblob.open("rb") or "rt"Reads in chunks
Create onlyif_generation_match=0Raises PreconditionFailed if the key exists
Share temporarilyblob.generate_signed_url(version="v4")Needs a signer on Cloud Run

How do you upload and download files?

Install the library with pip install google-cloud-storage and create one client per process. It picks up Application Default Credentials, which on Cloud Run or GKE means the runtime service account. This module covers the common operations:

from datetime import timedelta from typing import Optional import google.auth from google.auth.transport import requests as grequests from google.api_core.exceptions import NotFound, PreconditionFailed from google.cloud import storage client = storage.Client() # one client per process; picks up ADC bucket = client.bucket("acme-invoices") def upload_invoice(path: str, key: str) -> bool: blob = bucket.blob(key) try: # Files over 8 MiB switch to a resumable upload automatically. # if_generation_match=0 means "only create": an existing key fails. blob.upload_from_filename(path, content_type="application/pdf", if_generation_match=0) except PreconditionFailed: return False # an invoice with this key already exists return True def read_invoice(key: str) -> Optional[bytes]: try: return bucket.blob(key).download_as_bytes() except NotFound: return None def stream_lines(key: str) -> int: # blob.open streams in chunks instead of loading the object into memory. count = 0 with bucket.blob(key).open("rt") as f: for _ in f: count += 1 return count

client.bucket() only builds a reference without a network call, while client.get_bucket() fetches the bucket and needs storage.buckets.get, a permission a writer-only service account often lacks. Uploads above 8 MiB use resumable uploads, which retry interrupted chunks instead of restarting.

How do you avoid overwriting objects?

GCS replaces an object on upload unless you say otherwise. Request preconditions make writes conditional: if_generation_match=0 means "create only", and passing the generation you read earlier makes an update fail if someone else wrote in between. Preconditions also make retries safe, which is why the library only retries some mutating calls automatically when a precondition is set.

How do you generate signed URLs on Cloud Run?

Signed URLs let a browser download or upload one object for a limited time. Locally with a service account key file, generate_signed_url signs with the private key. On Cloud Run and GKE there is no key, so pass the service account email and an access token, and the library signs through the IAM Credentials API:

def download_url(key: str) -> str: # On Cloud Run and GKE there is no private key file, so sign through the # IAM Credentials API with the runtime service account's access token. credentials, _ = google.auth.default() credentials.refresh(grequests.Request()) return bucket.blob(key).generate_signed_url( version="v4", expiration=timedelta(minutes=15), method="GET", service_account_email=credentials.service_account_email, access_token=credentials.token, )

The service account needs roles/iam.serviceAccountTokenCreator on itself for the signing call, and V4 URLs expire after at most seven days.

How do you test it locally?

fake-gcs-server implements the JSON API well enough for the calls above. Start it, create the bucket directory, and point the client at it:

~
$ mkdir -p data/acme-invoices$ fake-gcs-server -scheme http -port 4443 -data data$ export STORAGE_EMULATOR_HOST=http://localhost:4443

With the emulator, create the client with AnonymousCredentials() and a project ID. Uploads, downloads, preconditions and streaming behave as they do against GCS, while signed URLs and IAM checks are not emulated.

Buckets declared in Python with Encore

Encore

Encore works alongside your Python framework, here FastAPI, as an SDK that declares infrastructure in code. Encore for Python is in private beta, and you can request an invite. A bucket is a module-level variable, encore run serves it from local storage, and a deploy creates a Cloud Storage bucket in your own GCP project, or an S3 bucket on AWS.

import encore from encore.storage import objects from fastapi import FastAPI, HTTPException, Request # Cloud Storage on GCP, S3 on AWS, local files under `encore run`. invoices = objects.Bucket("invoices", versioned=False) app = FastAPI() @app.put("/invoices/{invoice_id}", status_code=201) async def upload_invoice(invoice_id: str, request: Request) -> dict: data = await request.body() try: attrs = await invoices.upload( f"{invoice_id}.pdf", data, content_type="application/pdf", if_not_exists=True, # never overwrite an issued invoice ) except objects.PreconditionFailed: raise HTTPException(status_code=409, detail="invoice already uploaded") return {"key": attrs.name, "size": attrs.size} @app.get("/invoices/{invoice_id}/link") async def invoice_link(invoice_id: str) -> dict: key = f"{invoice_id}.pdf" if not await invoices.exists(key): raise HTTPException(status_code=404, detail="no such invoice") # Encore signs the URL; there is no key file or signing setup in your code. url = await invoices.signed_download_url(key, ttl=900) return {"url": url} if __name__ == "__main__": encore.run(app)

The bucket methods are async, so they fit FastAPI handlers without a thread pool, and the create-only upload maps to the same precondition as if_generation_match=0. The service is registered with an encore.service.json file next to the code, and the SDK needs Python 3.9 or later.

On deploy, Encore provisions an IAM role for the service, or a service account on GCP. A coding agent adding a thumbnails bucket declares it next to the handler, tests it under encore run, and gets the bucket on the next deploy.

Request a private-beta invite to try this setup:

Try Encore →

Frequently asked questions

How do I upload a file to Google Cloud Storage in Python?

Create a storage.Client(), get the bucket with client.bucket(name), then call bucket.blob(key).upload_from_filename(path, content_type=...). For data already in memory use upload_from_string, and for an open file use upload_from_file. Files over 8 MiB are sent as resumable uploads automatically.

How do I prevent overwriting an existing object?

Pass if_generation_match=0 to the upload, and GCS then only creates the object if no live version exists, and the call raises PreconditionFailed when the key is taken. The same parameter with a specific generation number makes an update conditional on nobody else having changed the object.

Why does generate_signed_url fail on Cloud Run?

Cloud Run credentials have no private key, so the default signing path raises an error saying the credentials need a private key. Pass service_account_email and access_token from the refreshed default credentials, which makes the library sign through the IAM Credentials API. The service account also needs the Service Account Token Creator role on itself.

How do I test Google Cloud Storage code locally?

Run fake-gcs-server and set STORAGE_EMULATOR_HOST, and the Python client sends requests to it with anonymous credentials. Encore takes another route, where encore run serves declared buckets from local storage with no emulator to manage.

Deploy to your cloud on AWS and Google Cloud
without managing infrastructure

Encore lets developers and agents define application infrastructure directly in code, then automatically provisions it from local development to production in your AWS or GCP account.

~/orders — claude
Claude Code
Claude Code v2.1.180Opus 5.5~/orders