Workers

A worker is a long-running process of a service that serves no requests. It runs next to the service's main process, from an entrypoint of its own in the service's code, and does whatever that code does: consume Pub/Sub messages, run a Temporal worker, poll a queue, reconcile state on a timer.

A worker is its service at runtime. It has the service's secrets, configs, databases and permissions, and it calls other services as the service does. There is nothing to grant or share: code a worker uses is the service's code.

Enabling workers

Workers are experimental: they work, but how they are declared may still change. An app uses them only by enabling the workers feature in its encore.app.json:

encore.app.json
{
  "features": ["workers"]
}

Without it, Encore refuses to parse an app that declares workers, with an error saying what to add.

Declaring workers

Workers are declared in the service's encore.service.json, by name:

orders/encore.service.json
{
  "name": "orders",
  "lang": "python",
  "api": [
    { "type": "http", "prefix": "/", "name": "orders", "expose": true }
  ],
  "workers": {
    "fulfillment": { "entry": "fulfillment_worker.py" },
    "temporal": { "entry": "-m orders.temporal_worker", "shutdown_timeout": "10m" },
    "reconciler": { "entry": "reconciler.py", "instances": "one" }
  }
}

A worker's name is lowercase letters, digits and underscores, unique within its service. Within the app, Encore calls it <service>.<worker>, as in orders.fulfillment.

Writing a worker

A worker's entrypoint is an ordinary Python program that starts Encore and runs until it is told to stop. One that only consumes Pub/Sub messages needs nothing more than encore.run() without an app:

orders/fulfillment_worker.py
import encore

import subscriptions  # declares the subscriptions this worker consumes

encore.run()

One with work of its own runs it until shutdown begins, then lets it finish. See Graceful shutdown for how:

orders/reconciler.py
import asyncio
import signal

import encore
from encore import shutdown


async def reconcile(stop: asyncio.Event) -> None:
    while not stop.is_set():
        ...  # compare and fix state, using Encore resources
        try:
            await asyncio.wait_for(stop.wait(), timeout=60)
        except asyncio.TimeoutError:
            pass


async def main() -> int:
    await encore.start()
    stop = asyncio.Event()
    asyncio.get_running_loop().add_signal_handler(signal.SIGTERM, stop.set)
    task = asyncio.create_task(reconcile(stop))

    await stop.wait()
    shutdown.begin(cooperative=True)
    await task
    shutdown.mark_application_drained()
    return (await shutdown.completed()).code()


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))

encore.env.app_meta().deploy.worker is the name of the worker a process runs, and None in the service's main process.

Consuming Pub/Sub messages

A subscription names the worker that consumes it with worker:

orders/subscriptions.py
from encore import pubsub

from topics import orders, Order


async def fulfil(message: pubsub.Message[Order]) -> None:
    ...


fulfil_orders = pubsub.Subscription[Order](
    orders, "fulfil-orders", handler=fulfil, worker="fulfillment"
)

Only that worker's process takes the subscription's messages. The service's main process, and its other workers, take none, even when their code imports the module that declares it. Without worker, the service's main process consumes the subscription, as it always has.

Encore reads worker from the source, so write it as a string, and it must name a worker the service declares.

A service with only workers

A service that serves nothing has no main process: set its run to "none".

indexer/encore.service.json
{
  "name": "indexer",
  "lang": "python",
  "run": "none",
  "workers": {
    "index": { "entry": "index_worker.py" }
  }
}

Such a service can't declare an api, a frontend or a gateway, or cron jobs, which call a path of their service. Each subscription it declares needs a worker, since no main process would consume it.

Cron jobs

A cron job calls an HTTP path of its service, which only the main process serves, so it can't target a worker. Declare it in the service, and have its endpoint publish a message a worker consumes if the work belongs in the worker.

Running workers locally

Under encore run, a service's workers start when the service does, and keep running while anything keeps the service up. Unlike the main process, a worker is never stopped for being idle, since no request would start it again. A message waiting on a subscription a worker consumes starts the worker too.

Each worker shows up in encore run and the dashboard as <service>.<worker>. Its log records are its service's, with an encore.worker.name attribute naming the worker. It counts as running once it has stayed up for 250 milliseconds; one that exits sooner failed to start, and its exit and last lines are shown. A worker that crashes stays down until the next rebuild, as a service does.

When the service's source changes, its workers are rebuilt with it: each is stopped, then started again, so a worker never runs twice.

Deploying workers

encore build docker packages a service's workers in its image, and the image runs them beside the service. A deployment can also run them on their own, in containers that run only workers. Its infra config's hosted_workers lists the workers a container runs, by <service>.<worker>, and encore infra config generate lists each worker where its service runs.

A worker that runs with its service scales with it. On compute that throttles CPU between requests, as Cloud Run does by default, a worker sharing a container with its service needs CPU allocated all the time.