# Graceful shutdown

> Drain work and release resources when a service stops

When a service is stopped, for example during a deploy or when `encore run` restarts it, it receives `SIGTERM`. Encore then shuts down gracefully: it stops taking new Pub/Sub messages, waits for message handlers in progress, closes database, Redis and object storage connections, and flushes metrics and traces.

Most apps need no code for this. Read on if your app runs its own background work that uses Encore resources, or if you want to control when Encore shuts down.

## Default behavior

Importing `encore` sets up shutdown for you:

- It registers an exit hook that shuts Encore down when the interpreter exits.
- It installs a `SIGTERM` handler that exits the interpreter, with exit code `0` if shutdown completed within its deadline and `1` otherwise. It does this only when the process has no `SIGTERM` handler yet at import time, and not under `encore test`, where the test runner handles the signal.

This fits servers like Uvicorn, which handle `SIGTERM` themselves: Uvicorn stops accepting connections and drains requests in progress, then restores the previous signal handler and raises the signal again. That hands over to Encore, which drains Pub/Sub and cleans up after your HTTP server has finished. A FastAPI app run with `uvicorn.run(app, ...)` shuts down gracefully without any shutdown code.

If your server handles `SIGTERM` without raising it again, Encore still shuts down from its exit hook as the interpreter exits.

## How shutdown proceeds

Shutdown moves through these phases, which `shutdown.phase()` returns as a `ShutdownPhase`:

| Phase | Meaning |
| --- | --- |
| `NOT_STARTED` | Shutdown hasn't started. |
| `STARTED` | Pub/Sub has stopped taking new messages. Encore waits for message handlers in progress, and in [cooperative mode](#cooperative-shutdown) for your application to signal that it has drained. Your HTTP server is still yours to stop. |
| `CLEANUP` | Encore closes SQL, Redis and object storage connections, then exports the final metrics and flushes traces. |
| `COMPLETED` | Cleanup has finished, whether it succeeded or not. |

Once cleanup has run, Encore's resources can't be used anymore, even through objects you still hold. New operations fail, interrupted object downloads raise an error rather than returning partial data, and Redis subscriptions end. Clients you create yourself from a resource's connection string are yours to close.

## Shutting down from your own code

To shut Encore down at a point you choose, call `shutdown.begin()` and await `shutdown.completed()`. For example, in a FastAPI lifespan handler, the code after `yield` runs once Uvicorn has drained HTTP requests:

```python
-- main.py --
import logging
from contextlib import asynccontextmanager

import encore
from encore import shutdown
from fastapi import FastAPI

log = logging.getLogger(__name__)


@asynccontextmanager
async def lifespan(app: FastAPI):
    await encore.start()
    yield
    # HTTP requests have drained: shut Encore down and wait for it.
    shutdown.begin()
    result = await shutdown.completed()
    log.info("encore shut down", extra={"result": result.value})


app = FastAPI(lifespan=lifespan)
```

`begin()` returns immediately, and all callers share one shutdown: repeated calls don't restart it or reset its deadline. When the interpreter exits afterwards, Encore's own hooks see that shutdown has already completed.

## Cooperative shutdown

Use cooperative shutdown when work your application started, such as background tasks, may still be using Encore resources when shutdown begins. Encore then stops Pub/Sub right away, but doesn't close resources until you call `mark_application_drained()` or the [deadline](#deadline-and-forced-cleanup) expires.

For example, a worker process with its own background tasks:

```python
-- worker.py --
import asyncio
import signal

import encore
from encore import shutdown


async def process_jobs(stop: asyncio.Event) -> None:
    while not stop.is_set():
        ...  # take a job and process it, using Encore resources


async def main() -> int:
    await encore.start()

    stop = asyncio.Event()
    asyncio.get_running_loop().add_signal_handler(signal.SIGTERM, stop.set)
    workers = [asyncio.create_task(process_jobs(stop)) for _ in range(4)]

    await stop.wait()
    shutdown.begin(cooperative=True)  # stop Pub/Sub, keep resources open

    await asyncio.gather(*workers)    # let the jobs in progress finish
    shutdown.mark_application_drained()

    result = await shutdown.completed()
    return result.code()


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))
```

Keep these rules in mind:

- Call `begin()` before awaiting `started()`, `cleanup()` or `completed()`. Awaiting them before `begin()` raises `ShutdownError`.
- The first call to `begin()` chooses the mode. Later calls don't change it.
- In cooperative mode, call `mark_application_drained()` before awaiting `cleanup()` or `completed()`. Otherwise Encore waits until the deadline expires.
- Keep the event loop running until `completed()` returns. Cancelling a wait doesn't cancel the shutdown itself.

`await shutdown.started()` and `await shutdown.cleanup()` wait until shutdown has reached or passed those phases, and return immediately if it already has. You don't need to await every phase in turn.

## Deadline and forced cleanup

Shutdown has a deadline, five seconds by default, which Encore sets for each deployment. It starts when shutdown begins and covers both draining and cleanup. When it expires:

1. The result becomes `TIMEOUT`, and stays that way even if cleanup later succeeds.
2. If Encore is still waiting for draining, it stops waiting and starts cleanup, even without `mark_application_drained()`.
3. Cleanup continues, and `completed()` returns once it has finished.

The deadline doesn't stop the process, and `completed()` can return after it. Your host or server decides when to terminate the process.

To start cleanup before the deadline without waiting for draining, call `force_cleanup()`:

```python
shutdown.force_cleanup()            # requires begin() first
await shutdown.cleanup()            # cleanup has started
result = await shutdown.completed() # cleanup has finished
```

`force_cleanup()` returns immediately and can be called repeatedly. Forcing cleanup doesn't make the result `TIMEOUT`.

Neither the deadline nor `force_cleanup()` stops handlers that are still running. They may continue and fail as the resources they use close, so drain your own work before `mark_application_drained()` where you can.

## Results and errors

`completed()` returns a `ShutdownResult`:

| Result | Meaning | `code()` |
| --- | --- | --- |
| `ShutdownResult.SUCCESS` | Cleanup finished before the deadline. | `0` |
| `ShutdownResult.TIMEOUT` | The deadline expired before cleanup finished. Cleanup has still finished when the result is returned, but a timeout doesn't mean every step succeeded. | `1` |

If cleanup fails, or shutdown's progress can't be observed, `completed()` raises `ShutdownError`. `code()` gives you a process exit code.