Using SQL databases
Provisioning, migrating, querying
Encore treats SQL databases as logical resources and natively supports PostgreSQL databases.
Creating a database
To create a database, import encore.dev/storage/sqldb
and call sqldb.NewDatabase
, assigning the result to a package-level variable.
Databases must be created from within an Encore service.
For example:
todo/db.gotodo/migrations/1_create_table.up.sqlpackage todo
// Create the todo database and assign it to the "tododb" variable
var tododb = sqldb.NewDatabase("todo", sqldb.DatabaseConfig{
Migrations: "./migrations",
})
// Then, query the database using db.QueryRow, db.Exec, etc.
As seen above, the sqldb.DatabaseConfig
specifies the directory containing the database migration files, which is how you define the database schema.
See the Defining the database schema section below for more details.
With this code in place Encore will automatically create the database using Docker when starting encore run
(locally).
For cloud environments, Encore automatically injects the appropriate configuration to authenticate and connect to the database, so once the application starts up the database is ready to be used.
Related example
$ encore app create --example=sql-database
Defining the database schema
Database schemas are defined by creating migration files in a directory named migrations
within an Encore service package. Each migration file is named <number>_<name>.up.sql
, where
<number>
is a sequence number for ordering the migrations and <name>
is a
descriptive name of what the migration does.
On disk it might look like this:
/my-app
├── encore.app // ... and other top-level project files
│
└── todo // todo service (a Go package)
├── migrations // todo service db migrations (directory)
│ ├── 1_create_table.up.sql // todo service db migration
│ └── 2_add_field.up.sql // todo service db migration
├── todo.go // todo service code
└── todo_test.go // tests for todo service
Each migration runs in order and expresses the change in the database schema from the previous migration.
The file name format is important. Migration files must be sequentially named, starting with 1_
and counting up for each migration.
Each file name must also end with .up.sql
.
The first migration usually defines the initial table structure. For example,
a todo
service might start out by creating todo/migrations/1_create_table.up.sql
with
the following contents:
CREATE TABLE todo_item (
id BIGSERIAL PRIMARY KEY,
title TEXT NOT NULL,
done BOOLEAN NOT NULL DEFAULT false
);
Inserting data into databases
Once you have created the database using var mydb = sqldb.NewDatabase(...)
you can start inserting data into the database
by calling methods on the mydb
variable.
The interface is similar to that of the Go standard library's database/sql
package.
Learn more in the package docs.
One way of inserting data is with a helper function that uses the package function sqldb.Exec
.
For example, to insert a single todo item using the example schema above, we can use the following helper function insert
:
todo/insert.gotodo/db.gotodo/migrations/1_create_table.up.sql// insert inserts a todo item into the database.
func insert(ctx context.Context, id, title string, done bool) error {
_, err := tododb.Exec(ctx, `
INSERT INTO todo_item (id, title, done)
VALUES ($1, $2, $3)
`, id, title, done)
return err
}
Querying databases
To query a database in your application, you similarly need to import encore.dev/storage/sqldb
in your service package or sub-package.
For example, to read a single todo item in the example schema above, we can use sqldb.QueryRow
:
var item struct {
ID int64
Title string
Done bool
}
err := tododb.QueryRow(ctx, `
SELECT id, title, done
FROM todo_item
LIMIT 1
`).Scan(&item.ID, &item.Title, &item.Done)
If QueryRow
does not find a matching row, it reports an error that can be checked against
by importing the standard library errors
package and calling errors.Is(err, sqldb.ErrNoRows)
.
Learn more in the package docs.
Provisioning databases
Encore automatically provisions databases to match what your application requires. When you define a database, Encore will provision the database at your next deployment.
Encore provisions databases in an appropriate way depending on the environment. When running locally, Encore creates a database cluster using Docker. In the cloud, it depends on the environment type:
- In
production
environments, the database is provisioned through the Managed SQL Database service offered by the chosen cloud provider. - In
development
environments, the database is provisioned as a Kubernetes deployment with a persistent disk attached.
See exactly what is provisioned for each cloud provider, and each environment type, in the infrastructure documentation.
Connecting to databases
It's often useful to be able to connect to the database from outside the backend application. For example for scripts, ad-hoc querying, or dumping data for analysis.
Using the Encore CLI
Encore's CLI comes with built-in support for connecting to databases:
encore db shell <database-name> [--env=<name>]
opens a psql shell to the database named<database-name>
in the given environment. Leaving out--env
defaults to the local development environment.encore db conn-uri <database-name> [--env=<name>]
outputs a connection string for the database named<database-name>
. When specifying a cloud environment, the connection string is temporary. Leaving out--env
defaults to the local development environment.encore db proxy [--env=<name>]
sets up a local proxy that forwards any incoming connection to the databases in the specified environment. Leaving out--env
defaults to the local development environment.
See encore help db
for more information on database management commands.
Using database user credentials
For cloud environments you can view database user credentials (created by Encore when provisioning databases) via the Cloud Dashboard:
- Open your app in the Cloud Dashboard, navigate to the Infrastructure page for the appropriate environment, and locate the
USERS
section within the relevant Database Cluster.
Handling migration errors
When Encore applies database migrations, there's always a possibility the migrations don't apply cleanly.
This can happen for many reasons:
- There's a problem with the SQL syntax in the migration
- You tried to add a
UNIQUE
constraint but the values in the table aren't actually unique - The existing database schema didn't look like you thought it did, so the database object you tried to change doesn't actually exist
- ... and so on
If that happens, Encore rolls back the migration. If it happens during a cloud deployment, the deployment is aborted.
Once you fix the problem, re-run encore run
(locally) or push the updated code (in the cloud) to try again.
Encore tracks which migrations have been applied in the schema_migrations
table:
database=# \d schema_migrations
Table "public.schema_migrations"
Column | Type | Collation | Nullable | Default
---------+---------+-----------+----------+---------
version | bigint | | not null |
dirty | boolean | | not null |
Indexes:
"schema_migrations_pkey" PRIMARY KEY, btree (version)
The version
column tracks which migration was last applied. If you wish to skip a migration or re-run a migration,
change the value in this column. For example, to re-run the last migration, run UPDATE schema_migrations SET version = version - 1;
.
Note that Encore does not use the dirty
flag by default.