gerodimos.dev

Architecture area

Database schemas

Code can be rolled back; data can't. So a service's database schema is released on its own, through its own pull request, and only ever moves forward. Deploying code never changes the database, and rolling code back never undoes a migration.

Watch it end to end: Onboard a Database, 3 min →

Why the schema has its own lifecycle

The common setup runs migrations as part of a deploy: an init container, a Helm hook or a pre-sync job. That ties two kinds of change to one lifecycle, and they don't behave alike. Rolling back an image is cheap and exact: the old pods come back. Rolling back a migration usually isn't: a dropped column doesn't come back with the old code.

Code: reversible

Dev follows every green build; prod moves by pull request, and back to any earlier version the same way. See GitOps delivery.

Schema: forward only

Applied when someone chooses to, through its own pull request. A mistake is fixed by the next migration, never by undoing one.

  • Code rollbacks leave the database alone. Rolling back never touches the schema, and because changes go in additive first, the previous version still runs against the newer one.
  • Expand, then contract. Additive changes can land before the code that uses them; destructive ones wait until no running version needs what they remove. It's the standard way to change a schema without downtime, and here it's simply how releases work.
  • The riskiest change gets its own review. A schema change has its own pull request, its own status and its own failure, instead of riding along inside a deploy.
  • Migrations run once, outside the service. Atlas applies each one, so replicas never race to migrate on startup and the service needs no migration logic.

The cost: deploying code and schema together

Nothing orders a schema apply and a code deploy merged at the same moment. New pods can start before the migration lands and fail the queries that need it, until Atlas finishes. Old pods are unaffected: rollouts never drop below capacity and changes are additive. Today the rule is apply the schema, wait for Applied, then deploy, and the portal warns when the code running in an environment ships a migration its database doesn't have. In dev, auto-deploy can ship such code on its own.

The fix: schema-aware readiness coming

The shape

Package row: migrations in the service repository are checked by the schema workflow, which replays them on Postgres 16 and publishes a schema package per commit to GHCR. Release row: Backstage's Apply database schema opens a pull request with a DatabaseSchema; schema-operator on the management cluster checks the package exists and commits a pointer file to application-repositories. Apply row: Argo CD deploys the package to the workload cluster as an ExternalSecret, a ConfigMap and an AtlasMigration; Atlas Operator checks them on a throwaway Postgres and applies the new ones to the database. PACKAGE · every commit that changes migrations/ RELEASE · per environment, by pull request APPLY · in the workload cluster Service repository migrations/*.sql atlas.sum Schema workflow atlas.sum up to date replayed on Postgres 16 Schema package ghcr.io/…/payments/schema 0.0.0-g<commit> Backstage Apply database schema opens a pull request DatabaseSchema componentRef · version on the management cluster schema-operator finds the package, the database and its published location Pointer file components/payments/ schema/dev.yaml exists? Argo CD one Application per service and environment The package, deployed ExternalSecret · ConfigMap AtlasMigration Atlas Operator checks on a throwaway Postgres, then applies only the new files Database records every applied version Argo CD reads the pointer and pulls the package
Scroll sideways to see all of it. The highlighted arrow is the only way a change reaches the database. The dashed arrow is a check: nothing is written until the package exists.

How it works

  1. Migrations live with the service

    Hand-written SQL files in the repository's migrations/, with an atlas.sum that fixes their content and order. Services scaffolded from golang-service 0.13.0 onwards start with this in place. See Golden-path scaffolding.

  2. Every change is checked, then packaged

    The schema workflow runs only when migrations/ changes. On a pull request and on main it fails if atlas.sum is stale and replays every migration, in order, on an empty Postgres 16. On main it then publishes the platform's schema chart with those migrations inside, versioned by the commit, next to the service's image. Every green run is a version that can be applied.

  3. Applying is a pull request

    Backstage's Apply database schema form takes a service, an environment and a version. It shows the migration files that version adds; the version already requested and older ones can't be picked. Its pull request adds one small file.

    kind: DatabaseSchema
metadata: {name: payments, namespace: dev}
spec:
  componentRef: {name: payments}
  version: "54f3f80…"   # the commit whose migrations/ to apply

  4. schema-operator writes a pointer

    On the management cluster it finds the service's repository through its Component, the environment's database and where that database published its connection details, and checks that the package exists. Then it commits a small file to application-repositories: which package, which version, which location. It never touches a workload cluster.

  5. Atlas applies it, inside the cluster

    An Argo CD ApplicationSet turns each pointer into one Application, which deploys the package to the service's namespace: an ExternalSecret for the database's connection details, a ConfigMap with the migrations and an AtlasMigration, in that order. Atlas Operator checks the migrations on a throwaway Postgres, applies only the files the database hasn't run, and records them there. Argo CD reports the Application Healthy only once Atlas reports Applied, and Degraded if a migration fails.

  6. The portal shows where each database stands

    A database's page lists every check a version passes (packaged, requested, applied, the newest one, matching the code) with the states each can be in, and offers the Apply action to the owner. The deployment card warns when the code running in an environment ships a migration its database doesn't have. See Developer experience.

Design choices

Versioned SQL, not a desired state

Reviewers see exactly the statements that will run, in order, rather than a target schema and a diff computed later.

Trade-off: migrations are written by hand; nothing generates them from a schema definition yet.

Forward only

There are no down migrations. Atlas refuses to migrate a database backwards, and the portal only offers newer versions. A bad migration is fixed by the next one.

Trade-off: undoing a change means writing its inverse as a new migration.

A package per commit, a pointer in GitOps

The SQL stays in the service's repository and its immutable package. The GitOps repository records which version should run where, and the location of the credentials, never the credentials themselves.

One platform chart

A service owns only its migrations/. How they're applied (the Secret, the ConfigMap, the AtlasMigration) is a versioned platform chart that CI packs around them.

Trade-off: the chart version is pinned in each service's workflow, so a newer chart reaches a service when that pin is bumped.

Independent of the binding

The schema reads the database's published location itself, the way a binding does, without needing one. A schema can be applied before any service binds the database. See Service bindings.

Its own Application

Each service's schema has its own Argo CD Application next to the service's, so the two sync and fail independently, a failed migration shows on its own, and rolling the code back leaves the schema where it is.

Checked twice before it counts

CI replays every migration on an empty Postgres before publishing; Atlas checks them again on a throwaway one before touching the real database. A broken file fails twice before it can fail there.

Trade-off: the throwaway database is pinned to Postgres 16 to match the platform's databases, by hand.

Not built yet

  • Schema-aware readiness coming. The service will know the newest migration it ships with and stay not-ready until its database has it. A rollout then waits for the schema while the old pods keep serving, so code and schema can be merged together in any order, and a rollback always passes.
  • Following main in dev. Applying each new schema version automatically, the way auto-deploy follows each green build, so dev needs no manual apply once readiness gates the rollout.
  • Destructive-change checks. CI proves migrations apply; nothing flags a dropped column or table yet, so contract steps rely on review.
  • Engine version from the database. The throwaway Postgres version is set by hand rather than read from the database it stands in for.
  • A separate migration user. Migrations and the service use the same database credentials today.

Code

schema-operator

The DatabaseSchema API and its controller.

platform-scaffolds

The migrations/ layout and the schema workflow.

helm-charts

The platform's schema chart.

backstage

The Apply database schema form and the database's checks.