Add janus_db_migrator container (#2177)

Janus now builds and ships a container image that bundles `sqlx` and the
current Janus version's SQL migration scripts. This makes it easier for
deployments to apply SQL migrations, as they no longer need to build a
`sqlx` image themselves and (more importantly) figure out how to provide
the appropriate set of migration scripts to whatever is applying the
migrations.

While updating the READMEs, I also fixed a few Markdown hyperlinks to
make some long lines less long.

.github/workflows/ci-build.yml

@@ -151,6 +151,7 @@ jobs:
     - run: docker run --rm janus_aggregation_job_driver --help
     - run: docker run --rm janus_collection_job_driver --help
     - run: docker run --rm janus_cli --help
+    - run: docker run --rm janus_db_migrator --help
 
   janus_interop_docker:
     runs-on: ubuntu-latest

.github/workflows/push-docker-images-release.yml

@@ -59,6 +59,7 @@ jobs:
     - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_aggregation_job_driver:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_collection_job_driver:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_cli:${{ steps.get_version.outputs.VERSION }}
+    - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_db_migrator:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_interop_client:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_interop_aggregator:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/janus-artifacts/janus/janus_interop_collector:${{ steps.get_version.outputs.VERSION }}
@@ -83,6 +84,7 @@ jobs:
     - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_aggregation_job_driver:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_collection_job_driver:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_cli:${{ steps.get_version.outputs.VERSION }}
+    - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_db_migrator:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_interop_client:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_interop_aggregator:${{ steps.get_version.outputs.VERSION }}
     - run: docker push us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_interop_collector:${{ steps.get_version.outputs.VERSION }}

Dockerfile.sqlx

@@ -0,0 +1,15 @@
+FROM rust:1.73.0-alpine as builder
+ARG SQLX_VERSION
+RUN apk add libc-dev
+RUN cargo install sqlx-cli \
+    --version ${SQLX_VERSION} \
+    --no-default-features --features rustls,postgres
+
+FROM alpine:3.18.4
+ARG SQLX_VERSION=unknown
+ARG GIT_REVISION=unknown
+LABEL revision ${GIT_REVISION}
+LABEL sqlx_version ${SQLX_VERSION}
+COPY --from=builder /usr/local/cargo/bin/sqlx /sqlx
+COPY db /migrations
+ENTRYPOINT ["/sqlx"]

README.md

@@ -22,21 +22,28 @@ branch.
 
 | Git branch | Draft version | Conforms to protocol? | Status |
 | ---------- | ------------- | --------------------- | ------ |
-| `release/0.1` | [`draft-ietf-ppm-dap-01`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/01/) | Yes | Unmaintained as of December 7, 2022 |
-| `release/0.2` | [`draft-ietf-ppm-dap-02`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/02/) | Yes | Unmaintained as of July 13, 2023 |
-| `release/0.3` | [`draft-ietf-ppm-dap-03`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/03/) | Yes | Unmaintained as of February 6, 2023 |
-| `release/0.4` | [`draft-ietf-ppm-dap-04`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/04/) | Yes | Unmaintained as of May 24, 2023 |
-| `release/0.subscriber-01` | [`draft-ietf-ppm-dap-02`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/02/) plus extensions | No | Supported |
-| `release/0.5` | [`draft-ietf-ppm-dap-04`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/04/) | Yes | Supported |
-| `main` | [`draft-ietf-ppm-dap-07`](https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/07/) | Yes | Supported |
+| `release/0.1` | [`draft-ietf-ppm-dap-01`][dap-01] | Yes | Unmaintained as of December 7, 2022 |
+| `release/0.2` | [`draft-ietf-ppm-dap-02`][dap-02] | Yes | Unmaintained as of July 13, 2023 |
+| `release/0.3` | [`draft-ietf-ppm-dap-03`][dap-03] | Yes | Unmaintained as of February 6, 2023 |
+| `release/0.4` | [`draft-ietf-ppm-dap-04`][dap-04] | Yes | Unmaintained as of May 24, 2023 |
+| `release/0.subscriber-01` | [`draft-ietf-ppm-dap-02`][dap-02] plus extensions | No | Supported |
+| `release/0.5` | [`draft-ietf-ppm-dap-04`][dap-04] | Yes | Supported |
+| `main` | [`draft-ietf-ppm-dap-07`][dap-07] | Yes | Supported |
 
 Note that no version of Janus supports `draft-ietf-ppm-dap-05` or `-06`. Draft
 05 was skipped because there were flaws in its usage of the new ping-pong
 topology introduced in `draft-irtf-cfrg-vdaf-06`. Draft 6 fixed those issues,
 but was skipped because it was published from the wrong commit of
-[`draft-ietf-ppm-dap`](https://github.com/ietf-wg-ppm/draft-ietf-ppm-dap) and so
-contains a couple of bugs. `draft-ietf-ppm-dap-07` is effectively identical to
-draft 6, but with those bugs fixed.
+[`draft-ietf-ppm-dap`][dap-gh] and so contains a couple of bugs.
+`draft-ietf-ppm-dap-07` is effectively identical to draft 6, but with those bugs
+fixed.
+
+[dap-01]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/01/
+[dap-02]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/02/
+[dap-03]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/03/
+[dap-04]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/04/
+[dap-07]: https://datatracker.ietf.org/doc/draft-ietf-ppm-dap/07/
+[dap-gh]: https://github.com/ietf-wg-ppm/draft-ietf-ppm-dap
 
 ## Building
 
@@ -51,7 +58,7 @@ subtle incompatibilities between the two that will cause tests to fail.
 To build container images, run `docker buildx bake --load`. This will produce
 images tagged `janus_aggregator`, `janus_aggregation_job_creator`,
 `janus_aggregation_job_driver`, `janus_collection_job_driver`, `janus_cli`,
-`janus_interop_client`, `janus_interop_aggregator`, and
+`janus_db_migrator`, `janus_interop_client`, `janus_interop_aggregator`, and
 `janus_interop_collector` by default.
 
 Pre-built container images are available at
@@ -67,22 +74,23 @@ preceding minor versions.
 Tests require that [`docker`](https://www.docker.com) and
 [`kind`](https://kind.sigs.k8s.io) be installed on the machine running the tests
 and in the `PATH` of the test-runner's environment. The `docker` daemon must be
-running. CI tests currently use [`kind`
-0.17.0](https://github.com/kubernetes-sigs/kind/releases/tag/v0.17.0) and the
+running. CI tests currently use [`kind` 0.17.0][kind-release] and the
 corresponding Kubernetes 1.24 node image
 (kindest/node:v1.24.7@sha256:577c630ce8e509131eab1aea12c022190978dd2f745aac5eb1fe65c0807eb315).
 Using the same versions for local development is recommended.
 
 To run Janus tests, execute `cargo test`.
 
+[kind-release]: https://github.com/kubernetes-sigs/kind/releases/tag/v0.17.0
+
 ### inotify limits
 
 If you experience issues with tests using Kind on Linux, you may need to [adjust
-inotify
-sysctls](https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files).
-Both systemd and Kubernetes inside each Kind node make use of inotify. When
-combined with other services and desktop applications, they may exhaust per-user
-limits.
+inotify sysctls][inotify]. Both systemd and Kubernetes inside each Kind node
+make use of inotify. When combined with other services and desktop applications,
+they may exhaust per-user limits.
+
+[inotify]: https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files
 
 ## Deploying Janus
 

docker-bake.hcl

@@ -6,6 +6,10 @@ variable "VERSION" {
   default = "latest"
 }
 
+variable "SQLX_VERSION" {
+  default = "0.7.2"
+}
+
 variable "GITHUB_REF_NAME" {}
 
 variable "GITHUB_BASE_REF" {}
@@ -21,6 +25,7 @@ group "janus" {
     "janus_aggregation_job_driver",
     "janus_collection_job_driver",
     "janus_cli",
+    "janus_db_migrator",
   ]
 }
 
@@ -43,6 +48,7 @@ group "janus_release" {
     "janus_aggregation_job_driver_release",
     "janus_collection_job_driver_release",
     "janus_cli_release",
+    "janus_db_migrator_release",
   ]
 }
 
@@ -167,6 +173,28 @@ target "janus_cli_release" {
   ]
 }
 
+target "janus_db_migrator" {
+  args = {
+    GIT_REVISION = "${GIT_REVISION}"
+    SQLX_VERSION = "${SQLX_VERSION}"
+  }
+  dockerfile = "Dockerfile.sqlx"
+  cache-from = [
+    "type=gha,scope=main-janus",
+    "type=gha,scope=${GITHUB_BASE_REF}-janus",
+    "type=gha,scope=${GITHUB_REF_NAME}-janus",
+  ]
+  tags = ["janus_db_migrator:${VERSION}"]
+}
+
+target "janus_db_migrator_release" {
+  inherits = ["janus_db_migrator"]
+  tags = [
+    "us-west2-docker.pkg.dev/janus-artifacts/janus/janus_db_migrator:${VERSION}",
+    "us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_db_migrator:${VERSION}",
+  ]
+}
+
 target "janus_interop_client" {
   args = {
     BINARY = "janus_interop_client"

docs/DEPLOYING.md

@@ -119,6 +119,10 @@ using the `--source` argument to point to `janus/db` and providing database
 connection information in any of the ways supported by `sqlx` (see its
 documentation).
 
+Note that migrations _must_ be applied using `sqlx` as Janus will fail to start
+if it cannot locate a `_sqlx_migrations` table to determine whether it supports
+the current schema version.
+
 For simple or experimental deployments where the complexity of `sqlx` is not
 warranted, it is possible to create a single schema file by concatenating the
 `.up.sql` scripts, in order, and applying this schema to the database. When
@@ -127,7 +131,20 @@ configuration file. Note that such deployments will not easily be able to
 migrate to later versions of the schema, so this technique is likely not
 appropriate for deployments which need to retain data across deployments.
 
+Janus also provides a container image called `janus_db_migrator` that makes it
+easier to apply SQL migrations in many deployment environments.
+`janus_db_migrator` contains the `sqlx` binary as well as the migration scripts
+from the corresponding Janus version in the `/migrations` directory inside the
+container so that deployments do not have to fetch the migrations from somewhere
+else. The image's entrypoint simply invokes `sqlx` so that deployments can pass
+the appropriate database configuration and subcommands. See [`sqlx`][sqlx-cli]'s
+documentation for more on working with that tool.
+
+Pre-built `janus_db_migrator` images are available at
+[us-west2-docker.pkg.dev/divviup-artifacts-public/janus/janus_db_migrator][migrator-images].
+
 [sqlx-cli]: https://crates.io/crates/sqlx-cli
+[migrator-images]: https://us-west2-docker.pkg.dev/divviup-artifacts-public/janus
 
 ### Datastore Keys