Merge pull request #88 from lsst-sqre/tickets/DM-48385

DM-48385: Add Alembic integration

.pre-commit-config.yaml

@@ -7,7 +7,7 @@ repos:
       - id: check-toml
 
   - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: v0.8.6
+    rev: v0.9.1
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

CHANGELOG.md

@@ -8,6 +8,22 @@ Collect fragments into this file with: scriv collect --version X.Y.Z
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-0.14.0'></a>
+
+## 0.14.0 (2025-01-13)
+
+### Other changes
+
+- Times Square now uses Alembic to manage database schema versioning and migrations.
+
+- Begin SQLAlchemy 2 adoption with the new `DeclarativeBase`, `mapped_column`, and the `Mapped` type.
+
+- Update the `make update` command in the Makefile to use the `--universal` flag with `uv pip compile`.
+
+- Fix type checking issues for Pydantic fields to use empty lists as the default, rather than using a default factory.
+
+- Explicitly set `asyncio_default_fixture_loop_scope` to `function`. An explicit setting is now required by pytest-asyncio.
+
 <a id='changelog-0.13.0'></a>
 
 ## 0.13.0 (2024-09-12)

Dockerfile

@@ -14,7 +14,7 @@
 #   - Runs a non-root user.
 #   - Sets up the entrypoint and port.
 
-FROM python:3.12.6-slim-bookworm as base-image
+FROM python:3.12.6-slim-bookworm AS base-image
 
 # Update system packages
 COPY scripts/install-base-packages.sh .
@@ -57,9 +57,20 @@ COPY --from=install-image /opt/venv /opt/venv
 
 COPY scripts/start-api.sh /start-api.sh
 
+# Copy the Alembic configuration and migrations, and set that path as the
+# working directory so that Alembic can be run with a simple entry command
+# and no extra configuration.
+COPY --from=install-image /workdir/alembic.ini /app/alembic.ini
+COPY --from=install-image /workdir/alembic /app/alembic
+WORKDIR /app
+
 # Make sure we use the virtualenv
 ENV PATH="/opt/venv/bin:$PATH"
 
+# Set environment variable for Alembic config; other variables are set
+# via Kubernetes.
+ENV TS_ALEMBIC_CONFIG_PATH="/app/alembic.ini"
+
 # Switch to the non-root user.
 USER appuser
 

README.md

@@ -8,9 +8,20 @@ Excellent applications for Times Square include:
 - Quick-look data previewing
 - Reports that incorporate live data sources
 
+## Deployments of Times Square
+
+| Link                                                                  | User guide                                                                    | Purpose                     |
+| --------------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------- |
+| [USDF RSP (dev)](https://usdf-rsp-dev.slac.stanford.edu/times-square) | [Documentation](https://rsp.lsst.io/v/usdfdev/guides/times-square/index.html) | For Rubin Observatory staff |
+| [IDF (dev)](https://data-dev.lsst.cloud/times-square)                 | [Documentation](https://rsp.lsst.io/v/idfdev/guides/times-square/index.html)  | For SQuaRE developers       |
+
+## Design and development
+
 The design and architecture of Times Square is described in [SQR-062: The Times Square service for publishing parameterized Jupyter Notebooks in the Rubin Science platform](https://sqr-062.lsst.io).
 Times Square uses [Noteburst](https://noteburst.lsst.io) ([GitHub](https://github.com/lsst-sqre/noteburst), [SQR-065](https://sqr-065.lsst.io)) to execute Jupyter Notebooks in Nublado (JupyterLab) instances, thereby mechanizing the RSP's notebook aspect.
 
 This Times Square API service is developed at https://github.com/lsst-sqre/times-square.
-It's user interface is part of [Squareone](https://github.com/lsst-sqre/squareone).
+Its user interface is part of [Squareone](https://github.com/lsst-sqre/squareone).
 Times Square is deployed with [Phalanx](https://phalanx.lsst.io/applications/times-square/index.html).
+
+REST API and developer documentation is at [times-square.lsst.io](https://times-square.lsst.io) and deployment documentation is [available at phalanx.lsst.io](https://phalanx.lsst.io/applications/times-square/index.html).

alembic.ini

@@ -0,0 +1,15 @@
+[alembic]
+script_location = %(here)s/alembic
+file_template = %%(year)d%%(month).2d%%(day).2d_%%(hour).2d%%(minute).2d_%%(rev)s_%%(slug)s
+prepend_sys_path = .
+timezone = UTC
+version_path_separator = os
+
+[post_write_hooks]
+hooks = ruff ruff_format
+ruff.type = exec
+ruff.executable = ruff
+ruff.options = check --fix REVISION_SCRIPT_FILENAME
+ruff_format.type = exec
+ruff_format.executable = ruff
+ruff_format.options = format REVISION_SCRIPT_FILENAME

alembic/README

@@ -0,0 +1 @@
+Generic single-database configuration.

alembic/env.py

@@ -0,0 +1,22 @@
+"""Alembic migration environment."""
+
+from alembic import context
+from safir.database import run_migrations_offline, run_migrations_online
+from safir.logging import configure_alembic_logging, configure_logging
+
+from timessquare.config import config
+from timessquare.dbschema import Base
+
+# Configure structlog.
+configure_logging(name="timessquare", log_level=config.log_level)
+configure_alembic_logging()
+
+# Run the migrations.
+if context.is_offline_mode():
+    run_migrations_offline(Base.metadata, config.database_url)
+else:
+    run_migrations_online(
+        Base.metadata,
+        config.database_url,
+        config.database_password,
+    )

alembic/script.py.mako

@@ -0,0 +1,26 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: str | None = ${repr(down_revision)}
+branch_labels: str | Sequence[str] | None = ${repr(branch_labels)}
+depends_on: str | Sequence[str] | None = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    ${downgrades if downgrades else "pass"}

alembic/versions/20250110_2121_487195933fee_initial_schema.py

@@ -0,0 +1,69 @@
+"""Initial schema.
+
+Revision ID: 487195933fee
+Revises:
+Create Date: 2025-01-10 21:21:26.706763+00:00
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = "487195933fee"
+down_revision: str | None = None
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "pages",
+        sa.Column("name", sa.Unicode(length=32), nullable=False),
+        sa.Column("ipynb", sa.UnicodeText(), nullable=False),
+        sa.Column("parameters", sa.JSON(), nullable=False),
+        sa.Column("title", sa.UnicodeText(), nullable=False),
+        sa.Column("date_added", sa.DateTime(), nullable=False),
+        sa.Column("authors", sa.JSON(), nullable=False),
+        sa.Column("tags", sa.JSON(), nullable=False),
+        sa.Column("uploader_username", sa.Unicode(length=64), nullable=True),
+        sa.Column("date_deleted", sa.DateTime(), nullable=True),
+        sa.Column("description", sa.UnicodeText(), nullable=True),
+        sa.Column("cache_ttl", sa.Integer(), nullable=True),
+        sa.Column("github_owner", sa.Unicode(length=255), nullable=True),
+        sa.Column("github_repo", sa.Unicode(length=255), nullable=True),
+        sa.Column("github_commit", sa.Unicode(length=40), nullable=True),
+        sa.Column(
+            "repository_path_prefix", sa.Unicode(length=2048), nullable=True
+        ),
+        sa.Column(
+            "repository_display_path_prefix",
+            sa.Unicode(length=2048),
+            nullable=True,
+        ),
+        sa.Column(
+            "repository_path_stem", sa.Unicode(length=255), nullable=True
+        ),
+        sa.Column(
+            "repository_source_extension",
+            sa.Unicode(length=255),
+            nullable=True,
+        ),
+        sa.Column(
+            "repository_sidecar_extension",
+            sa.Unicode(length=255),
+            nullable=True,
+        ),
+        sa.Column(
+            "repository_source_sha", sa.Unicode(length=40), nullable=True
+        ),
+        sa.Column(
+            "repository_sidecar_sha", sa.Unicode(length=40), nullable=True
+        ),
+        sa.PrimaryKeyConstraint("name"),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("pages")

docs/_rst_epilog.rst

@@ -4,4 +4,5 @@
 .. _scriv: https://scriv.readthedocs.io/en/latest/
 .. _semver: https://semver.org
 .. _tox: https://tox.readthedocs.io/en/latest/
+.. _Alembic: https://alembic.sqlalchemy.org/en/latest/
 

docs/dev/development.rst

@@ -24,12 +24,14 @@ Setting up a local development environment
 Times Square is a Python project that should be developed within a virtual environment.
 
 If you already have a Python virtual environment set up in your shell, you can use the :command:`make init` command to install Times Square and its development dependencies into it.
+This complete procedure shows how to use uv to create a virtual environment and install Times Square into it:
 
 .. code-block:: sh
 
     git clone https://github.com/lsst-sqre/times-square.git
     cd times-square
-    pip install tox
+    uv venv
+    source .venv/bin/activate
     make init
 
 .. _pre-commit-hooks:
@@ -39,12 +41,7 @@ Pre-commit
 
 The pre-commit hooks, which are automatically installed by running the :command:`make init` command on :ref:`set up <dev-environment>`, ensure that files are valid and properly formatted.
 Some pre-commit hooks automatically reformat code:
-
-``ruff``
-    Sorts Python imports and automatically fixes some common Python issues.
-
-``black``
-    Automatically formats Python code.
+The ``ruff`` hook automatically fixes some common Python issues and sorts Python imports.
 
 When these hooks fail, your Git commit will be aborted.
 To proceed, stage the new modifications and proceed with your Git commit.
@@ -68,6 +65,14 @@ To see a listing of specific tox sessions, run:
 
 Times Square requires Docker to run its tests.
 
+Database migrations
+===================
+
+Times Square uses Alembic_ for database migrations.
+If your work involves changing the database schema (in :file:`/src/timessquare/dbschema`) you will need to prepare an Alembic migration in the same PR.
+This process is outlined in the `Safir documentation <https://safir.lsst.io/user-guide/database/schema.html#testing-database-migrations>`__.
+Note that in Times Square the :file:`docker-compose.yaml` is hosted in the root of the repository rather than in the :file:`alembic` directory.
+
 Building documentation
 ======================
 
@@ -98,7 +103,7 @@ When preparing a pull request, run
 
 .. code-block:: sh
 
-   scrive create
+   scriv create
 
 This will create a change log fragment in :file:`changelog.d`.
 Edit that fragment, removing the sections that do not apply and adding entries for your pull request.

pyproject.toml

@@ -110,6 +110,13 @@ extend-exclude = [
 ]
 
 [tool.ruff.lint.per-file-ignores]
+"alembic/env.py" = [
+    "INP001",  # Alembic config isn't a package
+]
+"alembic/versions/**" = [
+    "INP001",  # Alembic config isn't a package
+    "D103",    # Alembic migrations don't need docstrings
+]
 "src/timessquare/config.py" = [
     "FBT001",  # Pydantic validators take boolean arguments
 ]
@@ -123,8 +130,10 @@ extend-exclude = [
     "S106",    # tests are allowed to hard-code dummy passwords
     "SLF001",  # tests are allowed to access private members
     "T201",    # tests are allowed to use print
+    "ASYNC221",  # async tests are allowed to use sync functions
 ]
 
 [tool.ruff.lint.isort]
 known-first-party = ["timessquare", "tests"]
+known-third-party = ["alembic"]
 split-on-trailing-comma = false