Merge pull request #3 from lsst-dm/tickets/DM-44260

DM-44260: Add migration script for schema 1.1.0

doc/lsst.dax.apdb_migrate/migrations/sql/schema.rst

@@ -13,7 +13,7 @@ The ``ApdbSql`` schema version stored in ``metadata`` will be ``0.1.0`` by defau
 
 An example of specifying different version number for ApdbSql::
 
-    $ apdb-migrate-sql upgrade --options apdb_sql_version=0.2.0 $APDB_URL schema_0.1.1
+    $ apdb-migrate-sql upgrade --options apdb_sql_version=0.2.0 -s SCHEMA_NAME $APDB_URL schema_0.1.1
 
 
 Upgrade from 0.1.1 to 1.0.0
@@ -36,3 +36,17 @@ An example of migrating APDB populated from LATISS data::
 
     $ setup -k obs_lsst
     $ apdb-migrate-sql upgrade -s SCHEMA_NAME --options butler-repo=/repo/main --options instrument=LATISS $APDB_URL schema_1.0.0
+
+
+Upgrade from 1.0.0 to 1.1.0
+===========================
+
+Migration script: `schema_1.1.0.py <https://github.com/lsst-dm/dax_apdb_migrate/blob/main/migrations/sql/schema/schema_1.1.0.py>`_
+
+This migration removes multiple BLOB columns from ``DiaObject`` table (and ``DiaObjectChunks`` if that is present).
+Column names are ``{b}_lcPeriodic`` and ``{b}_lcNonPeriodic`` where ``{b}`` is one of six band names (``ugrizy``).
+No additional parameters or packages are needed for this script.
+
+An example of migration::
+
+    $ apdb-migrate-sql upgrade -s SCHEMA_NAME $APDB_URL schema_1.1.0

doc/lsst.dax.apdb_migrate/typical-tasks.rst

@@ -23,6 +23,11 @@ SQL Backend
 
 Below a re examples of commands that are typically used with SQL backend.
 
+Note that our usual setup uses shared PostgreSQL database where each APDB instance is located in a separate schema.
+It is important to always use correct ``--schema/-s`` option providing correct schema name.
+SQLite databases do not use schemas and ``--schema/-s`` should not be given in that case.
+Examples below include ``-s SCHEMA_NAME`` for the commands that have that option.
+
 
 Revision stamping
 -----------------
@@ -33,15 +38,15 @@ Before executing any other migration commands the ``alembic_version`` table need
 
 To check the existence of the ``alembic_version`` table one can run the ``show-current`` command::
 
-    $ apdb-migrate-sql show-current $APDB_URL
+    $ apdb-migrate-sql show-current -s SCHEMA_NAME $APDB_URL
 
 If its output is empty then ``alembic_version`` does not exist and has to be created and current revisions stamped into it with this command::
 
-    $ apdb-migrate-sql stamp $APDB_URL
+    $ apdb-migrate-sql stamp -s SCHEMA_NAME $APDB_URL
 
 Once the revisions are stamped the ``show-current`` command should produce non-empty output::
 
-    $ apdb-migrate-sql show-current $APDB_URL
+    $ apdb-migrate-sql show-current -s SCHEMA_NAME $APDB_URL
     ApdbSql_0.1.0 (head)
     schema_0.1.0
 
@@ -58,7 +63,7 @@ The ``show-current`` command displays versions/revisions from both these tables.
 
 Without options it shows revisions from ``alembic_version`` table::
 
-    $ apdb-migrate-sql show-current $APDB_URL
+    $ apdb-migrate-sql show-current -s SCHEMA_NAME $APDB_URL
     ApdbSql_0.1.0 (head)
     schema_0.1.0
 
@@ -68,7 +73,7 @@ If ``(head)`` is missing then newer revisions exist in the revision history, and
 
 With the ``--metadata`` (or ``-m``) option this command displays information from the ``metadata`` table::
 
-    $ apdb-migrate-sql show-current --metadata $APDB_URL
+    $ apdb-migrate-sql show-current --metadata -s SCHEMA_NAME $APDB_URL
     ApdbSql: 0.1.0 -> ApdbSql_0.1.0 (head)
     schema: 0.1.0 -> schema_0.1.0
 
@@ -86,20 +91,20 @@ Suppose you look at the ``show-current`` output and noticed that it shows revisi
 To check which new revisions exist run the ``show-history`` command for that manager::
 
     $ apdb-migrate-sql show-history schema
-    schema_0.1.1 -> schema_0.2.0 (schema) (head), Migration script for schema 0.2.0.
+    schema_0.1.1 -> schema_1.0.0 (schema) (head), Migration script for schema 1.0.0.
     schema_0.1.0 -> schema_0.1.1 (schema), Migration script for schema 0.1.1.
     schema_root -> schema_0.1.0 (schema), Migration script for schema 0.1.0.
     <base> -> schema_root (schema), An initial pseudo-revision of the 'schema' tree.
 
-You can tell that revision ``schema_0.1.0`` can be upgraded to ``schema_0.1.1``, and the latter can be also upgraded to ``schema_0.2.0``.
+You can tell that revision ``schema_0.1.0`` can be upgraded to ``schema_0.1.1``, and the latter can be also upgraded to ``schema_1.0.0``.
 
-With Alembic each migration has to be performed as a separate step, providing an explicit revision number.
-The two commands that perform the upgrade to the latest version 6.0.2 are::
+Migration can be performed as separate steps, providing an explicit revision number for each step, or as one command specifying final revision.
+An example of migration with two separate steps that perform the upgrade to the latest version 1.0.0 are::
 
-    $ apdb-migrate-sql upgrade $APDB_URL schema_0.1.1
-    $ apdb-migrate-sql upgrade $APDB_URL schema_0.2.0
+    $ apdb-migrate-sql upgrade -s SCHEMA_NAME $APDB_URL schema_0.1.1
+    $ apdb-migrate-sql upgrade -s SCHEMA_NAME $APDB_URL schema_1.0.0
 
-After that ``show-current`` should show ``schema_0.2.0 (head)`` in its output.
+After that ``show-current`` should show ``schema_1.0.0 (head)`` in its output.
 
 Usually migration scripts are running in a single transaction, if migration fails for some reason, the state of the schema should remain unchanged.
 
@@ -112,8 +117,8 @@ Downgrading schema
 
 It is possible to also switch the schema to a previous revision via ``downgrade`` command.
 The command takes a revision number which should be a revision preceding the current one.
-For example, to downgrade ``schema`` tree from current ``schema_0.2.0`` to previous ``schema_0.1.1`` run this command::
+For example, to downgrade ``schema`` tree from current ``schema_1.0.0`` to previous ``schema_0.1.1`` run this command::
 
-    $ apdb-migrate-sql downgrade $APDB_URL schema_0.1.1
+    $ apdb-migrate-sql downgrade -s SCHEMA_NAME $APDB_URL schema_0.1.1
 
 Of course, the migration script has to implement the ``downgrade()`` method for this, which may not always be true.

migrations/sql/schema/schema_1.1.0.py

@@ -0,0 +1,56 @@
+"""Migration script for schema 1.1.0.
+
+Revision ID: schema_1.1.0
+Revises: schema_1.0.0
+Create Date: 2024-05-09 11:51:35.137715
+"""
+
+import logging
+
+import sqlalchemy
+from lsst.dax.apdb_migrate.sql.context import Context
+
+# revision identifiers, used by Alembic.
+revision = "schema_1.1.0"
+down_revision = "schema_1.0.0"
+branch_labels = None
+depends_on = None
+
+_LOG = logging.getLogger(__name__)
+
+
+def upgrade() -> None:
+    """Upgrade 'schema' tree from 1.0.0 to 1.1.0 (ticket DM-44092).
+
+    Summary of changes:
+
+      - Remove columns {b}_lcPeriodic and {b}_lcNonPeriodic from DiaObject
+        table ({b} is one of six bands {ugrizy}).
+    """
+    ctx = Context()
+
+    # In addition to DiaObject table there could be DiaObjectChunks, though it
+    # is unlikely that SQL backend will ever be configured for replication.
+    tables = ["DiaObject"]
+    inspect = sqlalchemy.inspect(ctx.bind)
+    chunks_table = "DiaObjectChunks"
+    if inspect.has_table(chunks_table, schema=ctx.schema):
+        _LOG.info("Found replication table %s", chunks_table)
+        tables.append(chunks_table)
+
+    for table in tables:
+        with ctx.batch_alter_table(table) as batch_op:
+            for column in ("lcPeriodic", "lcNonPeriodic"):
+                for band in "ugrizy":
+                    column_name = f"{band}_{column}"
+                    _LOG.info("Dropping column %s.%s", table, column_name)
+                    batch_op.drop_column(column_name)
+
+    # Update metadata version.
+    version = revision.split("_")[-1]
+    ctx.apdb_meta.update_tree_version("schema", version)
+
+
+def downgrade() -> None:
+    """Downgrade is not needed as those columns were never used."""
+    raise NotImplementedError()