Merge pull request #256 from plone/testing-matrix

Testing matrix

README.md

@@ -166,27 +166,28 @@ _your own configuration lines_
 ```
 
 
-#### `.github/workflows/meta.yml`
+#### `.github/workflows/test-matrix.yml`
+
+Run the distribution test on a combination of Plone and Python versions.
 
 > [!NOTE]
-> The variables `TEST_OS_VERSIONS` and `TEST_PYTHON_VERSIONS` variables need to exist, either at the GitHub organization level, or at the repository level.
+> See the `test_matrix` option in [`tox`](#toxini) configuration file.
+
+> [!TIP]
+> 🍀 `plone.meta` tries to be a bit more environmentally friendly.
+> On GitHub, only the first and last Python versions will be added for testing.
 
-See the [GitHub documentation about variables](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables).
 
-These variables are expected to be lists.
+#### `.github/workflows/meta.yml`
 
--   `TEST_OS_VERSIONS`: `["ubuntu-latest",]`
-    -   a list of valid operating system names according [to GitHub Actions](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources)
--   `TEST_PYTHON_VERSIONS`: `["3.13", "3.12", "3.11", "3.10", ]`
-    -   a list of valid Python versions according to [python-versions action](https://github.com/actions/python-versions/)
+Customize the GitHub Action jobs run on every change pushed to GitHub.
 
 Add the `[github]` TOML table in `.meta.toml`, and set the enabled jobs with the `jobs` key.
 
 ```toml
 [github]
 jobs = [
     "qa",
-    "test",
     "coverage",
     "dependencies",
     "release_ready",
@@ -220,17 +221,6 @@ To install specific operating system level dependencies, which must be Ubuntu pa
 os_dependencies = "git libxml2"
 ```
 
-To run tests against specific Python versions, specify the `py_versions` key as follows.
-
-> [!NOTE]
-> The GitHub Action expects a string to be parsed as a JSON array.
-> Unfortunately, quotes need to be escaped.
-
-```toml
-[github]
-py_versions = "[\"3.11\", \"3.10\"]"
-```
-
 Extend GitHub workflow configuration with additional jobs by setting the values for the `extra_lines` key.
 
 ```toml
@@ -253,13 +243,18 @@ _your own configuration lines_
 """
 ```
 
-Specify a custom Docker image, if the default does not fit your needs, in the `custom_image` key.
+Specify custom Docker images in the `custom_images` key.
+
+The dictionary keys needs to be Python versions and the value a Docker image for that Python version.
 
 ```toml
 [gitlab]
-custom_image = "python:3.11-bullseye"
+custom_images = {"3.13" = "python:3.13-bookworm", "3.12" = "python:3.12-bookworm"}
 ```
 
+> [!TIP]
+> To tweak the jobs that will be run, you can customize the `test_matrix` option from `[tox]` table.
+
 To install specific test and coverage dependencies, add the `os_dependencies` key as follows.
 
 ```toml
@@ -449,6 +444,15 @@ _your own configuration lines_
 """
 ```
 
+`plone.meta` generates a list of Python and Plone version combinations to run the distribution tests.
+
+You can customize that by defining your testing matrix:
+
+```toml
+[tox]
+test_matrix = { "6.1" = ["3.13", "3.10"], "6.0": ["3.13", "3.9"] }
+```
+
 Extend the list of default `tox` environments in the `envlist_lines` key.
 Add extra top level configuration for `tox` in the `config_lines` key.
 
@@ -493,14 +497,17 @@ test_deps_additional = """
 ```
 
 When using `plone.meta` outside of plone core packages, there might be extra version pins, or overrides over the official versions.
-To specify a custom constraints file, use the `constraints_file` key.
+To specify a custom constraints file, use the `constraints_files` key.
 
 Generating a custom `constraints.txt` is out of scope for `plone.meta` itself.
 There are plenty of tools that can do that though.
 
+> [!WARNING]
+> You need to specify the same Plone versions as in the `test_matrix` option or the default provided by `plone.meta`.
+
 ```toml
 [tox]
-constraints_file = "https://my-server.com/constraints.txt"
+constraints_files = { "6.1" = "https://my-server.com/constraints-6.1.txt", "6.0" = "https://my-server.com/constraints-6.0.txt" }
 ```
 
 Extend the list of custom environment variables that the test and coverage environments can get in the `test_environment_variables` key.

UPGRADE.md

@@ -0,0 +1,67 @@
+# Upgrade guide
+
+This file describes the breaking changes between major versions.
+
+## From `main` to `2.x`
+
+
+### Test matrix
+
+You can now define the combinations of Plone versions and Python versions to be tested.
+
+`plone.meta` provides a default combination (the same as used by Plone itself), but can be overridden in the file `.meta.toml`.
+
+```toml
+[tox]
+test_matrix = {"6.2" = ["3.14", "3.11"], "6.1" = ["3.13", "3.12", "3.11", "3.10", "3.9"]}
+```
+
+This will be used to generate the necessary `tox` environments and the GitHub Actions that will be run on each pull request.
+
+> [!TIP]
+> 🍀 `plone.meta` tries to be a bit more environmentally friendly.
+> On GitHub, only the first and last Python versions will be added for testing.
+
+
+### Constraints
+
+The `constraints_file` option in `.meta.toml`'s `[tox]` table was renamed to `constraints_files`, and the type of its value was changed from a string to a dictionary.
+
+This option continues to be optional.
+
+The dictionary keys must be Plone versions, and each key's value must be the constraints file for that Plone version.
+
+```toml
+[tox]
+# OLD
+constraints_file = "https://example.org/my-custom-constraints.txt"
+# NEW
+constraints_files = {"6.2" = "https://example.org/constraints.6.2.txt", "6.1" = "https://example.org/constraints.6.1.txt"}
+```
+
+
+### GitHub Actions
+
+The `py_versions` option in `.meta.toml`'s `[github]` table is deprecated.
+We use the new `test_matrix` option from `[tox]` table, as we now can run multiple Python versions from within `tox` itself.
+
+### GitHub variables
+
+The GitHub variables `TEST_OS_VERSIONS` and `TEST_PYTHON_VERSIONS` are deprecated and no longer used.
+
+
+### GitLab images
+
+The `custom_image` option in `.meta.toml`'s `[gitlab]` table was renamed to `custom_images`, and the type of its value was changed from a string to a dictionary.
+
+This option continues to be optional.
+
+The dictionary keys must be Python versions, and the values a Docker image for that Python version.
+
+```toml
+[tox]
+# OLD
+custom_images = "python:3.11-bullseye"
+# NEW
+custom_images = {"3.13" = "python:3.13-bookworm", "3.12" = "python:3.12-bookworm"}
+```

news/234.breaking

@@ -0,0 +1 @@
+Support combinations of multiple versions of Plone and Python in tox. @gforcada

news/234.documentation

@@ -0,0 +1 @@
+Add an upgrade guide. @gforcada

src/plone/meta/config_package.py

@@ -31,10 +31,21 @@
 --> """
 DEFAULT = object()
 
+# List all python versions we want to test a given Plone version against
+TOX_TEST_MATRIX = {
+    "6.1": ["3.13", "3.12", "3.11", "3.10"],
+    "6.0": ["3.13", "3.12", "3.11", "3.10", "3.9"],
+}
+
 MXDEV_CONSTRAINTS = "constraints-mxdev.txt"
-PLONE_CONSTRAINTS = "https://dist.plone.org/release/6.0-dev/constraints.txt"
 
-DOCKER_IMAGE = "python:3.11-bullseye"
+DOCKER_IMAGES = {
+    "3.13": "python:3.13-bookworm",
+    "3.12": "python:3.12-bookworm",
+    "3.11": "python:3.11-bookworm",
+    "3.10": "python:3.10-bookworm",
+    "3.9": "python:3.9-bookworm",
+}
 
 # Rather than pointing configured repositories to `plone.meta`'s `main` branch
 # to get their GHA workflows, point them to an ever evolving branch.
@@ -46,7 +57,6 @@
 
 GHA_DEFAULT_JOBS = [
     "qa",
-    "test",
     "coverage",
     "dependencies",
     "release_ready",
@@ -347,7 +357,7 @@ def tox(self):
             "tox",
             (
                 "constrain_package_deps",
-                "constraints_file",
+                "constraints_files",
                 "envlist_lines",
                 "testenv_options",
                 "use_mxdev",
@@ -358,6 +368,7 @@ def tox(self):
                 "extra_lines",
                 "use_pytest_plone",
                 "package_name",
+                "test_matrix",
             ),
         )
         use_mxdev = options.get("use_mxdev", False)
@@ -369,15 +380,72 @@ def tox(self):
 
         if not options["constrain_package_deps"]:
             options["constrain_package_deps"] = "false" if use_mxdev else "true"
-        if not options["constraints_file"]:
-            constraints_file = MXDEV_CONSTRAINTS if use_mxdev else PLONE_CONSTRAINTS
-            options["constraints_file"] = constraints_file
+
         if options["use_pytest_plone"] is not False:
             # Default is '', so turn it into True
             options["use_pytest_plone"] = True
 
+        options.update(self._handle_constraints_files(options))
+        options["plone_envlist_lines"] = self._handle_testing_matrix(
+            options["test_matrix"]
+        )
         return self.copy_with_meta("tox.ini.j2", **options)
 
+    def _handle_constraints_files(self, options):
+        if options.get("use_mxdev", False):
+            constraints = single_constraints = f"-c {MXDEV_CONSTRAINTS}"
+        else:
+            constraints = options["constraints_files"]
+            test_matrix = options.get("test_matrix")
+            if not test_matrix:
+                test_matrix = TOX_TEST_MATRIX
+            plone_versions = list(test_matrix.keys())
+
+            single_constraints = f"-c https://dist.plone.org/release/{plone_versions[0]}-dev/constraints.txt"
+            if constraints:
+                first_plone_version = list(constraints.keys())[0]
+                single_constraints = f"-c {constraints[first_plone_version]}"
+                if len(test_matrix.keys()) != len(constraints.keys()):
+                    raise ValueError(
+                        "`constraints_files` and `test_matrix` need to provide the same Plone versions."
+                        f"They provide {list(constraints.keys())} and {list(test_matrix.keys())} respectively."
+                    )
+
+            lines = []
+            for plone_version in plone_versions:
+                no_dot = plone_version.replace(".", "")
+                url = f"https://dist.plone.org/release/{plone_version}-dev/constraints.txt"
+                if constraints:
+                    url = constraints[plone_version]
+                lines.append(f"plone{no_dot}: -c {url}")
+            constraints = "\n    ".join(lines)
+        return {
+            "constraints_file": constraints,
+            "single_constraints_file": single_constraints,
+        }
+
+    def _handle_testing_matrix(self, test_matrix):
+        """Generate the tox environments matrix of Python and Plone versions to test
+
+        Either `options` provides a dictionary like:
+        {
+          'PLONE_VERSION_1': [LIST_OF_PYTHON_VERSIONS],
+          'PLONE_VERSION_2': [LIST_OF_PYTHON_VERSIONS],
+        }
+
+        Or the default `TOX_TEST_MATRIX` is used.
+        """
+        lines = []
+        matrix = TOX_TEST_MATRIX
+        if test_matrix:
+            matrix = test_matrix
+        for plone_version, python_versions in matrix.items():
+            no_dot_plone = plone_version.replace(".", "")
+            for python_version in python_versions:
+                no_dot_python = python_version.replace(".", "")
+                lines.append(f"py{no_dot_python}-plone{no_dot_plone}")
+        return "\n    ".join(lines)
+
     def _detect_robotframework(self):
         """Dynamically find out if robotframework is used in the package.
 
@@ -429,7 +497,6 @@ def gha_workflows(self):
                 "ref",
                 "jobs",
                 "os_dependencies",
-                "py_versions",
                 "extra_lines",
             ),
         )
@@ -450,27 +517,70 @@ def gha_workflows(self):
                 "A `dependabot.yml` file at the top-level was found, please remove it",
             )
 
-        return [meta_file, dependabot]
+        options["gh_config_lines"] = self.handle_gh_actions()
+        testing_file = self.copy_with_meta(
+            "test-matrix.yml.j2",
+            destination=workflows_folder / "test-matrix.yml",
+            **options,
+        )
+
+        return [meta_file, dependabot, testing_file]
+
+    def handle_gh_actions(self):
+        options = self._get_options_for("tox", ("test_matrix",))
+        test_matrix = getattr(options, "test_matrix", TOX_TEST_MATRIX)
+        combinations = []
+        for plone_version, python_versions in test_matrix.items():
+            no_dot_plone = plone_version.replace(".", "")
+            for py_version in (python_versions[0], python_versions[-1]):
+                no_dot_python = py_version.replace(".", "")
+                combinations.append(
+                    f'["{py_version}", "{plone_version} on py{py_version}", "py{no_dot_python}-plone{no_dot_plone}"]'
+                )
+        return "\n        - ".join(combinations)
 
     def gitlab_ci(self):
         if not self.is_gitlab:
             return []
         options = self._get_options_for(
             "gitlab",
             (
-                "custom_image",
+                "custom_images",
                 "os_dependencies",
                 "extra_lines",
                 "jobs",
             ),
         )
-        if not options["custom_image"]:
-            options["custom_image"] = DOCKER_IMAGE
+        options.update(self._gitlab_testing_matrix(options["custom_images"]))
         options["destination"] = self.path / ".gitlab-ci.yml"
         if not options.get("jobs"):
             options["jobs"] = GITLAB_DEFAULT_JOBS
         return self.copy_with_meta("gitlab-ci.yml.j2", **options)
 
+    def _gitlab_testing_matrix(self, custom_images):
+        options = self._get_options_for("tox", ("test_matrix",))
+        test_matrix = getattr(options, "test_matrix", TOX_TEST_MATRIX)
+        combinations = []
+        image = ""
+        for plone_version, python_versions in test_matrix.items():
+            no_dot_plone = plone_version.replace(".", "")
+            for py_version in (python_versions[0], python_versions[-1]):
+                no_dot_python = py_version.replace(".", "")
+                image = DOCKER_IMAGES.get(py_version)
+                if custom_images:
+                    image = custom_images.get(py_version)
+                if not image:
+                    raise ValueError(
+                        f"There is no Docker image defined for Python {py_version}. "
+                        "Either provide it in the `custom_images` option or report an issue to `plone.meta`."
+                    )
+
+                combinations.append((image, f"py{no_dot_python}-plone{no_dot_plone}"))
+        return {
+            "testing_matrix": combinations,
+            "custom_image": image,
+        }
+
     def flake8(self):
         options = self._get_options_for("flake8", ("extra_lines",))
         destination = self.path / ".flake8"