chore: bump Python version to 3.12 and update requirements (acts-proj…

…ect#2770)

This PR updates the existing requirements for python-based CI jobs to the newest possible version. It also tries to enforce `python 3.12` (newest) wherever possible.

## Performance boost

linux_examples_tests: 18.5 min -> 15 min

## Important changes/notes

The `pytest-check 2.2.0..2.2.2` soft-break some CI checks. Therefore, we fix it to `2.1.5.` More to this in issue acts-project#2777 :
- linux_test_examples (verified)
- test_exatrkx_python (very likely)

`uproot 5.x` and `awkward 2.x` change the root-hashes.

The test `test_exatrkx_python` got a new separate requirements-file, because the test only runs on `ubuntu 20.04` which uses `python 3.8`. The newest `numpy` (for `awkward` and `uproot`) doesn't support `python 3.8` anymore. When we migrate this test to a newer `OS`, we can use the shared `requirements.txt` again.

`clang_tidy` had to be adapted at a few points, since some of`pydantic`'s functionalities were deprecated.


## Side note to `pip-compile`

Before generating a new `requirements.txt` one should delete the old one. Otherwise, the versions for the packages are not updated (makes sense to some extent).

## Blocked by other PRs
We created files, that were referred to at places we did not check.
- acts-project#2778

## Other Questions

The release-workflows still use `3.8`. Should we upgrade as well, or are we too afraid, to break something in there?
https://github.com/acts-project/acts/blob/main/.github/workflows/release.yml
https://github.com/acts-project/acts/blob/main/.github/workflows/release_pr.yml
-> An update to the release script will come in
- acts-project#2779

.github/workflows/checks.yml

@@ -33,7 +33,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Install black
         run: pip install black==22.3.0
       - name: Run black format check
@@ -45,7 +45,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           sudo apt-get install -y git
@@ -56,7 +56,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           CI/check_include_guards.py . --fail-global --exclude "*thirdparty/*"
@@ -73,7 +73,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           CI/check_end_of_file.py . --exclude "thirdparty/*" --reject-multiple-newlines --github
@@ -83,7 +83,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           CI/check_size_t.py . --exclude "thirdparty/*"
@@ -100,7 +100,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           CI/check_smearing_config.py .
@@ -110,7 +110,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           docs/parse_cmake_options.py CMakeLists.txt --write docs/getting_started.md --verify
@@ -120,7 +120,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Install codespell
         run: >
           pip install codespell==2.2.5
@@ -143,7 +143,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Install dependencies
         run: >
           pip install -r CI/requirements_fpe_masks.txt
@@ -156,7 +156,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Check
         run: >
           CI/check_unused_files.py

.github/workflows/report.yml

@@ -57,7 +57,7 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |
             if(!${{ steps.dl-af.outputs.result }}) {
-              console.log("No artifact => no PR numner");
+              console.log("No artifact => no PR number");
               return false;
             }
             let fs = require('fs');
@@ -86,7 +86,7 @@ jobs:
 
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Render comment
         if: steps.get-pr-number.outputs.result != 'false'

.gitlab-ci.yml

@@ -132,7 +132,7 @@ test_exatrkx_python:
     - git clone $CLONE_URL src
     - cd src
     - git checkout $HEAD_SHA
-    - pip3 install -r Examples/Python/tests/requirements.txt
+    - pip3 install -r Examples/Python/tests/requirements_ubuntu2004.txt
     - nvidia-smi
     - pytest -rFsv -k test_exatrkx
 

CI/clang_tidy/check_clang_tidy.py

@@ -16,7 +16,7 @@
 from rich.table import Table
 
 
-from item import Item, ItemCollection
+from item import ItemCollection
 
 
 def main():
@@ -34,7 +34,7 @@ def main():
 
     data = []
     with args.report.open() as fh:
-        data = ItemCollection(__root__=json.load(fh)).__root__
+        data = ItemCollection(root=json.load(fh)).root
         for item in data:
             if args.strip_prefix_path and not item.path.is_absolute:
                 item.path = item.path.relative_to(args.strip_prefix_path)

CI/clang_tidy/item.py

@@ -1,4 +1,3 @@
-from dataclasses import dataclass
 from pathlib import Path
 from typing import List
 
@@ -25,5 +24,5 @@ def __eq__(self, other):
         )
 
 
-class ItemCollection(pydantic.BaseModel):
-    __root__: List[Item]
+class ItemCollection(pydantic.RootModel[List[Item]]):
+    pass

CI/clang_tidy/parse_clang_tidy.py

@@ -7,14 +7,8 @@
 
 import argparse
 import re
-from collections import namedtuple
-from itertools import groupby
 import os
-import html
 from fnmatch import fnmatch
-import json
-import sys
-from dataclasses import dataclass
 from pathlib import Path
 
 from item import Item, ItemCollection
@@ -148,7 +142,7 @@ def subpath(m):
 
     print("Write to", args.output)
     with open(args.output, "w+") as jf:
-        jf.write(ItemCollection(__root__=items).json(indent=2))
+        jf.write(ItemCollection(root=items).model_dump_json(indent=2))
 
 
 if "__main__" == __name__:

CI/clang_tidy/requirements.in

@@ -1,4 +1,4 @@
-PyYAML
+pyyaml
 rich
 pydantic
 codereport

CI/clang_tidy/requirements.txt

@@ -1,39 +1,47 @@
 #
-# This file is autogenerated by pip-compile with python 3.9
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.12
+# by the following command:
 #
 #    pip-compile CI/clang_tidy/requirements.in
 #
+annotated-types==0.6.0
+    # via pydantic
 appdirs==1.4.4
     # via fs
 codereport==0.3.2
     # via -r CI/clang_tidy/requirements.in
-commonmark==0.9.1
-    # via rich
 fs==2.4.16
     # via codereport
 jinja2==3.1.2
     # via codereport
-markupsafe==2.1.1
+markdown-it-py==3.0.0
+    # via rich
+markupsafe==2.1.3
     # via jinja2
-pydantic==1.9.0
+mdurl==0.1.2
+    # via markdown-it-py
+pydantic==2.5.2
     # via -r CI/clang_tidy/requirements.in
-pygments==2.11.2
+pydantic-core==2.14.5
+    # via pydantic
+pygments==2.17.2
     # via
     #   codereport
     #   rich
 python-slugify==6.1.2
     # via codereport
-pyyaml==6.0
+pyyaml==6.0.1
     # via -r CI/clang_tidy/requirements.in
-rich==12.2.0
+rich==13.7.0
     # via -r CI/clang_tidy/requirements.in
 six==1.16.0
     # via fs
 text-unidecode==1.3
     # via python-slugify
-typing-extensions==4.1.1
-    # via pydantic
+typing-extensions==4.8.0
+    # via
+    #   pydantic
+    #   pydantic-core
 
 # The following packages are considered to be unsafe in a requirements file:
 # setuptools

CI/commands/requirements.txt

@@ -1,32 +1,28 @@
 #
-# This file is autogenerated by pip-compile with Python 3.10
+# This file is autogenerated by pip-compile with Python 3.12
 # by the following command:
 #
 #    pip-compile CI/commands/requirements.in
 #
-aiohttp==3.8.5
+aiohttp==3.9.1
     # via -r CI/commands/requirements.in
 aiosignal==1.3.1
     # via aiohttp
-async-timeout==4.0.3
-    # via aiohttp
 attrs==23.1.0
     # via aiohttp
-cffi==1.15.1
+cffi==1.16.0
     # via cryptography
-charset-normalizer==3.2.0
-    # via aiohttp
 click==8.1.7
     # via typer
-cryptography==41.0.3
+cryptography==41.0.7
     # via pyjwt
 frozenlist==1.4.0
     # via
     #   aiohttp
     #   aiosignal
 gidgethub==5.3.0
     # via -r CI/commands/requirements.in
-idna==3.4
+idna==3.6
     # via yarl
 multidict==6.0.4
     # via
@@ -38,9 +34,9 @@ pyjwt[crypto]==2.8.0
     # via gidgethub
 typer==0.9.0
     # via -r CI/commands/requirements.in
-typing-extensions==4.7.1
+typing-extensions==4.8.0
     # via typer
 uritemplate==4.1.1
     # via gidgethub
-yarl==1.9.2
+yarl==1.9.3
     # via aiohttp

CI/requirements_fpe_masks.txt

@@ -1,26 +1,22 @@
 #
-# This file is autogenerated by pip-compile with Python 3.10
+# This file is autogenerated by pip-compile with Python 3.12
 # by the following command:
 #
 #    pip-compile CI/requirements_fpe_masks.in
 #
-aiohttp==3.8.4
+aiohttp==3.9.1
     # via -r CI/requirements_fpe_masks.in
 aiosignal==1.3.1
     # via aiohttp
-async-timeout==4.0.2
-    # via aiohttp
 attrs==23.1.0
     # via aiohttp
 cffi==1.15.1
     # via cryptography
-charset-normalizer==3.1.0
-    # via aiohttp
 click==8.1.4
     # via typer
 cryptography==41.0.1
     # via pyjwt
-frozenlist==1.3.3
+frozenlist==1.4.0
     # via
     #   aiohttp
     #   aiosignal

Examples/Python/tests/requirements.in

@@ -1,7 +1,6 @@
 pytest
-pytest-check
+pytest-check==2.1.5
 uproot
 awkward
-pytest-rerunfailures
 pytest-xdist
 pyyaml

Examples/Python/tests/requirements.txt

@@ -1,47 +1,41 @@
 #
-# This file is autogenerated by pip-compile with Python 3.8
+# This file is autogenerated by pip-compile with Python 3.12
 # by the following command:
 #
 #    pip-compile Examples/Python/tests/requirements.in
 #
-attrs==21.4.0
-    # via pytest
-awkward==1.7.0
-    # via -r Examples/Python/tests/requirements.in
-execnet==1.9.0
+awkward==2.5.0
+    # via
+    #   -r Examples/Python/tests/requirements.in
+    #   uproot
+awkward-cpp==26
+    # via awkward
+execnet==2.0.2
     # via pytest-xdist
-iniconfig==1.1.1
+iniconfig==2.0.0
     # via pytest
-numpy==1.22.2
+numpy==1.26.2
     # via
     #   awkward
+    #   awkward-cpp
     #   uproot
-packaging==21.3
-    # via pytest
-pluggy==1.0.0
-    # via pytest
-py==1.11.0
+packaging==23.2
+    # via
+    #   awkward
+    #   pytest
+    #   uproot
+pluggy==1.3.0
     # via pytest
-pyparsing==3.0.7
-    # via packaging
-pytest==7.0.1
+pytest==7.4.3
     # via
     #   -r Examples/Python/tests/requirements.in
     #   pytest-check
-    #   pytest-rerunfailures
     #   pytest-xdist
-pytest-check==1.0.4
-    # via -r Examples/Python/tests/requirements.in
-pytest-rerunfailures==10.2
+pytest-check==2.1.5
     # via -r Examples/Python/tests/requirements.in
-pytest-xdist==3.2.1
+pytest-xdist==3.5.0
     # via -r Examples/Python/tests/requirements.in
-pyyaml==6.0
+pyyaml==6.0.1
     # via -r Examples/Python/tests/requirements.in
-tomli==2.0.1
-    # via pytest
-uproot==4.1.9
+uproot==5.1.2
     # via -r Examples/Python/tests/requirements.in
-
-# The following packages are considered to be unsafe in a requirements file:
-# setuptools