Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependencies 2025-01-13 #56

Merged
merged 4 commits into from
Jan 13, 2025
Merged

Update dependencies 2025-01-13 #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
877da70
Update dependencies 2025-01-13
Nicoretti Jan 13, 2025
c1332e2
Update github workflows
Nicoretti Jan 13, 2025
f3d23be
Update github workflows
Nicoretti Jan 13, 2025
1c19573
Fix code format
Nicoretti Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-and-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0

- name: Build Artifacts
run: poetry build
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/check-release-tag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0

- name: Check Tag Version
# make sure the pushed/created tag matched the project version
Expand Down
62 changes: 38 additions & 24 deletions .github/workflows/checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
fetch-depth: 0

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0

- name: Check Version(s)
run: |
Expand All @@ -35,27 +35,30 @@ jobs:
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0

- name: Build Documentation
run: |
poetry run python -m nox -s docs:build

build-matrix:
name: Generate Build Matrix
uses: ./.github/workflows/matrix-python.yml

Lint:
name: Linting (Python-${{ matrix.python-version }})
needs: [ Version-Check ]
needs: [ Version-Check, build-matrix ]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
matrix: ${{ fromJson(needs.build-matrix.outputs.matrix) }}

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0
with:
python-version: ${{ matrix.python-version }}

Expand All @@ -71,19 +74,18 @@ jobs:

Type-Check:
name: Type Checking (Python-${{ matrix.python-version }})
needs: [ Version-Check ]
needs: [ Version-Check, build-matrix ]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
matrix: ${{ fromJson(needs.build-matrix.outputs.matrix) }}

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0
with:
python-version: ${{ matrix.python-version }}

Expand All @@ -92,19 +94,18 @@ jobs:

Security:
name: Security Checks (Python-${{ matrix.python-version }})
needs: [ Version-Check ]
needs: [ Version-Check, build-matrix ]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
matrix: ${{ fromJson(needs.build-matrix.outputs.matrix) }}

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0
with:
python-version: ${{ matrix.python-version }}

Expand All @@ -118,34 +119,47 @@ jobs:
path: .security.json
include-hidden-files: true

Format:
name: Format Check
runs-on: ubuntu-latest

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]
with:
python-version: "3.9"

- name: Run format check
run: poetry run nox -s project:format

Tests:
name: Tests (Python-${{ matrix.python-version }}, Exasol-${{ matrix.exasol-version}})
needs: [ Documentation, Lint, Type-Check, Security ]
name: Unit-Tests (Python-${{ matrix.python-version }}, Exasol-${{ matrix.exasol-version}})
needs: [ Documentation, Lint, Type-Check, Security, Format, build-matrix ]
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.ALTERNATIVE_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
exasol-version: ["7.1.9"]
matrix: ${{ fromJson(needs.build-matrix.outputs.matrix) }}

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0
with:
python-version: ${{ matrix.python-version }}

- name: Run Tests and Compute Coverage
run: poetry run nox -s test:coverage -- -- --db-version ${{ matrix.exasol-version }}
- name: Run Tests and Collect Coverage
run: poetry run nox -s test:unit -- -- --coverage

- name: Upload Artifacts
uses: actions/[email protected]
with:
name: coverage-python${{ matrix.python-version }}
name: coverage-python${{ matrix.python-version }}-fast
path: .coverage
include-hidden-files: true

6 changes: 3 additions & 3 deletions .github/workflows/gh-pages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Publish Documentation

on:
on:
workflow_call:
workflow_dispatch:

Expand All @@ -16,11 +16,11 @@ jobs:
fetch-depth: 0

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0

- name: Build Documentation
run: |
poetry run sphinx-multiversion doc/ .html-documentation
poetry run nox -s docs:multiversion

- name: Deploy
uses: JamesIves/[email protected]
Expand Down
30 changes: 30 additions & 0 deletions .github/workflows/matrix-all.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Build Matrix (All Versions)

on:
workflow_call:
outputs:
matrix:
description: "Generates the all versions build matrix"
value: ${{ jobs.all_versions.outputs.matrix }}

jobs:
all_versions:

runs-on: ubuntu-latest

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]

- name: Generate matrix
run: poetry run nox -s matrix:all

- id: set-matrix
run: |
echo "matrix=$(poetry run nox -s matrix:all)" >> $GITHUB_OUTPUT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am correct, then nox task matrix:all is defined in PTB.
But still uses database version 7.1.9, which is quite outdated.
As 7.1.30 and 8.32.0 are already available.
Do you plan to update this?
How can a single project override the default settings?
I propose to support using an environment variable here and to simply set it in the workflow file.

Copy link
Contributor Author

@Nicoretti Nicoretti Jan 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are multiple ways to do that, for details see the changlog where this was introduced: https://exasol.github.io/python-toolbox/main/changes/changes_0.20.0.html#added


outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
30 changes: 30 additions & 0 deletions .github/workflows/matrix-exasol.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Build Matrix (Exasol)

on:
workflow_call:
outputs:
matrix:
description: "Generates the exasol version build matrix"
value: ${{ jobs.exasol_versions.outputs.matrix }}

jobs:
exasol_versions:

runs-on: ubuntu-latest

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]

- name: Generate matrix
run: poetry run nox -s matrix:exasol

- id: set-matrix
run: |
echo "matrix=$(poetry run nox -s matrix:exasol)" >> $GITHUB_OUTPUT

outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
30 changes: 30 additions & 0 deletions .github/workflows/matrix-python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Build Matrix (Python)

on:
workflow_call:
outputs:
matrix:
description: "Generates the python version build matrix"
value: ${{ jobs.python_versions.outputs.matrix }}

jobs:
python_versions:

runs-on: ubuntu-latest

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]

- name: Generate matrix
run: poetry run nox -s matrix:python

- id: set-matrix
run: |
echo "matrix=$(poetry run nox -s matrix:python)" >> $GITHUB_OUTPUT

outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
22 changes: 3 additions & 19 deletions .github/workflows/merge-gate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,33 +14,17 @@ jobs:

slow-checks:
name: Slow
runs-on: ubuntu-latest

# Even though the environment "manual-approval" will be created automatically,
# it still needs to be configured to require interactive review.
# See project settings on GitHub (Settings / Environments / manual-approval).
environment: manual-approval

# Replace the steps below with the required actions
# and/or add additional jobs if required
# Note:
# If you add additional jobs, make sure they are added as a requirement
# to the approve-merge job's input requirements (needs).
steps:
- name: Tests
run: |
echo "Slow tests ran successfully"

uses: ./.github/workflows/slow-checks.yml

# This job ensures inputs have been executed successfully.
approve-merge:
name: Allow Merge
name: Allow Merge
runs-on: ubuntu-latest
# If you need additional jobs to be part of the merge gate, add them below
needs: [ fast-checks, slow-checks ]

# Each job requires a step, so we added this dummy step.
steps:
- name: Approve
- name: Approve
run: |
echo "Merge Approved"
5 changes: 3 additions & 2 deletions .github/workflows/report.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
fetch-depth: 0

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/python-environment@0.17.0
uses: exasol/python-toolbox/.github/actions/python-environment@0.20.0

- name: Download Artifacts
uses: actions/[email protected]
Expand All @@ -30,7 +30,8 @@ jobs:
- name: Copy Artifacts into Root Folder
working-directory: ./artifacts
run: |
cp coverage-python3.9/.coverage ../
poetry run coverage combine --keep coverage-python3.9*/.coverage
cp .coverage ../
cp lint-python3.9/.lint.txt ../
cp security-python3.9/.security.json ../

Expand Down
46 changes: 46 additions & 0 deletions .github/workflows/slow-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Slow-Checks

on:
workflow_call:
secrets:
ALTERNATIVE_GITHUB_TOKEN:
required: false

jobs:

build-matrix:
name: Generate Build Matrix
uses: ./.github/workflows/matrix-all.yml

Tests:
name: Integration-Tests (Python-${{ matrix.python-version }}, Exasol-${{ matrix.exasol-version}})
needs: [ build-matrix ]
runs-on: ubuntu-latest
# Even though the environment "manual-approval" will be created automatically,
# it still needs to be configured to require interactive review.
# See project settings on GitHub (Settings / Environments / manual-approval).
environment: manual-approval
env:
GITHUB_TOKEN: ${{ secrets.ALTERNATIVE_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.build-matrix.outputs.matrix) }}

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Setup Python & Poetry Environment
uses: exasol/python-toolbox/.github/actions/[email protected]
with:
python-version: ${{ matrix.python-version }}

- name: Run Tests and Collect Coverage
run: poetry run nox -s test:integration -- -- --coverage --db-version ${{ matrix.exasol-version }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the coverage results be merged with the results for the unit tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In bucketfs-python we needed to use --itde-db-version.
Which one is correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the coverage results be merged with the results for the unit tests?

yes the results will be merged.

In bucketfs-python we needed to use --itde-db-version.
Which one is correct?

Depends on what tools and versions of those tools are used

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In bucketfs-python we needed to use --itde-db-version. Which one is correct?

Depends on what tools and versions of those tools are used

Can you explain this a bit more?
Does error-reporting-python use a database at all?
is --db-version correct here?
Why does bucketfs-python need to use --itde-db-version?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does bucketfs-python need to use --itde-db-version?

Feel free to call me and we have a look at it, but I don't think this is subject of this PR.
That said happy to investigate your issues/question at hand regarding bucketfs-python with yout, but not in the context and PR of another project.

Does error-reporting-python use a database at all?

No, but it does have integration tests

is --db-version correct here?

Should be ok, afik if no DB is requested nothing happens with that parameter.
Keeping it aligned with the template will simplify future udpates though.


- name: Upload Artifacts
uses: actions/[email protected]
with:
name: coverage-python${{ matrix.python-version }}-slow
path: .coverage
include-hidden-files: true
1 change: 1 addition & 0 deletions doc/changes/unreleased.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
## Internal

* Relocked Dependencies
* Update toolbox workflows
Loading
Loading