ci: Use uv for all pip installs #2444

matthewfeickert · 2024-02-16T20:50:11Z

Description

Use uv to try to speed up the installs of all the Python dependencies.

Fall back to using pip for Python 3.8 until astral-sh/uv#2062 is resolved.

Apply subtle changes to install commands in .github/workflows/dependencies-head.yml.

uv pip install --upgrade will try to upgrade all dependencies of the target package as well, which for the dependencies-head workflow isn't the goal. So remove the --upgrade from calls that also install from the scientific-python-nightly-wheels package index when testing only particular packages.
up pip and pip have different behavior with regards to --extra-index-url, as uv pip gives --extra-index-url priority over --index-url, where pip does not give priority to either. Use this with uv pip to give priority to the scientific-python-nightly-wheels package index.
- c.f. Clarify use of index-url vs. extra-index-url scientific-python/upload-nightly-action#76

Checklist Before Requesting Reviewer

Tests are passing
"WIP" removed from the title of the pull request
Selected an Assignee for the PR to be responsible for the log summary

Before Merging

For the PR Assignees:

Summarize commit messages into a comprehensive review of the PR

* Use 'uv pip' for all calls to 'pip install' and 'pip uninstall' in
  CI workflows.
   - c.f. https://github.com/astral-sh/uv/
   - Still use pip for Python 3.8 until https://github.com/astral-sh/uv/issues/2062
     is resolved.
* Apply subtle changes to install commands in .github/workflows/dependencies-head.yml.
   - 'uv pip install --upgrade' will try to upgrade all dependencies of the target
     package as well, which for the dependencies-head workflow isn't the goal. So
     remove the '--upgrade' from calls that also install from the
     scientific-python-nightly-wheels package index when testing only particular packages.
   - 'up pip' and 'pip' have different behavior with regards to --extra-index-url, as
     'uv pip' gives --extra-index-url priority over --index-url, where 'pip' does
     not give priority to either. Use this with 'uv pip' to give priority to the
     scientific-python-nightly-wheels package index.
* Add uv to the 'develop' extras.

matthewfeickert · 2024-02-16T21:07:02Z

Need astral-sh/uv#1526 or an alternative for this to work first.

Edit: Can do this for the time being:

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}

    - name: Set the VIRTUAL_ENV variable for uv to work
      run: |
        echo "VIRTUAL_ENV=${Python_ROOT_DIR}" >> $GITHUB_ENV

    - name: Install dependencies
      run: |
        python -m pip install uv
        python -m uv pip install --upgrade <stuff>

which comes from astral-sh/uv#1386 (comment).

matthewfeickert · 2024-02-16T21:14:31Z

Also needs astral-sh/uv#1531 so will have to wait for the next release of uv (if that PR goes in).

matthewfeickert · 2024-02-18T07:10:14Z

Given astral-sh/uv#1531 (comment) we also need astral-sh/uv#313 to go in.

matthewfeickert · 2024-02-28T23:21:19Z

Okay with astral-sh/uv#1526 in and using the

install "pyhf[extras] @ ."

pattern over

install ".[extras]"

pattern I think this should be good.

edit: No, uv pip is resolving dependencies for tensorflow different than pip does and this breaks.

matthewfeickert · 2024-02-29T00:25:51Z

Reported failure in astral-sh/uv#2062

matthewfeickert · 2024-03-28T06:39:16Z

Also blocked by astral-sh/uv#2685

matthewfeickert · 2024-04-26T05:02:58Z

So this is really blocked by papermill given nteract/papermill#789 is in and so just waiting for a papermill v2.5.1 or v2.6.0 release.

matthewfeickert · 2024-04-27T18:05:53Z

With the release of papermill v2.6.0 (nteract/papermill#785 (comment)) and relaxing to

...
    "papermill>=2.5.0",
    "scrapbook>=0.5.0",
...

astral-sh/uv#2685 is sidestepped, but the issue with TensorFlow dependency resolution in astral-sh/uv#2062 is still blocking.

codecov · 2024-04-27T18:14:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.21%. Comparing base (682ce76) to head (854906d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2444   +/-   ##
=======================================
  Coverage   98.21%   98.21%           
=======================================
  Files          69       69           
  Lines        4543     4543           
  Branches      804      804           
=======================================
  Hits         4462     4462           
  Misses         48       48           
  Partials       33       33

Flag	Coverage Δ
contrib	`97.79% <ø> (ø)`
doctest	`98.08% <ø> (ø)`
unittests-3.10	`96.23% <ø> (ø)`
unittests-3.11	`96.23% <ø> (ø)`
unittests-3.12	`96.23% <ø> (ø)`
unittests-3.8	`96.25% <ø> (ø)`
unittests-3.9	`96.27% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pyproject.toml

* Loosen the install requirements for papermill, scrapbook, and sphinxcontrib-bibtex to use lower bounds instead of compatible release constraints. * Required for PR #2444

This reverts commit 44fdb31.

matthewfeickert · 2024-04-27T18:56:01Z

.github/workflows/ci.yml

+      if: matrix.python-version != '3.8'
+      run: |
+        python -m pip install uv
+        uv pip install --system --upgrade pip setuptools wheel
+        uv pip install --system --upgrade ".[all,test]"
+
+      # c.f. https://github.com/astral-sh/uv/issues/2062
+    - name: Install dependencies (Python 3.8)
+      if: matrix.python-version == '3.8'


A bit of a hack, but until astral-sh/uv#2062 gets resolves, we might as well speed up everything else.

.github/workflows/dependencies-head.yml

matthewfeickert · 2024-04-27T19:02:31Z

.github/workflows/dependencies-head.yml

        # Need to use --extra-index-url as dependencies aren't on scientific-python-nightly-wheels package index.
        # Need to use --pre as dev releases will need priority over stable releases.
-        python -m pip install \
+        uv pip install --system \


Double check this as pip and uv handle --extra-index-url differently. c.f. scientific-python/upload-nightly-action#76

Yeah, this needed to get flipped to

uv pip install --system \ --pre \ --index-url https://pypi.org/simple/ \ --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \ matplotlib

as uv give priority to --extra-index-url.

$ pip list Package Version --------------- ------------------------- contourpy 1.3.0.dev1 cycler 0.12.1 fonttools 4.51.0 kiwisolver 1.4.5 matplotlib 3.10.0.dev114+g4cbef2d4c0 numpy 2.1.0.dev0 packaging 24.0 pillow 10.3.0 pip 24.0 pyparsing 3.1.2 python-dateutil 2.9.0.post0 setuptools 69.5.1 six 1.16.0 uv 0.1.39 wheel 0.43.0

* uv pip install --upgrade is going to try to upgrade numpy along with scipy, so to avoid this, don't use --upgrade when installing scipy from the nightly wheel index.

matthewfeickert · 2024-04-27T19:50:30Z

.github/workflows/dependencies-head.yml

+        # uv wants to upgrade dependencies (numpy) to a dev release too, so don't --upgrade
+        uv pip install --system --index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple scipy


This was previously forcing an upgrade of numpy with uv as well, but just want to test scipy which should work with numpy v1 or v2, and trying to test numpy v2 will break jax at the moment.

matthewfeickert · 2024-04-27T22:45:10Z

Okay, I think this is finally working as expected across all workflows now.

matthewfeickert · 2024-04-27T22:49:56Z

Some examples of install time drops:

OS	Python	`pip`	`uv pip`
Ubuntu	3.12	2m 57 s	38 s
macOS (Intel)	3.12	2m 51 s	50 s
macOS (Apple silicon)	3.12	1m 6 s	16 s

🚀

matthewfeickert · 2024-04-30T05:37:05Z

I'm going to approve and merge this myself. As always, PRs approved by a single core dev can be reverted as needed by the rest of the dev team.

* Backport: - PR #2444 - PR #2475

matthewfeickert added CI CI systems, GitHub Actions need-to-backport tmp label until can be backported to patch release branch labels Feb 16, 2024

matthewfeickert self-assigned this Feb 16, 2024

matthewfeickert mentioned this pull request Feb 29, 2024

uv pip and python -m pip resolve different versions of tensorflow in pyhf developer environment astral-sh/uv#2062

Open

matthewfeickert mentioned this pull request Mar 27, 2024

Allow pre-releases when using compatible release syntax astral-sh/uv#2685

Open

ikrommyd mentioned this pull request Mar 28, 2024

ci: use micromamba and uv and also install ROOT zfit/zfit-physics#69

Merged

matthewfeickert force-pushed the ci/use-uv-to-speed-up-installs branch 2 times, most recently from d5d84c6 to 01f001b Compare April 2, 2024 07:29

matthewfeickert force-pushed the ci/use-uv-to-speed-up-installs branch from 01f001b to f6d34f3 Compare April 27, 2024 18:00

matthewfeickert commented Apr 27, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

matthewfeickert mentioned this pull request Apr 27, 2024

build: Use lower bounds for all dependencies #2472

Merged

4 tasks

matthewfeickert added 9 commits April 27, 2024 13:54

Add uv to dev dependencies

fea7834

Ad uv to all pip install commands

02a891a

install quiet too

8eee37a

Add uv to pip uninstall

66811f2

install uv

3ed06bd

Require v0.1.12 for --system

96b6f0d

Specify 'install --system' to use CI provided Python

4ce9ccd

Use '--no-cache' as '--no-cache-dir' is for normal pip only

7117d0c

Use 'pyhf[extras] @ .' pattern over '.[extras]' pattern

d803c48

matthewfeickert added 8 commits April 27, 2024 13:54

Revert "Use 'pyhf[extras] @ .' pattern over '.[extras]' pattern"

3758cd4

This reverts commit 44fdb31.

Update uv lower bound

249241e

Use 'installer uv' option for build

9b7570a

quote local installs with extras

fd7b114

Update uv

3bd1ff1

fix: Fall back to pip for Python 3.8 as uv fails here

3eef2a7

Just give in and use 'uv pip' for everything

b8f2958

fix: Still need to call build from 'python -m' if not using pipx

d4f6e7c

matthewfeickert force-pushed the ci/use-uv-to-speed-up-installs branch from 1c2e4ef to d4f6e7c Compare April 27, 2024 18:55

matthewfeickert commented Apr 27, 2024

View reviewed changes

matthewfeickert marked this pull request as ready for review April 27, 2024 18:58

matthewfeickert requested review from kratsg and henryiii April 27, 2024 18:58

matthewfeickert commented Apr 27, 2024

View reviewed changes

matthewfeickert added 2 commits April 27, 2024 14:06

fix: uv pip uninstall needs 'system' but not 'yes'

aa7955d

ci: Adapt to uv's --upgrade rules to allow scipy without new numpy

49b8ed2

* uv pip install --upgrade is going to try to upgrade numpy along with scipy, so to avoid this, don't use --upgrade when installing scipy from the nightly wheel index.

matthewfeickert commented Apr 27, 2024

View reviewed changes

Flip index-url and extra-index-url as uv and pip differ

854906d

matthewfeickert merged commit 08755e4 into main Apr 30, 2024
26 checks passed

matthewfeickert deleted the ci/use-uv-to-speed-up-installs branch April 30, 2024 05:38

matthewfeickert mentioned this pull request May 1, 2024

ci: Use uv for all pip installs ssl-hep/ServiceX_frontend#365

Merged

matthewfeickert mentioned this pull request May 30, 2024

ci(backport): Use uv for all pip installs #2494

Merged

4 tasks

matthewfeickert removed the need-to-backport tmp label until can be backported to patch release branch label May 30, 2024

matthewfeickert added a commit that referenced this pull request May 30, 2024

ci(backport): Use uv for all pip installs (#2494)

902ffce

* Backport: - PR #2444 - PR #2475

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Use uv for all pip installs #2444

ci: Use uv for all pip installs #2444

matthewfeickert commented Feb 16, 2024 •

edited

Loading

matthewfeickert commented Feb 16, 2024 •

edited

Loading

matthewfeickert commented Feb 16, 2024

matthewfeickert commented Feb 18, 2024

matthewfeickert commented Feb 28, 2024 •

edited

Loading

matthewfeickert commented Feb 29, 2024

matthewfeickert commented Mar 28, 2024

matthewfeickert commented Apr 26, 2024

matthewfeickert commented Apr 27, 2024

codecov bot commented Apr 27, 2024 •

edited

Loading

matthewfeickert Apr 27, 2024

matthewfeickert Apr 27, 2024

matthewfeickert Apr 27, 2024

matthewfeickert Apr 27, 2024

matthewfeickert commented Apr 27, 2024

matthewfeickert commented Apr 27, 2024 •

edited

Loading

matthewfeickert commented Apr 30, 2024

		# uv wants to upgrade dependencies (numpy) to a dev release too, so don't --upgrade
		uv pip install --system --index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple scipy

ci: Use uv for all pip installs #2444

ci: Use uv for all pip installs #2444

Conversation

matthewfeickert commented Feb 16, 2024 • edited Loading

Description

Checklist Before Requesting Reviewer

Before Merging

matthewfeickert commented Feb 16, 2024 • edited Loading

matthewfeickert commented Feb 16, 2024

matthewfeickert commented Feb 18, 2024

matthewfeickert commented Feb 28, 2024 • edited Loading

matthewfeickert commented Feb 29, 2024

matthewfeickert commented Mar 28, 2024

matthewfeickert commented Apr 26, 2024

matthewfeickert commented Apr 27, 2024

codecov bot commented Apr 27, 2024 • edited Loading

Codecov Report

matthewfeickert Apr 27, 2024

Choose a reason for hiding this comment

matthewfeickert Apr 27, 2024

Choose a reason for hiding this comment

matthewfeickert Apr 27, 2024

Choose a reason for hiding this comment

matthewfeickert Apr 27, 2024

Choose a reason for hiding this comment

matthewfeickert commented Apr 27, 2024

matthewfeickert commented Apr 27, 2024 • edited Loading

matthewfeickert commented Apr 30, 2024

matthewfeickert commented Feb 16, 2024 •

edited

Loading

matthewfeickert commented Feb 16, 2024 •

edited

Loading

matthewfeickert commented Feb 28, 2024 •

edited

Loading

codecov bot commented Apr 27, 2024 •

edited

Loading

matthewfeickert commented Apr 27, 2024 •

edited

Loading