Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+14 −2
Open

feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c9d687d
feat: relax dependencies in pyproject.toml and add a requirements.txt
mistercrunch Nov 26, 2024
db860b3
pip install in GHA workflows
mistercrunch Nov 26, 2024
a893a3c
add uv to local-requirements.txt
mistercrunch Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/publish_docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,6 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install -r local-requirements.txt
pip install -r requirements.txt
pip install -e .
- run: ./utils/docker/publish_docker.sh stable
1 change: 1 addition & 0 deletions .github/workflows/test_docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install -r local-requirements.txt
pip install -r requirements.txt
pip install -e .
- name: Build Docker image
run: bash utils/docker/build.sh --amd64 ${{ matrix.docker-image-variant }} playwright-python:localbuild-${{ matrix.docker-image-variant }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ htmlcov/
.coverage*
.DS_Store
.vscode/
.venv
.eggs
_repo_version.py
coverage.xml
Expand Down
1 change: 1 addition & 0 deletions local-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ service_identity==24.2.0
twisted==24.11.0
types-pyOpenSSL==24.1.0.20240722
types-requests==2.32.0.20241016
uv==0.5.4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to not introduce a new package for this. What do we loose if we don't use it? afaik its only used to update the command in the requirements.txt?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed something to compile the requirements, it's either this uv or pip-tools or maybe poetry (?). Unless conda has options to do this but I think conda is slowing down, and uv is the future of package management in python, replacing/improving-on/consolidating pip, virtualenv, pyenv, conda, ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to 'compile' the requirements? Maybe I miss something. I was thinking about updating them manually (or dependabot does it) - since these are only two - we should be able to deal with it? They get automatically installed using pip install -r requirements.txt anyways?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here https://chatgpt.com/share/6764e629-4344-8010-8465-c1efe5e1813e

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see - so both dependencies: pyee and greenlet have no dependencies on itself - this is where I got confused. uv helps us to pin the transitive dependencies (where there aren't any yet). In the future there might be some. Is this correct?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's right, generally good to have pin deps for CI otherwise a new lib comes out some day and your CI starts failing.

4 changes: 2 additions & 2 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ license = {text = "Apache-2.0"}
dynamic = ["version"]
requires-python = ">=3.9"
dependencies = [
"greenlet==3.1.1",
"pyee==12.1.1",
"pyee>=12.1.1",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure to relax the version in the meta.yaml as well for our conda customers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mmmmh yeah it's not great for this to not be DRY. I'm not familiar enough with conda to know the best practices on that side of the fence. Here's a related GPT thread discussing the different options/pros/cons -> https://chatgpt.com/share/67476ce2-0288-8010-bc05-e20ff72d15c0

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the fact conda doesn't seem to be compatible with dependabot. Is it a requirement to support conda? Seems it's fading https://theregister.com/2024/08/08/anaconda_puts_the_squeeze_on/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'd prefer to not drop it for now. Updating it manually on conda-side is good enough for us.

"greenlet>=3.1.1"
]
classifiers = [
"Topic :: Software Development :: Testing",
Expand Down
8 changes: 8 additions & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml -o requirements.txt
greenlet==3.1.1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm worried about here is that when e.g. greenlet 3.1.2 comes out, which includes e.g. a breaking change or a bug for some reason, our CI would still run with 3.1.1 and not install the bad version.

Why is the requirements.txt file needed at all in this case? To provide reproducible builds?

Copy link
Author

@mistercrunch mistercrunch Nov 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

our CI would still run with 3.1.1
Right, this is currently the case. But now you can enable @dependabot, who will auto-submit PRs periodically as new versions of dependencies come out. Those PRs will run your CI, so you can confirm that bumping is safe.

To provide reproducible builds?
Yes, so you get reproducible builds, and you let people who use your library choose their version within the specified range in pyproject.toml . Say in our case for Apache Superset, we have multiple other dependencies that rely on greenlet, each one specifying a know version range.

About the topic of maintaining the range, clearly you'll be running-CI only against what's in the pinned file. If/when people report something like "playwright doesn't work on top of greenlet>=5.0.0", as a maintainer you typically want to go and edit the known working range in pyproject.toml. Meaning we'd change the rule to greenlet>=3.1.1,<5.0.0 then, and dependabot would respect that rule, only submitting PRs to bump within the specified range.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah lets do something like: greenlet>=3.1.1,<4.0.0. For pyee something similar, so its only 12.x for now and we can increase if there is a new version at some point. Just to make sure 13.x does not break us. Typing extensions is backed by the Python team, I think usually you don't depend on a max version there..

# via playwright (pyproject.toml)
pyee==12.1.1
# via playwright (pyproject.toml)
typing-extensions==4.12.2
# via pyee