Recommend and use uv instead of pipx in a few remaining places (

apache#43866) The ``pipx`` has still been used and recommended in a few places in our documentation and in a few scripts/github actions. We also installed pipx in the container image to be able to run mmsql-cli, but this is not needed any more as mssql is not supported as metadata backend. This PR updates the remaining parts.
astronomer · Nov 11, 2024 · c89ab99 · c89ab99
1 parent cf3f9c7
commit c89ab99
Show file tree

Hide file tree

Showing 23 changed files with 268 additions and 317 deletions.
diff --git a/.github/workflows/basic-tests.yml b/.github/workflows/basic-tests.yml
@@ -187,9 +187,8 @@ jobs:
         working-directory: ./airflow-client-python
       - name: Install hatch
         run: |
-          python -m pip install --upgrade pipx
-          pipx ensurepath
-          pipx install hatch
+          python -m pip install --upgrade uv
+          uv tool install hatch
       - name: Run tests
         run: hatch run run-coverage
         env:

diff --git a/.github/workflows/check-providers.yml b/.github/workflows/check-providers.yml
@@ -102,8 +102,8 @@ jobs:
         run: breeze release-management prepare-airflow-package --version-suffix-for-pypi dev0
       - name: "Verify wheel packages with twine"
         run: |
-          pipx uninstall twine || true
-          pipx install twine && twine check dist/*.whl
+          uv tool uninstall twine || true
+          uv tool install twine && twine check dist/*.whl
       - name: "Test providers issue generation automatically"
         run: >
           breeze release-management generate-issue-content-providers
@@ -170,8 +170,8 @@ jobs:
           --version-suffix-for-pypi dev0 --package-format sdist
       - name: "Verify sdist packages with twine"
         run: |
-          pipx uninstall twine || true
-          pipx install twine && twine check dist/*.tar.gz
+          uv tool uninstall twine || true
+          uv tool install twine && twine check dist/*.tar.gz
       - name: "Generate source constraints from CI image"
         shell: bash
         run: >

diff --git a/.github/workflows/news-fragment.yml b/.github/workflows/news-fragment.yml
@@ -38,7 +38,7 @@ jobs:
 
       - name: Check news fragment
         run: >
-          pipx run towncrier check
+          uv tool run towncrier check
           --dir .
           --config newsfragments/config.toml
           --compare-with origin/${{ github.base_ref }}

diff --git a/.github/workflows/task-sdk-tests.yml b/.github/workflows/task-sdk-tests.yml
@@ -77,8 +77,8 @@ jobs:
           breeze release-management prepare-task-sdk-package --package-format wheel
       - name: "Verify wheel packages with twine"
         run: |
-          pipx uninstall twine || true
-          pipx install twine && twine check dist/*.whl
+          uv tool uninstall twine || true
+          uv tool install twine && twine check dist/*.whl
       - name: >
           Run unit tests for Airflow Task SDK:Python ${{ matrix.python-version }}
         run: >

diff --git a/Dockerfile.ci b/Dockerfile.ci
@@ -634,35 +634,6 @@ function common::import_trusted_gpg() {
 }
 EOF
 
-# The content below is automatically copied from scripts/docker/install_pipx_tools.sh
-COPY <<"EOF" /install_pipx_tools.sh
-#!/usr/bin/env bash
-. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
-
-function install_pipx_tools() {
-    echo
-    echo "${COLOR_BLUE}Installing pipx tools${COLOR_RESET}"
-    echo
-    # Make sure PIPX is installed in latest version
-    ${PACKAGING_TOOL_CMD} install ${EXTRA_INSTALL_FLAGS} --upgrade "pipx>=1.2.1"
-    if [[ $(uname -m) != "aarch64" ]]; then
-        # Do not install mssql-cli for ARM
-        # Install all the tools we need available in command line but without impacting the current environment
-        pipx install mssql-cli
-
-        # Unfortunately mssql-cli installed by `pipx` does not work out of the box because it uses
-        # its own execution bash script which is not compliant with the auto-activation of
-        # pipx venvs - we need to manually patch Python executable in the script to fix it: ¯\_(ツ)_/¯
-        sed "s/python /\/root\/\.local\/pipx\/venvs\/mssql-cli\/bin\/python /" -i /root/.local/bin/mssql-cli
-    fi
-}
-
-common::get_colors
-common::get_packaging_tool
-
-install_pipx_tools
-EOF
-
 # The content below is automatically copied from scripts/docker/install_airflow.sh
 COPY <<"EOF" /install_airflow.sh
 #!/usr/bin/env bash
@@ -1381,13 +1352,10 @@ ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
 # The PATH is needed for PIPX to find the tools installed
 ENV PATH="/root/.local/bin:${PATH}"
 
-COPY --from=scripts install_pipx_tools.sh /scripts/docker/
-
 # Install useful command line tools in their own virtualenv so that they do not clash with
 # dependencies installed in Airflow also reinstall PIP and UV to make sure they are installed
 # in the version specified above
-RUN bash /scripts/docker/install_packaging_tools.sh; \
-    bash /scripts/docker/install_pipx_tools.sh
+RUN bash /scripts/docker/install_packaging_tools.sh
 
 # Airflow sources change frequently but dependency configuration won't change that often
 # We copy pyproject.toml and other files needed to perform setup of dependencies

diff --git a/INSTALL b/INSTALL
@@ -47,7 +47,8 @@ you get all sources in one place. This is the most convenient way to develop Air
 Otherwise, you have to install Airflow and Providers separately from sources in the same environment, which
 is not as convenient.
 
-## Creating virtualenv
+Creating virtualenv
+-------------------
 
 Airflow pulls in quite a lot of dependencies to connect to other services. You generally want to
 test or run Airflow from a virtualenv to ensure those dependencies are separated from your system-wide versions. Using system-installed Python installation is strongly discouraged as the versions of Python
@@ -59,7 +60,8 @@ Once you have a suitable Python version installed, you can create a virtualenv a
     python3 -m venv PATH_TO_YOUR_VENV
     source PATH_TO_YOUR_VENV/bin/activate
 
-## Installing Airflow locally
+Installing Airflow locally
+--------------------------
 
 Installing Airflow locally can be done using pip - note that this will install "development" version of
 Airflow, where all providers are installed from local sources (if available), not from `pypi`.
@@ -79,7 +81,6 @@ this one contains the minimum set of tools and dependencies needed to run unit t
 
     pip install -e ".[devel]"
 
-
 You can also install optional packages that are needed to run certain tests. In case of local installation
 for example, you can install all prerequisites for Google provider, tests, and
 all Hadoop providers with this command:
@@ -97,122 +98,78 @@ Additionally when you want to develop providers you need to install providers co
 
     pip install -e "./providers"
 
-# Using Hatch to manage your Python, virtualenvs, and build packages
-
-Airflow uses [hatch](https://hatch.pypa.io/) as a build and development tool. It is one of the popular
-build tools and environment managers for Python, maintained by the Python Packaging Authority.
-It is an optional tool that is only really needed when you want to build packages from sources, but
-it is also very convenient to manage your Python versions and virtualenvs.
-
-Airflow project contains some pre-defined virtualenv definitions in `pyproject.toml` that can be
-easily used by Hatch to create your local venvs. This is not necessary for you to develop and test
-Airflow, but it is a convenient way to manage your local Python versions and virtualenvs.
-
-Installing Hatch
-----------------
-
-You can install Hatch using various other ways (including Gui installers).
-
-Example using `pipx`:
-
-    pipx install hatch
-
-We recommend using `pipx` as you can manage installed Python apps easily and later use it
-to upgrade `hatch` easily as needed with:
+Using ``uv`` to manage your Python, virtualenvs, and install airflow for development
+====================================================================================
 
-    pipx upgrade hatch
+While you can manually install airflow locally from sources, Airflow committers recommend using
+[uv](https://docs.astral.sh/uv/) as a build and development tool. It is a modern,
+recently introduced popular packaging front-end tool and environment managers for Python.
+It is an optional tool that is only really needed when you want to build packages from sources, you can use
+many other packaging frontends (for example ``hatch``) but ``uv`` is very fast and convenient to manage
+also your Python versions and virtualenvs. Also we use ``
 
-## Using Hatch to manage your Python versions
-
-You can also use Hatch to install and manage airflow virtualenvs and development
-environments. For example, you can install Python 3.10 with this command:
-
-    hatch python install 3.10
-
-or install all Python versions that are used in Airflow:
-
-    hatch python install all
-
-## Using Hatch to manage your virtualenvs
-
-Airflow has some pre-defined virtualenvs that you can use to develop and test airflow.
-You can see the list of available envs with:
-
-    hatch env show
-
-This is what it shows currently:
+Installing ``uv``
+-----------------
 
-┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
-┃ Name        ┃ Type    ┃ Description                                                   ┃
-┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
-│ default     │ virtual │ Default environment with Python 3.9 for maximum compatibility │
-├─────────────┼─────────┼───────────────────────────────────────────────────────────────┤
-│ airflow-39  │ virtual │ Environment with Python 3.9. No devel installed.              │
-├─────────────┼─────────┼───────────────────────────────────────────────────────────────┤
-│ airflow-310 │ virtual │ Environment with Python 3.10. No devel installed.             │
-├─────────────┼─────────┼───────────────────────────────────────────────────────────────┤
-│ airflow-311 │ virtual │ Environment with Python 3.11. No devel installed              │
-├─────────────┼─────────┼───────────────────────────────────────────────────────────────┤
-│ airflow-312 │ virtual │ Environment with Python 3.12. No devel installed              │
-└─────────────┴─────────┴───────────────────────────────────────────────────────────────┘
+You can install uv following [the instructions](https://docs.astral.sh/uv/getting-started/installation/ on
 
-The default env (if you have not used one explicitly) is `default` and it is a Python 3.9
-virtualenv for maximum compatibility with `devel` extra installed - this devel extra contains the minimum set
-of dependencies and tools that should be used during unit testing of core Airflow and running all `airflow`
-CLI commands - without support for providers or databases.
+Using ``uv`` to manage your virtualenvs
+---------------------------------------
 
-The other environments are just bare-bones Python virtualenvs with Airflow core requirements only,
-without any extras installed and without any tools. They are much faster to create than the default
-environment, and you can manually install either appropriate extras or tools that you need for
-testing or development.
+You can create a virtualenv with ``uv`` using the following command:
 
-    hatch env create
+    uv venv create
 
-You can create specific environments by using them in create command:
+You can sync to latest versions of airflow dependencies using:
 
-    hatch env create airflow-310
+    uv sync
 
-You can install extras in the environment by running pip command:
+And if you want to use some extras (for example because you want to develop providers) you can add
+extras to the command:
 
-    hatch -e airflow-310 run -- pip install -e ".[devel,google]"
+    uv sync --extra devel
 
-And you can enter the environment by running a shell of your choice (for example, zsh) where you
-can run any commands
+You can also synchronize all extras:
 
-    hatch -e airflow-310 shell
+    uv sync --all-extras
 
-Once you are in the environment (indicated usually by an updated prompt), you can just install
-the extra dependencies you need:
+Building airflow packages with Hatch
+====================================
 
-    [~/airflow] [airflow-310] pip install -e ".[devel,google]"
+While building packages will work with any compliant packaging front-end tool, for reproducibility, we
+recommend using ``hatch``. It is a modern, fast, and convenient tool to build packages from sources managed
+by the Python Packaging Authority. It is also used by Airflow to build packages in CI/CD as well as by
+release managers to build locally packages for verification of reproducibility of the build.
 
-You can exit the environment by just exiting the shell.
+Installing ``hatch``
+--------------------
 
-You can also see where Hatch created the virtualenvs and use it in your IDE or activate it manually:
+More information about hatch can be found in https://hatch.pypa.io/
 
-    hatch env find airflow-310
+We recommend to install ``hatch`` using ```uv tool`` command which will make hatch available as a CLI
+command globally:
 
-You will get a path similar to the following:
+    uv tool install hatch
 
-    /Users/jarek/Library/Application Support/hatch/env/virtual/apache-airflow/TReRdyYt/apache-airflow
+You can still install ``hatch`` using ``pipx`` if you prefer:
 
-Then you will find `python` binary and `activate` script in the `bin` sub-folder of this directory, and
-you can configure your IDE to use this python virtualenv if you want to use that environment in your IDE.
+    pipx install hatch
 
-You can also set the default environment name by the HATCH_ENV environment variable.
 
-You can clean the environment by running the following:
+It's important to keep your hatch up to date. You can do this by running:
 
-    hatch env prune
+    uv tool upgrade hatch
 
-More information about hatch can be found in https://hatch.pypa.io/1.9/environment/
 
-## Using Hatch to build your packages
+Using Hatch to build your packages
+----------------------------------
 
 You can use Hatch to build installable packages from the Airflow sources. Such package will
-include all metadata configured in `pyproject.toml` and will be installable with pip.
+include all metadata configured in `pyproject.toml` and will be installable with ``pip`` and and any other
+PEP-compliant packaging front-end.
 
-The packages will have pre-installed dependencies for providers that are available when Airflow is installed from PyPI. Both `wheel` and `sdist` packages are built by default.
+The packages will have pre-installed dependencies for providers that are available when Airflow is i
+onstalled from PyPI. Both `wheel` and `sdist` packages are built by default.
 
     hatch build
 
@@ -221,7 +178,8 @@ You can also build only `wheel` or `sdist` packages:
     hatch build -t wheel
     hatch build -t sdist
 
-## Installing recommended version of dependencies
+Installing recommended version of dependencies
+==============================================
 
 Whatever virtualenv solution you use, when you want to make sure you are using the same
 version of dependencies as in main, you can install the recommended version of the dependencies by using