Support installing packages by pip install

[suzuki-shunsuke]

Feature Overview

Support installing packages by pip install.

Why is the feature needed?

To support packages written in Python.

Workaround

Install tools with other tools.

1. pyenv and virtualenv
2. homebrew

Example Code

registry.yaml

packages:
  - name: pypi.org/pre-commit
    type: pip
    pip_name: pre-commit
    files:
      - name: pre-commit

Reference

• Add pre-commit / pre-commit-terraform aqua-registry#14024

[suzuki-shunsuke]

pip install supports --root option.

  --user                      Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation
                              for site.USER_BASE for full details.)
  --root <dir>                Install everything relative to this alternate root directory.

$ pip --version
pip 21.1.2 from /Users/shunsuke-suzuki/.pyenv/versions/3.9.5/lib/python3.9/site-packages/pip (python 3.9)

$ pip install --help

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:
  
  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.
  
  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  -r, --requirement <file>    Install from the given requirements file. This option can be used multiple times.
  -c, --constraint <file>     Constrain versions using the given constraints file. This option can be used multiple times.
  --no-deps                   Don't install package dependencies.
  --pre                       Include pre-release and development versions. By default, pip only finds stable versions.
  -e, --editable <path/url>   Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
  -t, --target <dir>          Install packages into <dir>. By default this will not replace existing files/folders in <dir>. Use --upgrade to replace existing packages in <dir>
                              with new versions.
  --platform <platform>       Only use wheels compatible with <platform>. Defaults to the platform of the running system. Use this option multiple times to specify multiple
                              platforms supported by the target interpreter.
  --python-version <python_version>
                              The Python interpreter version to use for wheel and "Requires-Python" compatibility checks. Defaults to a version derived from the running
                              interpreter. The version can be specified using up to three dot-separated integers (e.g. "3" for 3.0.0, "3.7" for 3.7.0, or "3.7.3"). A major-
                              minor version can also be given as a string without dots (e.g. "37" for 3.7.0).
  --implementation <implementation>
                              Only use wheels compatible with Python implementation <implementation>, e.g. 'pp', 'jy', 'cp',  or 'ip'. If not specified, then the current
                              interpreter implementation is used.  Use 'py' to force implementation-agnostic wheels.
  --abi <abi>                 Only use wheels compatible with Python abi <abi>, e.g. 'pypy_41'. If not specified, then the current interpreter abi tag is used. Use this option
                              multiple times to specify multiple abis supported by the target interpreter. Generally you will need to specify --implementation, --platform, and
                              --python-version when using this option.
  --user                      Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation
                              for site.USER_BASE for full details.)
  --root <dir>                Install everything relative to this alternate root directory.
  --prefix <dir>              Installation prefix where lib, bin and other top-level folders are placed
  --src <dir>                 Directory to check out editable projects into. The default in a virtualenv is "<venv path>/src". The default for global installs is "<current
                              dir>/src".
  -U, --upgrade               Upgrade all specified packages to the newest available version. The handling of dependencies depends on the upgrade-strategy used.
  --upgrade-strategy <upgrade_strategy>
                              Determines how dependency upgrading should be handled [default: only-if-needed]. "eager" - dependencies are upgraded regardless of whether the
                              currently installed version satisfies the requirements of the upgraded package(s). "only-if-needed" -  are upgraded only when they do not satisfy
                              the requirements of the upgraded package(s).
  --force-reinstall           Reinstall all packages even if they are already up-to-date.
  -I, --ignore-installed      Ignore the installed packages, overwriting them. This can break your system if the existing package is of a different version or was installed
                              with a different package manager!
  --ignore-requires-python    Ignore the Requires-Python information.
  --no-build-isolation        Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is
                              used.
  --use-pep517                Use PEP 517 for building source distributions (use --no-use-pep517 to force legacy behaviour).
  --install-option <options>  Extra arguments to be supplied to the setup.py install command (use like --install-option="--install-scripts=/usr/local/bin"). Use multiple
                              --install-option options to pass multiple options to setup.py install. If you are using an option with a directory path, be sure to use absolute
                              path.
  --global-option <options>   Extra global options to be supplied to the setup.py call before the install or bdist_wheel command.
  --compile                   Compile Python source files to bytecode
  --no-compile                Do not compile Python source files to bytecode
  --no-warn-script-location   Do not warn when installing scripts outside PATH
  --no-warn-conflicts         Do not warn about broken dependencies
  --no-binary <format_control>
                              Do not use binary packages. Can be supplied multiple times, and each time adds to the existing value. Accepts either ":all:" to disable all binary
                              packages, ":none:" to empty the set (notice the colons), or one or more package names with commas between them (no colons). Note that some
                              packages are tricky to compile and may fail to install when this option is used on them.
  --only-binary <format_control>
                              Do not use source packages. Can be supplied multiple times, and each time adds to the existing value. Accepts either ":all:" to disable all source
                              packages, ":none:" to empty the set, or one or more package names with commas between them. Packages without binary distributions will fail to
                              install when this option is used on them.
  --prefer-binary             Prefer older binary packages over newer source packages.
  --require-hashes            Require a hash to check each requirement against, for repeatable installs. This option is implied when any package in a requirements file has a
                              --hash option.
  --progress-bar <progress_bar>
                              Specify type of progress to be displayed [off|on|ascii|pretty|emoji] (default: on)
  --no-clean                  Don't clean up build directories.

Package Index Options:
  -i, --index-url <url>       Base URL of the Python Package Index (default https://pypi.org/simple). This should point to a repository compliant with PEP 503 (the simple
                              repository API) or a local directory laid out in the same format.
  --extra-index-url <url>     Extra URLs of package indexes to use in addition to --index-url. Should follow the same rules as --index-url.
  --no-index                  Ignore package index (only looking at --find-links URLs instead).
  -f, --find-links <url>      If a URL or path to an html file, then parse for links to archives such as sdist (.tar.gz) or wheel (.whl) files. If a local path or file:// URL
                              that's a directory,  then look for archives in the directory listing. Links to VCS project URLs are not supported.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --no-input                  Disable prompting for input.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to PEM-encoded CA certificate bundle. If provided, overrides the default. See 'SSL Certificate Verification' in pip documentation for more
                              information.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.
  --no-color                  Suppress colored output.
  --no-python-version-warning
                              Silence deprecation warnings for upcoming unsupported Pythons.
  --use-feature <feature>     Enable new functionality, that may be backward incompatible.
  --use-deprecated <feature>  Enable deprecated functionality, that will be removed in the future.

e.g.

 $ pip install --root pip-root pre-commit

$ ls pip-root/Users/shunsuke-suzuki/.local/bin 
identify-cli  nodeenv  pre-commit  virtualenv

[suzuki-shunsuke]

Directory structure

<AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<package name>/<version>/bin/<command>

AQUA_ROOT_DIR/pkgs/
  pip/
    pypi.org/
      pre-commit/
        3.3.3/
          bin/
            pre-commit
          lib/

[suzuki-shunsuke]

Renovate

Pypi datasource

https://docs.renovatebot.com/modules/datasource/pypi/

[suzuki-shunsuke]

Package name

Packages in https://pypi.org/ should start with pypi.org/ so that aqua-renovate-config can get versions from pypi datasource and aqua g gets the version from https://pypi.org/ if API exists.

In the repository aquaproj/aqua-registry, pip packages should be located in pkgs/pypi.org similar to crates.io.

[suzuki-shunsuke]

pip install supports --root option.

--target is more useful than --root.

$ pip install -t target pre-commit
$ ls target/bin 
identify-cli  nodeenv  pre-commit  virtualenv

[suzuki-shunsuke]

• feat: support a new package type pypi #2142

[suzuki-shunsuke]

⚠️ It failed to execute pre-commit installed by pip install --target command

Hmm. I installed pre-commit by pip install --target target pre-commit, but it failed to execute pre-commit command.

$ python --version
Python 3.10.5

$ pip --version
pip 22.2 from /Users/shunsukesuzuki/.pyenv/versions/3.10.5/lib/python3.10/site-packages/pip (python 3.10)

$ pip install --target target pre-commit

$ ./target/bin/pre-commit --help
Traceback (most recent call last):
  File "/Users/shunsukesuzuki/Documents/test/pip/./target/bin/pre-commit", line 5, in <module>
    from pre_commit.main import main
ModuleNotFoundError: No module named 'pre_commit'

---

• No module named pre_commit pre-commit/pre-commit#2153
  • python3Packages.pre-commit: fix hook-tmpl NixOS/nixpkgs#148353

target/bin/pre-commit

#!/Users/shunsukesuzuki/.pyenv/versions/3.10.5/bin/python3.10
# -*- coding: utf-8 -*-
import re
import sys
from pre_commit.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

pre_commit exists in target.

The environment variable PYTHONPATH is needed?

• https://dev.classmethod.jp/articles/pythonpath-be-careful/

Succeeded.

$ env PYTHONPATH=$PWD/target ./target/bin/pre-commit --help
usage: pre-commit [-h] [-V]
                  {autoupdate,clean,gc,init-templatedir,install,install-hooks,migrate-config,run,sample-config,try-repo,uninstall,validate-config,validate-manifest,help,hook-impl}
                  ...

positional arguments:
  {autoupdate,clean,gc,init-templatedir,install,install-hooks,migrate-config,run,sample-config,try-repo,uninstall,validate-config,validate-manifest,help,hook-impl}
    autoupdate          Auto-update pre-commit config to the latest repos' versions.
    clean               Clean out pre-commit files.
    gc                  Clean unused cached repos.
    init-templatedir    Install hook script in a directory intended for use with `git config init.templateDir`.
    install             Install the pre-commit script.
    install-hooks       Install hook environments for all environments in the config file. You may find `pre-commit install
                        --install-hooks` more useful.
    migrate-config      Migrate list configuration to new map configuration.
    run                 Run hooks.
    sample-config       Produce a sample .pre-commit-config.yaml file
    try-repo            Try the hooks in a repository, useful for developing new hooks.
    uninstall           Uninstall the pre-commit script.
    validate-config     Validate .pre-commit-config.yaml files
    validate-manifest   Validate .pre-commit-hooks.yaml files
    help                Show help for a specific command.

options:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit

I'm not sure if this is a correct solution.

[suzuki-shunsuke]

⚠️ the behaviour of pip command depends on Python and pip version

BTW, the behaviour of pip command depends on Python and pip version.
This means pip package strongly depends on the environment and the reproductivity is low.

We have no plan (probably we won't) to support managing isolated Python environment like pyenv and virtualenv.

If the project uses tools such as pyenv and virtualenv, I guess it's better to manage pip packages without aqua.

[suzuki-shunsuke]

Get the list of available versions

https://stackoverflow.com/a/27239645/6364492

curl "https://pypi.org/pypi/pre-commit/json" | jq -r '.releases | keys[]'

[suzuki-shunsuke]

Document (Draft)

Support installing packages by pip install

#2128 #2142 aqua >= v2.11.0

Feature Overview

Support installing packages by pip install.

Why is the feature needed?

To support packages written in Python.

Requirements

• Python https://www.python.org/
• pip

aqua doesn't install these requirements automatically. Please install them yourself.

This feature depends on versions of Python and pip.
We verified this feature with the following versions.

• Python 3.10.5
• pip 22.2

$ python --version
$ pip --version

Getting Started

Let's install pre-commit by aqua.

aqua init

aqua g -i pypi.org/pre-commit

You can also select the version by -s option.

aqua g -i -s pypi.org/pre-commit

aqua i -l
aqua which pre-commit
pre-commit --version

aqua-renovate-config

https://github.com/aquaproj/aqua-renovate-config

From 1.8.0, aqua-renovate-config supports updating pypi packages.

⚠️ Note that package names must be pypi.org/<pypi package name>.

e.g. pypi.org/pre-commit.

How to add new pypi packages to Standard Registry

Please send a pull request to https://github.com/aquaproj/aqua-registry .
Package names must be pypi.org/<pypi package name>.

e.g. pypi.org/pre-commit.

How does it work?

⚠️ This includes details of the internal implementation, which may be changed without notification.
Please skip this section if you're not interested in the detail. You can use pypi packages even if you don't know this.

pypi packages are installed in <AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version>,
and executable files are installed in <AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version>/bin/<command>.

aqua internally runs python -m pip install commands.

python -m pip install --target  "<AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version>" "<pypi package>==<version>"

aqua g -s gets the list of pypi package versions from the endpoint https://pypi.org/pypi/<pypi package name>/json.

<AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version> is added to the environment variable PYTHONPATH when pypi packages are executed.

Registry

e.g.

packages:
  - type: pypi
    pypi_name: pre-commit

type must be pypi. pypi_name is required. Other fields are optional.
The above setting is equivalent to the following setting.

packages:
  - name: pypi.org/pre-commit
    type: pypi
    pypi_name: pre-commit
    files:
      - name: pre-commit

pypi packages don't support the following fields.

• format, format_overrides
• checksum
• slsa_provenance
• cosign

[suzuki-shunsuke]

TODO

• aqua-registry: Add pre-commit feat: add some pypi packages aqua-registry#14185
• aqua-renovate-config: Support pip packages feat: support pypi packages aqua-renovate-config#420
• aquaproj.github.io: Add the document: add pypi package aquaproj.github.io#495

[suzuki-shunsuke]

I decided to rename the package type name pip to pypi.

[suzuki-shunsuke]

Hi @Gowiem ,

I published a pre-release version v2.11.0-3 v2.11.0-4.
Could you try this? If you have any feedback, please let me know.

Document: #2128 (comment)

aqua update-aqua v2.11.0-4
aqua -v
mkdir workspace
cd workspace
aqua init
vi aqua.yaml

aqua.yaml

---
registries:
- type: standard
  ref: cd6610dcdee69814d8f72e06e9557d42abad4298 # https://github.com/aquaproj/aqua-registry/pull/14185
packages:

Install pre-commit.

aqua g -i pypi.org/pre-commit
aqua i -l

Please check if pre-commit installed by aqua is executed.

command -v pre-commit

If pre-commit installed by other than aqua is executed, you have some options.

1. Uninstall it once
2. Fix the order of the environment variable PATH
3. Run pre-commit via aqua exec command. aqua exec -- pre-commit

Then please confirm that pre-commit is available.

pre-commit --help

[suzuki-shunsuke]

📝 What is the effect of using python -m pip instead of just pip?

[suzuki-shunsuke]

⚠️ Some pypi packages don't work well

I'm trying some tools that can be installed by pip install command.

https://github.com/topics/cli?l=python

Unfortunately, some tools don't work well.

textual

https://github.com/Textualize/textual

ModuleNotFoundError: No module named 'aiohttp'

$ textual console
Traceback (most recent call last):
  File "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/pypi/pypi.org/textual/0.9.1/bin/textual", line 8, in <module>
    sys.exit(run())
  File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/pypi/pypi.org/textual/0.9.1/textual/cli/cli.py", line 23, in console
    from textual.devtools.server import _run_devtools
  File "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/pypi/pypi.org/textual/0.9.1/textual/devtools/server.py", line 4, in <module>
    from aiohttp.web import run_app
ModuleNotFoundError: No module named 'aiohttp'

[suzuki-shunsuke]

I'm adding some pypi packages.

• feat: add some pypi packages aqua-registry#14185

[suzuki-shunsuke]

⚠️ Should we really introduce this feature?

I implemented this feature, but now I hesitate to introduce it because other tools such as pyenv, virtualenv, and pipx seem to be better than aqua.
I'm not familiar with Python, but Python has already several version management tools.
They support managing Python and dependencies in isolated environments, and Renovate supports updating them.

I tried to install some pypi packages by aqua, but some of them don't work well because they depend on other libraries. #2128 (comment)
I don't want to handle these troubles.

aqua has various advantages compared with other tools such as Homebrew and asdf,
but in case of pypi packages, aqua doesn't have so many advantages compared with other tools such as pyenv, virtualenv, and pipx.

pypi packages can be installed by pip easily, but to make them available by aqua, we need to add them to Registries.
It's a little bothersome.

[dudicoco]

@suzuki-shunsuke note that pipx is not suitable for project specific tool versions, see pypa/pipx#1031

Perhaps aqua can provide a wrapper for both pipx and npx to install pip and node packages declaratively?

Aqua could run pipx under the hood and then just add the created binary to its path, for example /Users/xxxx/.local/pipx/venvs/copier/bin/copier

[dudicoco]

So rethinking this, i'm not sure pipx is a good solution since I believe it can produce non-deterministic builds: pypa/pipx#1050

I believe that in order to make aqua work with pip packages and make them reproducible, aqua would need to do the following:

1. Use pyenv to set a consistent python version
2. Use an automatically generated lock file for each version of a pip package - this can be part of the workflow which updates the registry.
   Options for generation the lock file:
   • pip freeze
   • https://github.com/pypa/pipfile
   • https://github.com/jazzband/pip-tools/
3. Install the pip package using virtualenv within the aqua cache directory
4. Add the executable to the path

[suzuki-shunsuke]

I see. Thank you. I misunderstood pipx.

How about direnv?

https://github.com/direnv/direnv/wiki/Python

.envrc

layout python

requirements.txt

pre-commit==3.3.3

direnv allow
pip install -r requirements.txt

$ command -V pre-commit
pre-commit is /Users/shunsukesuzuki/Documents/test/foo/.direnv/python-3.10/bin/pre-commit

$ pre-commit -V
pre-commit 3.3.3

To specify Python version, direnv + pyenv is useful.

https://github.com/direnv/direnv/wiki/Python#pyenv

[dudicoco]

Yeah i've also considered direnv as an alternative, there are a few issues though:

1. The user must manage the requirements.txt file, also if you want a few packages in different versions you might have to manage multiple requirements.txt files due to possible conflicting versions.
2. I'm not sure if direnv sets a shared cache so if i've installed package x with version 1.0.0 in repo A I might not be able to use the cached package in repo B as well. Even if this is possible, the repos might have different dependencies in the requirements.txt file which will make them incompatible.
3. Just like in aqua, we do not want the user to run pip install manually, we would like it to run automatically and i'm not sure if direnv can do that