poetry install does not work for cvxpy dependency when updating poetry version from 1.1.15 to 1.2.1 on a mac m1

[adelaharpe]

• Poetry version: 1.2.1
• Python version: pyenv local 3.10.0
• OS version and name: Mac OS 12.6 (Apple M1 Pro)
• pyproject.toml: gist

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I have encountered an issue that I could not resolve. I created a poetry-demo folder following : https://python-poetry.org/docs/basic-usage/ and updated the pyproject.toml file as in the gist link. Then when running poetry install the command fails (see below for the error message).

When I do the exact same process with the poetry version 1.1.15 installed it works fine. Or when I do the same with the same poetry version (1.2.1) on ubuntu it works as well.

The workflow to produce my error.

• go in home folder
• run pyenv local 3.10.0
• run poetry new poetry-demo
• update the pyproject.toml as in the gist link
• in the poetry-demo folder run poetry install

Here is the error message I got:

`poetry install -vvv
Loading configuration file /Users/axeldelaharpe/Library/Preferences/pypoetry/config.toml
Using virtualenv: /Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 5 updates, 0 removals, 2 skipped

• Updating qdldl (0.1.5.post2 /Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz -> 0.1.5.post2): Pending...
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/qdldl/0.1.5.post2/json HTTP/1.1" 304 0
Skipping wheel qdldl-0.1.5.post2-cp310-cp310-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp310-cp310-win_amd64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp36-cp36m-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp36-cp36m-win_amd64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp37-cp37m-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp37-cp37m-win_amd64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp38-cp38-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp38-cp38-win_amd64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp39-cp39-macosx_10_9_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl as this is not supported by the current environment
Skipping wheel qdldl-0.1.5.post2-cp39-cp39-win_amd64.whl as this is not supported by the current environment
• Updating qdldl (0.1.5.post2 /Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz -> 0.1.5.post2): Installing...
• Updating qdldl (0.1.5.post2 /Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz -> 0.1.5.post2): Failed

Stack trace:

2 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1472 in _run
1470│ )
1471│ else:
→ 1472│ output = subprocess.check_output(
1473│ command, stderr=subprocess.STDOUT, env=env, **kwargs
1474│ )

1 ~/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py:420 in check_output
418│ kwargs['input'] = empty
419│
→ 420│ return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
421│ **kwargs).stdout
422│

CalledProcessError

Command '['/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10', '--upgrade', '--no-deps', '/Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz']' returned non-zero exit status 1.

at ~/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py:524 in run
520│ # We don't call process.wait() as .exit does that for us.
521│ raise
522│ retcode = process.poll()
523│ if check and retcode:
→ 524│ raise CalledProcessError(retcode, process.args,
525│ output=stdout, stderr=stderr)
526│ return CompletedProcess(process.args, retcode, stdout, stderr)
527│
528│

The following error occurred when trying to handle this error:

Stack trace:

3 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:49 in pip_install
47│
48│ try:
→ 49│ return environment.run_pip(*args)
50│ except EnvCommandError as e:
51│ raise PoetryException(f"Failed to install {path.as_posix()}") from e

2 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1435 in run_pip
1433│ pip = self.get_pip_command()
1434│ cmd = pip + list(args)
→ 1435│ return self._run(cmd, **kwargs)
1436│
1437│ def run_python_script(self, content: str, **kwargs: Any) -> int | str:

1 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1712 in _run
1710│ def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
1711│ kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
→ 1712│ return super()._run(cmd, **kwargs)
1713│
1714│ def get_temp_environ(

EnvCommandError

Command ['/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10', '--upgrade', '--no-deps', '/Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz'] errored with the following return code 1, and output:
Processing /Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [25 lines of output]
    /Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10/bin/python: No module named pip
    Traceback (most recent call last):
      File "<string>", line 24, in __init__
    ModuleNotFoundError: No module named 'pybind11'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
        main()
      File "/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Users/axeldelaharpe/Library/Caches/pypoetry/virtualenvs/poetry-demo-RrhhU6tg-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/private/var/folders/h_/mnx8ymsx41jd3t7w_xq27_nh0000gn/T/pip-build-env-8dzijn3z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
        return self._get_build_requires(config_settings, requirements=['wheel'])
      File "/private/var/folders/h_/mnx8ymsx41jd3t7w_xq27_nh0000gn/T/pip-build-env-8dzijn3z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
        self.run_setup()
      File "/private/var/folders/h_/mnx8ymsx41jd3t7w_xq27_nh0000gn/T/pip-build-env-8dzijn3z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 482, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/private/var/folders/h_/mnx8ymsx41jd3t7w_xq27_nh0000gn/T/pip-build-env-8dzijn3z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
        exec(code, locals())
      File "<string>", line 98, in <module>
      File "<string>", line 28, in __init__
    RuntimeError: pybind11 install failed.
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1476 in run
1472│ output = subprocess.check_output(
1473│ command, stderr=subprocess.STDOUT, env=env, **kwargs
1474│ )
1475│ except CalledProcessError as e:
→ 1476│ raise EnvCommandError(e, input=input)
1477│
1478│ return decode(output)
1479│
1480│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

Stack trace:

6 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:253 in _execute_operation
251│
252│ try:
→ 253│ result = self._do_execute_operation(operation)
254│ except EnvCommandError as e:
255│ if e.e.returncode == -2:

5 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:326 in _do_execute_operation
324│ return 0
325│
→ 326│ result: int = getattr(self, f"execute{method}")(operation)
327│
328│ if result != 0:

4 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:453 in _execute_update
451│
452│ def _execute_update(self, operation: Install | Update) -> int:
→ 453│ status_code = self._update(operation)
454│
455│ self._save_url_reference(operation)

3 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:491 in _update
489│
490│ def _update(self, operation: Install | Update) -> int:
→ 491│ return self._install(operation)
492│
493│ def _remove(self, operation: Uninstall) -> int:

2 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:488 in _install
486│ )
487│ self._write(operation, message)
→ 488│ return self.pip_install(archive, upgrade=operation.job_type == "update")
489│
490│ def _update(self, operation: Install | Update) -> int:

1 ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:123 in pip_install
121│ ) -> int:
122│ try:
→ 123│ pip_install(req, self._env, upgrade=upgrade, editable=editable)
124│ except EnvCommandError as e:
125│ output = decode(e.e.output)

PoetryException

Failed to install /Users/axeldelaharpe/Library/Caches/pypoetry/artifacts/4f/3e/45/a608f7643f52658bb7cee0e0fce6b7b5cc1728ce62512187c20f77f6eb/qdldl-0.1.5.post2.tar.gz

at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:51 in pip_install
47│
48│ try:
49│ return environment.run_pip(*args)
50│ except EnvCommandError as e:
→ 51│ raise PoetryException(f"Failed to install {path.as_posix()}") from e
52│

`

[dimbleby]

this is an upstream issue, qdldl should declare its build-time dependency on pybind11 in a pyproject.toml. Please report it at that project.

You can reproduce this without poetry like so:

pip install --use-pep517 --no-binary qdldl qdldl

[neersighted]

You can see #6407 (comment) for more details with a package that had an identical issue.

[adelaharpe]

this is an upstream issue, qdldl should declare its build-time dependency on pybind11 in a pyproject.toml. Please report it at that project.

You can reproduce this without poetry like so:

pip install --use-pep517 --no-binary qdldl qdldl

But pip install --no-binary qdldl qdldl works. So does that means that poetry 1.1.15 did not use the --use-pep517 flag and 1.2.1 does and thus that is why it is working with 1.1.15?

[dimbleby]

yes

[tommyjcarpenter]

@dimbleby since we hit this too, I looked into making a pull request to their repo. However, qdldl is not using poetry, and they do declare pybind11 in their setup: https://github.com/osqp/qdldl-python/blob/master/setup.py

Ive never seen this before... a setup.py trying to.. call pip:

        try:
            import pybind11
            pybind11
        except ImportError:
            if call([sys.executable, '-m', 'pip', 'install', 'pybind11']):
                raise RuntimeError('pybind11 install failed.')

I have a suspicion that the above code (from https://github.com/osqp/qdldl-python/blob/master/setup.py#L24-L28) is not poetry safe. Becuase when I install with poetry verbose, I see:

× Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [25 lines of output]
        /Users/tommy/Library/Caches/pypoetry/virtualenvs/myenv-5hFYC3k8-py3.10/bin/python: No module named pip

I think that is actually the culprit..

I dont mind doing a pull request into qd, but since I've not seen something like that before, I'm a little unclear on the change to make.. install pip in the poetry env? try poetry install pybind11 as well as -m pip??

[dimbleby]

https://pybind11.readthedocs.io/en/stable/compiling.html#pep-518-requirements-pip-10-required describe ways to do this, this is out of scope for poetry

you can verify that this is currently broken with pip install --use-pep517 --no-binary qdldl qdldl, and do similar to test your fix

[xcrzx]

@dimbleby Is it possible to disable the usage of the --use-pep517 flag by Poetry as it turns out to cause failures?

[Secrus]

@dimbleby Is it possible to disable the usage of the --use-pep517 flag by Poetry as it turns out to cause failures?

It is not possible and won't ever be. PEP-517 is the way Python packaging ecosystem is supposed to go with. If there is a package that fails to install with PEP-517 enabled, you should raise it as an issue with package maintainers.

[xcrzx]

It is not possible and won't ever be. PEP-517 is the way Python packaging ecosystem is supposed to go with. If there is a package that fails to install with PEP-517 enabled, you should raise it as an issue with package maintainers.

@Secrus, I understand your stance on this question, but I cannot support it. It is well known that many package authors have not adopted PEP-517 yet. A quick search by issues in this repo shows that many people are struggling with the --use-pep517 flag enabled by default.

In an ideal world, package authors adopt standards as soon as they arise. But that is obviously not the case. It would probably take years before popular packages get their dependencies sorted out properly. And meanwhile, the job of package managers is to make this transition as smooth as possible and don't break installation processes.

you should raise it as an issue with package maintainers

I see that for the package I use, there's already an issue opened for almost a month: osqp/qdldl-python#25. But maintainers don't seem to take any action. What would be your proposed way of going forward with the broken installation process?

[dimbleby]

Clearly you can't use qdldl together with recent poetry. You'll have to take your choice: either don't use that package, or find some other way to install it.

osqp/qdldl-python#27 (comment) might help

[neersighted]

Indeed, Poetry cannot walk away from use of --use-pep517 without regressing in important functionality that was recently introduced (the ability to depend on and manage setuptools and pip like any other package); it is also part of our incremental work towards a new installer that completely removes Poetry's usage of pip.

In essence, if you are not PEP 517 compatible, you are not compatible with any documented build system or behavior; you are only fully installable with pip. Poetry is not pip and we have to make different design choices. Becoming PEP 517 compatible is trivial for over 90% of cases -- and if upstream is unresponsive, I would suggest forking and fixing it, or building wheels with pip and installing them with Poetry.

[leonox]

qdldl = { git = "https://github.com/osqp/qdldl-python.git"}

Clearly you can't use qdldl together with recent poetry. You'll have to take your choice: either don't use that package, or find some other way to install it.

osqp/qdldl-python#27 (comment) might help

I can confirm (for iOS) this resolves issues with several packages depending on qdldl (cvxpy, pyportfolioopt, etc.)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.