micropipenv future & python dependency pinning standardization

[VannTen]

We want to shift micropipenv direction toward the currently discussed "lock file" standard.

We believe that standardizing dependency pinning in the Python ecosystem (for
resuming the purpose of the above discussion) can answer to the problem
micropipenv currently solves.

As such, we intent to focus our work on working with the Python community on
defining that standard.
This implies refining the scope and purpose of micropipenv a bit:

1. micropipenv should not be a "stack resolver", aka, it should not try to be
   pipenv, poetry or pdm.
2. it shall become a converter from those packages managers lock files to the future standard
   format.
3. once the standard is adopted and supported, it will either stay a minimal
   installer, or be obsoleted if pip support the new format sufficiently.

For micropipenv users

There won't be any short term impact on micropipenv usage.
On the mid or long term, we might deprecate some functionally, particularly
around resolving (aka, getting from a version requirement to an exact version).

On the other hand, if you use micropipenv, knowing your use case could probably
help the new standard discussion; if you can, please join the discussion,
or expose it in this issue.

[VannTen]

#256 (comment)

Responding there as this is relevant and would apply to similar issues.

@fridex

Ideally yes, if there is only one standard for defining requirements and lockfiles, this would not be needed. But I would expect it will take some time (years?) to have the lockfile standard accepted and used widely. In the meantime, micropipenv could be helpful.

There is kind of a standard already for defining "requirements" (aka, non-exact deps) in wheels metadata: either specify it in pyproject.toml or mark it as dynamic and delegate it ot the build-backend / another tool (for example, see https://github.com/asyncon/setuptools-pipfile )

Where micropipenv could be useful, IMO, is to install resolved python stack regardless of the package manager used (poetry, pdm, pipenv, other).

Handling the "requirements files" (aka Pipfile and equivalent for others) leans dangerously into doing the work of all those packages managers all at once, which why narrowing the scope looks like the best path forward to me.

[fridex]

Where micropipenv could be useful, IMO, is to install resolved python stack regardless of the package manager used (poetry, pdm, pipenv, other).

If I understand the tool correctly, micropipenv does exactly this.

1. micropipenv should not be a "stack resolver", aka, it should not try to be
   pipenv, poetry or pdm.
2. it shall become a converter from those packages managers lock files to the future standard
   format.
3. once the standard is adopted and supported, it will either stay a minimal
   installer, or be obsoleted if pip support the new format sufficiently.

These are also true already. Not sure if I'm missing something.

[VannTen]

On Thu, Jan 26, 2023 at 08:05:50AM -0800, Fridolín Pokorný wrote: > Where micropipenv could be useful, IMO, is to install resolved python stack regardless of the package manager used (poetry, pdm, pipenv, other). If I understand the tool correctly, micropipenv does exactly this.

Well, it should. We don't have support for pdm.

> 1. micropipenv should not be a "stack resolver", aka, it should not try to be > pipenv, poetry or pdm. > 2. it shall become a converter from those packages managers lock files to the future standard > format. > 3. once the standard is adopted and supported, it will either stay a minimal > installer, or be obsoleted if pip support the new format sufficiently. These are also true already. Not sure if I'm missing something.

Which ones ? 2 and 3 are not yet there because there isn't a standard. 1 is precisely why I'd like to move away from handling Pipfile (among others) and only focusing on lock files.

[fridex]

... are not yet there because there isn't a standard.

Yes, it will most probably take some time to have a standardized lockfile in Python. In the meantime, micropipenv can be valuable for other lockfile formats.

[VannTen]

Precisely. But Pipfile, poetry.toml etc are not lockfiles ; they are dependency solver specifications, which is a bit different.

[frenzymadness]

My point of view is that we should focus on lock files but Pipfile or pyproject.toml might sometimes have important info as well. Do you see a way how to support the current set of functionalities without touching anything else than lock files?

[VannTen]

On Fri, Jan 27, 2023 at 01:45:26AM -0800, frenzymadness wrote: My point of view is that we should focus on lock files but Pipfile or pyproject.toml might sometimes have important info as well. Do you see a way how to support the current set of functionalities without touching anything else than lock files?

pyproject.toml info and similar should not be relevant to us, as far as I can tell ; it's for build frontend (most of the times pip). For a dependency installer (locked dependencies), we don't need anything other than lock files. For some of the current functionality of micropipenv, yeah, we do. But precisely the point of that issue is to reduce our scope. One possibility would be to split the sub-command in different packages (like `micropipenv[install]` / `micropipenv[requirements]` for example). But I thinks the requirements command for example duplicate functionality from pipenv.

[frenzymadness]

For a dependency installer (locked dependencies), we don't need anything other than lock files.

It seems that the current version of poetry needs pyproject.toml and poetry.lock both to be present. They for example removed the category key from lock files so the following pyproject.toml:

[tool.poetry]
name = "aaa"
version = "0.1.0"
description = ""
authors = ["Fridolin Pokorny <[email protected]>"]

[tool.poetry.dependencies]
daiquiri = "2.0.0"

[tool.poetry.group.test.dependencies]
flexmock = "^0.10.4"

[build-system]
requires = ["poetry>=1.5.0"]
build-backend = "poetry.masonry.api"

now produces:

# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.

[[package]]
name = "daiquiri"
version = "2.0.0"
description = "Library to configure Python logging easily"
optional = false
python-versions = "*"
files = [
    {file = "daiquiri-2.0.0-py2.py3-none-any.whl", hash = "sha256:d57b9fd5432933c6e899054eb62cee22eab89f560c8493254d327ec27893c866"},
    {file = "daiquiri-2.0.0.tar.gz", hash = "sha256:6b235ed15b73b87fd3cc2521aacbb727bf8443a0896dc534b07503841d03cfdb"},
]

[package.dependencies]
python-json-logger = "*"

[package.extras]
systemd = ["systemd-python (>=234)"]
test = ["mock", "pytest", "six"]

[[package]]
name = "flexmock"
version = "0.10.10"
description = "flexmock is a testing library for Python that makes it easy to create mocks,stubs and fakes."
optional = false
python-versions = "*"
files = [
    {file = "flexmock-0.10.10.tar.gz", hash = "sha256:8bb073f4b7b590672e8c312e73d6a14f88ae624a867b691462f9e8c24b9f19d1"},
]

[[package]]
name = "python-json-logger"
version = "0.1.11"
description = "A python library adding a json log formatter"
optional = false
python-versions = ">=2.7"
files = [
    {file = "python-json-logger-0.1.11.tar.gz", hash = "sha256:b7a31162f2a01965a5efb94453ce69230ed208468b0bbc7fdfc56e6d8df2e281"},
]

[metadata]
lock-version = "2.0"
python-versions = "*"
content-hash = "d3213b26a683ac3ef9b891ad700f40a2e768a20e637e6bb1e8b7cd858903102f"

and I don't see a way how to distinguish between runtime and dev dependencies only from the lock file.

[frenzymadness]

Unfortunately, I don't have the capacity to lead this effort. @goern how the situation looks on your side? I can imagine that some of the bugs we have here would be solvable by the new locking standard but it's too much for me. I'm gonna take a look at the reported bugs but any help with the maintenance of the project would be appreciated.