ignore installed packages during solving

[dimbleby]

fixes #8328, #9774

Currently-installed packages have no place in the locking process. Folk who are doing development against unreleased projects can and should use a path dependency, or a git dependency, or a dependency on a private index, or a privately built wheel

[abn]

@dimbleby doesn't this break the case where system site packages are used for packages that are only available via os package management? While this is not an explicitly supported case, we have been relaxed on such scenarios - this change will start breaking those cases. Might not be a bad thing, just raising this as I have seen this workaround being used in the wild by some clients of mine - particularly for packages with native extensions.

In a away would be good to see if anyone complains and if so, we might need to figure out a better way to support a "system"or "external" source more explicitly when the behavior is desired.

[dimbleby]

doesn't this break the case where system site packages are used for packages that are only available via os package management?

in such cases: just do not declare the dependency

indeed - if such cases even exist - declaring a dependency on such a package would open the door to horrors where I publish a package with a name that deliberately clashes with your system package. Then poetry will upgrade that package and break your system.

[abn]

Can you add a small FAQ entry for this (why installed packages are not used) as well so we can point to it when inevitably it comes up in an issue?

[dimbleby]

I am not sure that I believe there even are any such cases: in that context I expect zero questions about it rather than frequent questions!

Do you have any examples? Without a concrete example I fear that a FAQ would have to be written rather abstractly...

I think I would suggest deferring FAQ entries until such time as the question is frequently asked.

[dimbleby]

Ah, perhaps we are at cross-purposes. I doubt that the os-package-manager-only case exists, but maybe you think a FAQ is worthwhile anyway for the more likely case of local development?

Even then I would propose starting with a changelog / release note entry and promote that only if and when there are frequent questions. But I can leave some helpful text here for when that changelog comes, perhaps something like:

Poetry no longer pays attention to packages installed in your environment during locking. If you have declared a dependency on a package that is unpublished, and available only in your local environment, you should do one of:

• remove that dependency from pyproject.toml
• declare it instead in a way that does not rely on a public index, for instance as a path dependency

[abn]

Sorry, should have been a bit clearer there.

I was thinking about the "developing against unreleased packages" case, not the system package case. The latter is definitely rarer, I agree.

I am okay with deferring it if you reckon that is not a common enough scenario.

[abn]

Approving this as I concur with the rationale that installed packages should not be a source of truth when solving. Doing so is decidedly an anti-pattern.

If, in the future, there is a legitimate reason to use a source for package metadata that is not already a valid source, this must be implemented as a new source type with explicit details around how it gets handled.