Support the metadata directory (as per PEP-517) for build_wheel

[pelson]

Summary of changes

Allow dist-info-dir to be passed to bdist_wheel.
More importantly, fixes the build_wheel PEP-517 interface to treat the given metadata as per the spec.

With this change, I was able to verify that I could:

python setup.py dist_info
touch setuptools-74.1.2.post20240913.dist-info/FOO
python setup.py bdist_wheel --dist-info-dir setuptools-74.1.2.post20240913.dist-info/

And the resulting wheel contained my desired metadata file. My personal goal is to support this behaviour in PEP-517 build backends, and to write a setuptools build backend wrapper which injects such metadata (I need to do something like setuptools-ext does). Presently, the functionality does not exist, you end up having to post-process the built wheel (hence the complexity that exists in setuptools-ext).

Honestly, despite having written many Command subclasses over many years, I am very unfamiliar with the intricacies of distutils.Command and setuptools in general, and may have implemented a solution which is not quite right due to a lack of that knowledge.

Closes #1825. Follows a prototype I did in pypa/wheel#611.

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

[jaraco]

This looks sound to me.

I don't love the complexity this adds, but I also recognize that refactoring the behavior, such as to re-order operations or not generate egg-info might be dangerous.

[jaraco]

Do we know why egg info is still generated? Is it because other commands might have generated it? What happens if the line for rmtree is removed (will the tests fail)?

While I'm okay with this approach, I'm also somewhat apprehensive, as it is perpetuating and reinforcing the expectation that egg-info is generated, even when it's not needed, making it harder in the future to correct this suboptimal approach. For example, this project also has the goal to someday obviate egg-info altogether, and to generate dist-info directly, and adding this additional handling of egg-info will likely complicate that work (though maybe not too much).

I wonder - how much do other commands from bdist_wheel depend on the generation of egg-info and could those commands get by without generating it if bdist_wheel could pass through the instruction to re-use existing dist-info?

[pelson]

Do we know why egg info is still generated?

Looks like install calls install_egg_info (a registered sub-command, which is automatically run), which then calls egg_info. I wasn't brave enough to try to touch this.

What happens if the line for rmtree is removed (will the tests fail)?

I've added a specific test for this. Previously the test would have passed.

this project also has the goal to someday obviate egg-info altogether

Yes, I appreciate that. IMO, this change does add an additional place where you have to think about egg-info, but I believe it is already necessary to deal with bdist_wheel wrt egg-info, and the added code-path now explicitly rejects its use - I think that ultimately makes that easier to remove egg-info in the future.

[jaraco]

Also, I just noticed you're a first-time contributor. Thanks for the contribution and nice work!

[pelson]

I noticed that this PR needed a rebase, so have just done so. Please let me know if there is anything else you need from my side.

[abravalheri]

Thank you very much!

[pelson]

Thanks for your review 👍 🎉

[tacaswell]

This seems to have broken building some projects. https://github.com/yaml/pyyaml I am seeing failures like:

Processing /home/tcaswell/source/p/yaml/pyyaml
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: PyYAML
  Building wheel for PyYAML (pyproject.toml): started
  Building wheel for PyYAML (pyproject.toml): finished with status 'error'
Failed to build PyYAML

  error: subprocess-exited-with-error

  × Building wheel for PyYAML (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      usage: _in_process.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
         or: _in_process.py --help [cmd1 cmd2 ...]
         or: _in_process.py --help-commands
         or: _in_process.py cmd --help

      error: option --dist-info-dir not recognized
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for PyYAML
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (PyYAML)

with the main branch of setuptools, but going back to the commit before this was merged (3106af0 which is v75.1.0) works again.

I have not had time to debug this further and it is not all packages nor do I know if this is the only one.

[pelson]

Looks like that project is importing bdist_wheel from wheel (https://github.com/yaml/pyyaml/blob/69c141adcf805c5ebdc9ba519927642ee5c7f639/setup.py#L97) and setting that as the bdist_wheel command.

To be honest, that implementation looks to be rather unnecessary - I'm not sure on the backwards compat policy of setuptools, but I think this one is best addressed in pyyaml otherwise it will never be possible to add and use new features to the bdist_wheel command in setuptools.

[pelson]

Looks like that project is importing bdist_wheel from wheel
To be honest, that implementation looks to be rather unnecessary

@henryiii already has a proposed fix in yaml/pyyaml#823 (a few weeks before I made this MR 😂).

[abravalheri]

OK, I hit the PyYAML problem in the integration tests when trying to cut a release. I wonder if there is any workaround for this problem. Trying out #4684.