404 responses on --extra-index should be cached.

[zopfli]

Description

If packages are distributed over multiple indices, at least one index contains the requested package while the other indices will return a 404. However, this 404 response is not cached, so the non-containing index will repeatedly be queried for package existence. This hurts performance.

Expected behavior

In the cache controller, HTTP status code 404 should be cached.

pip version

main

Python version

3.10

OS

MacOS

How to Reproduce

• Run pip compile with multiple indices, e.g. A and B.
• Request packages from index A that are not on B, and v.v.
• Note that B will respond 404 to attempts searching for A-exclusive packages, and v.v.
• Note that these responses are not cached.

Output

When fetching:

Status code 404 not in (200, 203, 300, 301, 308)

and

Request header has "max_age" as 0, cache bypassed

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[zopfli]

Related to #12397

[notatallshaw]

I'm concerned about side effects this might have, such as a user uploading a package running pip install my_package==new_version, would caching the 404 prevent them from seeing the new package? Or other similar scenarios.

At the moment I believe pip checks all indexes for all packages, every request. Because the behavior of handling multiple indexes, and more so what to do with non-200 responses, is not standardized, I would be surprised if someone isn’t relying on pip’s current behavior here.