Pull-through: pip gets the wheel instead of its PEP 658 metadata (or the reverse), hash mismatch #1428

Versions: pulp-python 3.36.2, pulpcore 3.118.1 (pulp/pulp:3.118.1 image), CACHE_ENABLED = True, pip 26.2.1.
The code paths involved are unchanged in pulp-python 3.37.1 and pulpcore 3.121.3.

What happens

Installing from a pull-through distribution (remote https://pypi.org/) fails intermittently:

$ pip install --no-cache-dir --index-url https://pulp.example/pypi/pypi-proxy/simple/ playwright==1.57.0
  Obtaining dependency information for playwright==1.57.0 from
    https://pulp.example/pulp/content/pypi-proxy/playwright-1.57.0-py3-none-manylinux1_x86_64.whl?redirect=https://files.pythonhosted.org/packages/56/61/.../playwright-1.57.0-py3-none-manylinux1_x86_64.whl.metadata
  Downloading playwright-1.57.0-py3-none-manylinux1_x86_64.whl (3.5 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.
    Expected sha256 284ed5a7...   (the wheel)
         Got        1e874296...   (its METADATA file)

Depending on which request came first, the opposite also happens: the .metadata request returns the 46 MB wheel and pip reports the metadata hash as mismatching.

Cause

The simple index publishes pull-through files as

/pulp/content/<base_path>/<filename>.whl?redirect=<upstream url>#sha256=...

together with data-core-metadata / data-dist-info-metadata. PEP 658 says the metadata lives at the file URL + .metadata, and pip builds that URL from the whole link (f"{link.url_without_fragment}.metadata"). With a query string in the link, the suffix lands in the query:

/pulp/content/<base_path>/<filename>.whl?redirect=<upstream url>.metadata

PythonRemote.get_remote_artifact_url() already knows about this (_real_relative_path), and on a cold request it does stream the right upstream file. But the request path is still the wheel's, and everything else in the content app keys on the path:

1. Content cache. AsyncContentCache.make_key() builds the key from request.path and request.method only (pulpcore/cache/cache.py). The wheel and its metadata share one Redis entry, so whichever is requested first is served for both until the entry expires (X-PULP-CACHE: HIT, 3453 bytes returned for the wheel URL in our case).
2. Repository lookup. Once the wheel has been added to the distribution's repository (async_pull_through_add_content), Handler._match_and_stream() finds a ContentArtifact with relative_path=<filename>.whl before it ever reaches the pull-through branch that reads ?redirect=. The metadata request is then answered with the wheel, cache or no cache.

Reproduce

B=https://pulp.example/pulp/content/pypi-proxy
W=playwright-1.57.0-py3-none-manylinux1_x86_64.whl
R=https://files.pythonhosted.org/packages/56/61/3a803cb5ae0321715bfd5247ea871d25b32c8f372aeb70550a90c5f586df/$W

curl -s "$B/$W?redirect=$R.metadata" | sha256sum   # 1e874296... metadata, X-PULP-CACHE: MISS
curl -s "$B/$W?redirect=$R"          | sha256sum   # 1e874296... metadata again, X-PULP-CACHE: HIT — should be the wheel

Workaround we deployed

A reverse-proxy rewrite that moves the suffix onto the path before the content app sees it. With <filename>.whl.metadata?redirect=<url>.metadata both mechanisms behave, because the path is now distinct:

set $pep658 "$uri?$arg_redirect";
if ($pep658 ~ "^/pulp/content/.+\.whl\?.+\.whl\.metadata$") {
    rewrite ^(.+)$ $1.metadata last;
}

Suggested fix

The robust fix is in pulp_python: do not carry the upstream location in a query string, since PEP 658 clients append to the URL as a whole.

• Publish pull-through links whose path ends with the filename and that have no query, e.g. /pulp/content/<base_path>/<filename>.whl, and resolve the upstream URL server-side in get_remote_artifact_url() (from the upstream simple/JSON page for the project, which the plugin already fetches to render the index; it can be cached). <filename>.whl.metadata then works with no special case and the _real_relative_path hack can go.
• If the redirect hint must stay client-visible, encode it as a path segment in front of the filename (/<base_path>/<encoded upstream dir>/<filename>.whl) rather than as a query parameter.

Independently, pulpcore could protect plugins from this class of bug: let a distribution/remote opt query parameters into the content cache key (a CacheKeys.query alongside path/host/method), or skip the cache when a remote's get_remote_artifact_url() depends on request.query. That alone would not fix mechanism 2 above, so it complements the pulp_python change rather than replacing it.