Skip to content

feat(pypi): add pip.configure.testonly to make packages testonly - #4428

Draft
aignas wants to merge 28 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.pip-configure-testonly
Draft

aignas wants to merge 28 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.pip-configure-testonly

Conversation

@aignas

@aignas aignas commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

feat(pypi): add pip.configure.testonly to make packages testonly

Before this change all of the packages of a hub could be used by any target,
even if they were only meant to be used by tests.

This adds the testonly attribute to pip.configure. The packages selected
by a tag with testonly = True are testonly, unless they are also selected by
a tag without testonly:

pip.configure(
    hub_name = "pypi",
    dependency_groups = ["default"],
)
pip.configure(
    hub_name = "pypi",
    dependency_groups = ["group:test"],
    testonly = True,
)

Bazel does not enforce testonly on aliases, so the pkg, whl, data,
dist_info and extracted_whl_files targets of a testonly package are
testonly py_library and filegroup wrappers around the aliases. The
testonly packages are listed in all_testonly_requirements instead of the
all_*requirements lists of the hub's requirements.bzl, and a package is
testonly in the unified @pypi hub if it is testonly in any of the hubs.

This is only supported with bzlmod; the WORKSPACE code is unchanged.

Stacked on #4395.

Work towards #3413

…oups

Read the dependencies, the optional dependencies and the PEP 735
dependency groups from the pyproject.toml and add a helper to select the
packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set,
the aliases of the packages that are not exposed are only visible to the
wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub
exposes to the ones listed in the dependency groups of a `pyproject.toml`
file. The other packages of the hub are still used by the wheels that need
them, but their aliases are not public.
Otherwise the unified hub would point to the hidden aliases of the hub and
fail with a visibility error instead of the missing package error.
Bazel does not enforce `testonly` on aliases, so the main public targets
of a testonly package are testonly wrappers around the aliases. The
testonly packages are listed in `all_testonly_requirements` instead of the
`all_*requirements` lists of the hub's requirements.bzl.
The packages selected by a pip.configure tag with `testonly = True` are
testonly, unless they are also selected by a tag without `testonly`.
@aignas
aignas requested a review from rickeylev as a code owner November 7, 2026 17:23
@aignas

aignas commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was written by an AI coding agent (Claude Code) under the direction of @aignas, who is responsible for the change and reviews it before merge (semi-autonomous). The agent monitors CI and pushes fixes for CI failures; review comments are handled by the human author, who may use the agent to prepare follow-up changes.

This is an agent-generated comment, posted at the request of the PR author.

pip.configure with ["*", "transitive"] used the packages that are
available on all platforms, which made the platform-specific packages
private. Treat it as if the hub was not configured instead, which keeps
these packages public and out of all_requirements as before.
…igure

Replace the nested comprehensions and the long inline conditions with
explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the
selection is tested in hub_exposure_tests, and test that the pip.default
pyproject_toml is used. Also link the issue in the news entry.
…onfigure-testonly

# Conflicts:
#	python/private/pypi/hub_exposure.bzl
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
#	tests/pypi/extension/extension_tests.bzl
Compute the testonly packages of each hub and of the unified hub in
parse_modules instead of using long inline conditions, and test that a
package is testonly in the unified hub if it is testonly in any hub.
The existing tests mock the alias without a testonly argument.
aignas and others added 10 commits November 15, 2026 14:29
Align the new `pip.configure` documentation, docstrings, and Starlark
files with repository style and documentation conventions.

Add `requires-python` to the `docs/pypi/download.md` example, move the
`pip.configure` section to top-level `##` heading order, clarify the
`pyproject_toml` attribute docstrings and `{ref}` links, document helper
arguments/returns and returned `struct` fields, flatten `hub_exposure`
guards, move module-level test constants before functions, and wrap
Starlark lines and `doc` strings to 80 columns.
Restore the `pip.default.pyproject_toml` and `pip.parse.pyproject_toml`
attribute docstrings to their original text per review decision.
When a non-aborting `_fail` callback is injected in tests,
`select_dependency_groups` and `_expand_dependency_groups` previously
continued executing after `_fail(...)`, causing `hub_exposure` to fail
on `None` attribute access or return partial results instead of `None`.

Return `None` immediately after each `_fail(...)` call in
`dependency_groups.bzl`, guard `selection == None` in
`hub_exposure.bzl`, and assert `None` return values in error tests.
Single-level self-reference expansion dropped nested extras from
composite extras such as `my-project[all]` and added the project name
itself to `packages`, and `dependency_groups` type comments omitted list
element types.

Expand self-referenced optional groups iteratively with a bounded queue
in `dependency_groups.bzl` (with unit tests covering nested, bare, and
cyclic self-references), and specify parameterized element types in
`pyproject_utils.bzl`, `dependency_groups.bzl`, and
`render_pkg_aliases.bzl`.
…oject.toml

The dependency_groups default is now empty, so a pip.configure tag only
needs a pyproject.toml when it selects dependency groups. A pyproject_toml
set on a tag is used by the later tags of the same module and hub.
… into aignas.feat.pip-configure-testonly

# Conflicts:
#	docs/pypi/download.md
#	python/private/pypi/extension.bzl
#	tests/pypi/extension/pip_parse.bzl
#	tests/pypi/hub_exposure/hub_exposure_tests.bzl

@rickeylev rickeylev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit hard to review this diff with all the changes from the base PR, so this will have to wait until the other PR is merged. Or rebase this one so its clear which commits to start at

Comment thread docs/pypi/download.md
Comment on lines +267 to +268
the [Unified @pypi Hub](unified-pypi-hub), a package is testonly if it is
testonly in any of the hubs.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be: a package is testonly in the unified hub if all hubs mark it testonly

?

@aignas aignas Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has to be "any": the unified alias selects a hub by the venv flag, but testonly is not configurable, so a non-testonly alias pointing at a testonly hub target fails analysis for that flag value.

This is an agent-generated comment unreviewed by the user.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants