Skip to content

feat(pypi): add pip.configure.venv to create named package selections - #4429

Draft
aignas wants to merge 38 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.pip-configure-venvs
Draft

aignas wants to merge 38 commits into
bazel-contrib:mainfrom
aignas:aignas.feat.pip-configure-venvs

Conversation

@aignas

@aignas aignas commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

feat(pypi): add pip.configure.venv to create named package selections

Before this change the only way to use a different set of packages through
the unified @pypi hub was to create a separate hub with its own
requirements, which downloads the shared packages again.

This adds the venv attribute to pip.configure. A venv is a named
selection of the packages of a hub that is used through the unified @pypi
hub by setting --@rules_python//python/config_settings:venv to its name:

pip.configure(
    hub_name = "pypi",
    venv = "pypi_docs",
    dependency_groups = ["default", "group:docs"],
)

A venv does not create a new repository and it does not change the hub. The
unified hub resolves the selected packages of the venv to the hub's packages
and reports the other packages as missing. The hubs and venvs that resolve to
the same target share a selects.config_setting_group(match_any = ...), so
the selects stay small. If the hub hides some of its packages, the hidden
aliases are visible to the unified hub, because a venv may select them.

This is only supported with bzlmod; the WORKSPACE code is unchanged.

Stacked on #4428.

Fixes #3413

…oups

Read the dependencies, the optional dependencies and the PEP 735
dependency groups from the pyproject.toml and add a helper to select the
packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set,
the aliases of the packages that are not exposed are only visible to the
wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub
exposes to the ones listed in the dependency groups of a `pyproject.toml`
file. The other packages of the hub are still used by the wheels that need
them, but their aliases are not public.
Otherwise the unified hub would point to the hidden aliases of the hub and
fail with a visibility error instead of the missing package error.
Bazel does not enforce `testonly` on aliases, so the main public targets
of a testonly package are testonly wrappers around the aliases. The
testonly packages are listed in `all_testonly_requirements` instead of the
`all_*requirements` lists of the hub's requirements.bzl.
The packages selected by a pip.configure tag with `testonly = True` are
testonly, unless they are also selected by a tag without `testonly`.
A venv is an extra value of the `venv` flag that resolves the packages to
a hub. The hubs and venvs that resolve to the same target share a
`config_setting_group` so that the selects stay small.
A venv is a named selection of the packages of a hub that is used through
the unified @pypi hub by setting the `venv` flag. It does not create a new
repository and it does not change the hub.
@aignas

aignas commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was written by an AI coding agent (Claude Code) under the direction of @aignas, who is responsible for the change and reviews it before merge (semi-autonomous). The agent monitors CI and pushes fixes for CI failures; review comments are handled by the human author, who may use the agent to prepare follow-up changes.

This is an agent-generated comment, posted at the request of the PR author.

aignas added 14 commits November 8, 2026 13:05
pip.configure with ["*", "transitive"] used the packages that are
available on all platforms, which made the platform-specific packages
private. Treat it as if the hub was not configured instead, which keeps
these packages public and out of all_requirements as before.
…igure

Replace the nested comprehensions and the long inline conditions with
explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the
selection is tested in hub_exposure_tests, and test that the pip.default
pyproject_toml is used. Also link the issue in the news entry.
…onfigure-testonly

# Conflicts:
#	python/private/pypi/hub_exposure.bzl
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
#	tests/pypi/extension/extension_tests.bzl
Compute the testonly packages of each hub and of the unified hub in
parse_modules instead of using long inline conditions, and test that a
package is testonly in the unified hub if it is testonly in any hub.
…p-configure-venvs

# Conflicts:
#	python/private/pypi/extension.bzl
#	python/private/pypi/hub_exposure.bzl
#	python/private/pypi/hub_repository.bzl
#	tests/pypi/extension/extension_tests.bzl
…oups

A select may map multiple keys to the same target, so map each venv to the
target of its hub directly. Also always render the venvs in the unified
hub and drop the unreachable hub name collision check.
Also document the reserved venv names and test the extra internal
visibility of the hidden packages.
The existing tests mock the alias without a testonly argument.
rickeylev and others added 10 commits November 18, 2026 19:45
Align the new `pip.configure` documentation, docstrings, and Starlark
files with repository style and documentation conventions.

Add `requires-python` to the `docs/pypi/download.md` example, move the
`pip.configure` section to top-level `##` heading order, clarify the
`pyproject_toml` attribute docstrings and `{ref}` links, document helper
arguments/returns and returned `struct` fields, flatten `hub_exposure`
guards, move module-level test constants before functions, and wrap
Starlark lines and `doc` strings to 80 columns.
Restore the `pip.default.pyproject_toml` and `pip.parse.pyproject_toml`
attribute docstrings to their original text per review decision.
When a non-aborting `_fail` callback is injected in tests,
`select_dependency_groups` and `_expand_dependency_groups` previously
continued executing after `_fail(...)`, causing `hub_exposure` to fail
on `None` attribute access or return partial results instead of `None`.

Return `None` immediately after each `_fail(...)` call in
`dependency_groups.bzl`, guard `selection == None` in
`hub_exposure.bzl`, and assert `None` return values in error tests.
Single-level self-reference expansion dropped nested extras from
composite extras such as `my-project[all]` and added the project name
itself to `packages`, and `dependency_groups` type comments omitted list
element types.

Expand self-referenced optional groups iteratively with a bounded queue
in `dependency_groups.bzl` (with unit tests covering nested, bare, and
cyclic self-references), and specify parameterized element types in
`pyproject_utils.bzl`, `dependency_groups.bzl`, and
`render_pkg_aliases.bzl`.
…oject.toml

The dependency_groups default is now empty, so a pip.configure tag only
needs a pyproject.toml when it selects dependency groups. A pyproject_toml
set on a tag is used by the later tags of the same module and hub.
… into aignas.feat.pip-configure-testonly

# Conflicts:
#	docs/pypi/download.md
#	python/private/pypi/extension.bzl
#	tests/pypi/extension/pip_parse.bzl
#	tests/pypi/hub_exposure/hub_exposure_tests.bzl
… into aignas.feat.pip-configure-testonly

# Conflicts:
#	docs/pypi/download.md
…ly' into aignas.feat.pip-configure-venvs

A venv tag now needs the dependency_groups, because they no longer default
to all of the groups.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pip.parse should allow hiding transitive dependencies

2 participants