Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3ef53b2
refactor(pypi): select packages from the pyproject.toml dependency gr…
aignas Nov 7, 2026
7557370
refactor(pypi): support hiding the unexposed packages of a hub
aignas Nov 7, 2026
23127e5
feat(pypi): add pip.configure to select the exposed packages of a hub
aignas Nov 7, 2026
6bb7f83
fix(pypi): only add the exposed packages of a configured hub to @pypi
aignas Nov 7, 2026
fcaac66
refactor(pypi): support testonly packages in the hub repositories
aignas Nov 7, 2026
0630613
feat(pypi): add pip.configure.testonly to make packages testonly
aignas Nov 7, 2026
b31a734
fix(pypi): treat exposing all packages as not configuring the hub
aignas Nov 8, 2026
9d97360
refactor(pypi): use explicit loops and simpler conditions in pip.conf…
aignas Nov 8, 2026
35c0660
test(pypi): test the pip.default pyproject_toml with pip.configure
aignas Nov 8, 2026
ffa3e55
Merge branch 'aignas.feat.dep-group-selectors' into aignas.feat.pip-c…
aignas Nov 8, 2026
003655c
refactor(pypi): drop the unreachable testonly check in pkg_aliases
aignas Nov 8, 2026
5e4c809
refactor(pypi): compute the testonly packages once in parse_modules
aignas Nov 8, 2026
41a0664
news: name the pip.configure.testonly entry after the PR
aignas Nov 8, 2026
71be0c0
fix(pypi): only pass testonly to the package alias when it is set
aignas Nov 8, 2026
a2049ee
Merge branch 'main' into aignas.feat.dep-group-selectors
aignas Nov 9, 2026
330eb71
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas Nov 15, 2026
3887ace
Merge remote-tracking branch 'upstream/main' into aignas.feat.pip-con…
aignas Nov 15, 2026
7765a28
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas Nov 17, 2026
568c362
Merge branch 'aignas.feat.dep-group-selectors' into aignas.feat.pip-c…
aignas Nov 17, 2026
1ffebb8
refactor(pypi): polish pip.configure docs, docstrings, and formatting
rickeylev Nov 19, 2026
a40d1d5
refactor(pypi): restore pyproject_toml attribute docstrings
rickeylev Nov 19, 2026
1291919
refactor(pypi): short-circuit _fail paths with None in group selection
rickeylev Nov 19, 2026
28b48d5
refactor(pypi): expand nested self-extras and refine type comments
rickeylev Nov 19, 2026
0f35bd1
feat(pypi): make pip.configure a builder that does not require a pypr…
aignas Nov 19, 2026
828be47
Merge remote-tracking branch 'origin/aignas.feat.dep-group-selectors'…
aignas Nov 19, 2026
591ccf1
fix(pypi): fail on pip.configure.testonly without the dependency groups
aignas Nov 19, 2026
7c58d7b
docs(pypi): rewrap the pip.configure docs
aignas Nov 19, 2026
ddd8de2
Merge remote-tracking branch 'origin/aignas.feat.dep-group-selectors'…
aignas Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions docs/pypi/download.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,104 @@ re-executed to pick up a non-hermetic change to your environment (e.g., updating
`python` interpreter), you can force it to re-execute by running `bazel sync --only [pip_parse
name]`.

(pip-configure)=
## Configuring the Exposed Packages (pip.configure)

By default a hub exposes all of the packages from its requirements files or
`uv.lock` file, including the transitive dependencies. A `pip.configure` tag
can limit the hub to the packages listed in a `pyproject.toml` file: the
`[project] dependencies`, `[project.optional-dependencies]` and PEP 735
`[dependency-groups]` tables.

```toml
[project]
requires-python = "==3.13.0"
dependencies = ["numpy"]

[dependency-groups]
dev = ["pytest"]
```

```starlark
pip.parse(
hub_name = "pypi_hub",
pyproject_toml = "//:pyproject.toml",
requirements_lock = "//:requirements_lock.txt",
)

pip.configure(
hub_name = "pypi_hub",
dependency_groups = ["default", "group:dev"],
)
```

Here `@pypi_hub//numpy` and `@pypi_hub//pytest` can be used, but the
dependencies of `numpy` are only used through `@pypi_hub//numpy`. The
`all_requirements` lists in the hub's `requirements.bzl` and the
[Unified @pypi Hub](unified-pypi-hub) only contain the exposed packages.

Use `["*"]` to select every group, or `["*", "transitive"]` to expose all of
the packages of the hub, including the transitive dependencies, which is the
same as not configuring the hub. Selecting the transitive dependencies of only
some of the groups is not supported. A tag without
{attr}`pip.configure.dependency_groups` does not limit the hub and does not
need a `pyproject.toml`.

The `pip.configure` tags work like a builder: they are applied in order and
the tags for the same hub are combined. A `pyproject_toml` set on a tag is used
by the later tags of the same module and hub, so it only needs to be set once:

```starlark
pip.configure(
hub_name = "pypi_hub",
pyproject_toml = "//tools:pyproject.toml",
dependency_groups = ["default"],
)

# Uses //tools:pyproject.toml
pip.configure(
hub_name = "pypi_hub",
dependency_groups = ["group:dev"],
)
```

Only the root module and the module that created the hub can configure it; if
both do, the root module wins. A self reference such as `my-project[test]` in
the `pyproject.toml` selects the optional dependencies of the project. The
packages that are not part of the hub are ignored, so a single `pyproject.toml`
can list tools that are missing from the lock file.

### Testonly Packages

Set {attr}`pip.configure.testonly` to make the selected packages testonly, so
that they can only be used by tests and other testonly targets:

```starlark
pip.configure(
hub_name = "pypi_hub",
dependency_groups = ["default"],
)
pip.configure(
hub_name = "pypi_hub",
dependency_groups = ["group:test"],
testonly = True,
)
```

A package that is also selected by a tag without `testonly` is not testonly.
The testonly packages are left out of the `all_requirements`,
`all_whl_requirements` and `all_data_requirements` lists in the hub's
`requirements.bzl` and are listed in `all_testonly_requirements` instead. In
the [Unified @pypi Hub](unified-pypi-hub), a package is testonly if it is
testonly in any of the hubs.
Comment on lines +267 to +268

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be: a package is testonly in the unified hub if all hubs mark it testonly

?

@aignas aignas Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has to be "any": the unified alias selects a hub by the venv flag, but testonly is not configurable, so a non-testonly alias pointing at a testonly hub target fails analysis for that flag value.

This is an agent-generated comment unreviewed by the user.


The other packages of the hub depend on the testonly packages through the same
targets, so a package that a non-testonly package depends on can't be
testonly.

:::{versionadded} VERSION_NEXT_FEATURE
:::

(per-os-arch-requirements)=
## Requirements for a specific OS/Architecture

Expand Down
3 changes: 3 additions & 0 deletions news/4395.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
(pypi) Added {obj}`pip.configure` to limit the packages that a hub exposes to
the ones listed in the dependency groups of a `pyproject.toml` file
([#3413](https://github.com/bazel-contrib/rules_python/issues/3413)).
3 changes: 3 additions & 0 deletions news/4428.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
(pypi) Added {attr}`pip.configure.testonly` to make the packages selected by
{obj}`pip.configure` testonly
([#3413](https://github.com/bazel-contrib/rules_python/issues/3413)).
21 changes: 21 additions & 0 deletions python/private/pypi/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ bzl_library(
deps = [
":default_platforms",
":hub_builder",
":hub_exposure",
":hub_repository",
":parse_whl_name",
":pep508_env",
Expand Down Expand Up @@ -352,6 +353,7 @@ bzl_library(
deps = [
":labels",
":whl_library_deps_targets",
"//python:py_library",
"//python/private:common_labels",
"//python/private:text_util",
"@bazel_skylib//lib:selects",
Expand Down Expand Up @@ -629,6 +631,25 @@ bzl_library(
],
)

bzl_library(
name = "dependency_groups",
srcs = ["dependency_groups.bzl"],
deps = [
":pep508_requirement",
"//python/private:normalize_name",
"@bazel_skylib//lib:types",
],
)

bzl_library(
name = "hub_exposure",
srcs = ["hub_exposure.bzl"],
deps = [
":dependency_groups",
"//python/private:pyproject_utils",
],
)

bzl_library(
name = "argparse",
srcs = ["argparse.bzl"],
Expand Down
177 changes: 177 additions & 0 deletions python/private/pypi/dependency_groups.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
"""Select PyPI packages using pyproject.toml dependency groups."""

load("@bazel_skylib//lib:types.bzl", "types")
load("//python/private:normalize_name.bzl", "normalize_name")
load(":pep508_requirement.bzl", "requirement")

def select_dependency_groups(pyproject, specs, _fail = fail):
"""Select the packages named by the given dependency group specifiers.

Supported specifiers are:

* `"default"`: the `[project] dependencies`.
* `"optional:<name>"`: a `[project.optional-dependencies]` entry.
* `"group:<name>"`: a PEP 735 `[dependency-groups]` entry.
* `"optional:*"` and `"group:*"`: all entries of the table.
* `"*"`: all of the above.
* `"<name>"`: a dependency group, or an optional dependency group if
there is no dependency group with that name.
* `"transitive"`: only supported together with `"*"` and selects all
of the packages, including the transitive dependencies.

Args:
pyproject: {type}`struct` returned by `read_pyproject`.
specs: {type}`list[str]` the specifiers.
_fail: {type}`callable` the failure function, mainly for testing.

Returns:
{type}`struct | None` with the attributes:
* `packages`: {type}`list[str]` the sorted normalized package names.
* `all`: {type}`bool` whether all of the packages including the
transitive dependencies should be selected, i.e. `"*"` and
`"transitive"` were given.
Returns `None` on failure.
"""
groups = {"default": pyproject.dependencies}
for name, reqs in pyproject.optional_dependencies.items():
groups["optional:" + normalize_name(name)] = reqs
expanded = _expand_dependency_groups(pyproject.dependency_groups, _fail)
if expanded == None:
return None
for name, reqs in expanded.items():
groups["group:" + name] = reqs

if "transitive" in specs:
if "*" not in specs:
_fail("The 'transitive' dependency group specifier is only supported together with '*', which selects all of the packages, got: {}".format(
specs,
))
return None
return struct(
# list[str]: sorted normalized package names selected by `specs`.
packages = [],
# bool: True when all hub packages are selected.
all = True,
)

selected = {}
for spec in specs:
keys = _match(spec, groups)
if not keys:
_fail("Unknown dependency group '{}'. Available groups: {}".format(
spec,
", ".join(sorted(groups)),
))
return None
for key in keys:
selected[key] = None

# A self reference like `my-project[test]` selects the optional
# dependencies, including nested composite extras.
project = normalize_name(pyproject.name or "")
packages = {}
queue = list(selected)
for _ in range(len(groups)):
if not queue:
break
for req in groups[queue.pop(0)]:
req = requirement(req)
if req.name_ != project:
packages[req.name_] = None
continue
for extra in req.extras:
opt_key = "optional:" + extra
if opt_key in groups and opt_key not in selected:
selected[opt_key] = None
queue.append(opt_key)

return struct(
# list[str]: sorted normalized package names selected by `specs`.
packages = sorted(packages),
# bool: True when all packages including transitive ones are selected.
all = False,
)

def _match(spec, groups):
"""Match a dependency group specifier against normalized group keys.

Args:
spec: {type}`str` the dependency group specifier.
groups: {type}`dict[str, list[str]]` available groups keyed by
prefixed group name.

Returns:
{type}`list[str]` matching keys in `groups`.
"""
if spec == "*":
return groups.keys()
if spec in ("optional:*", "group:*"):
prefix = spec[:-1]
return [key for key in groups if key.startswith(prefix)]

kind, _, name = spec.rpartition(":")
name = normalize_name(name)
if kind:
candidates = [kind + ":" + name]
elif name == "default":
candidates = ["default"]
else:
candidates = ["group:" + name, "optional:" + name]

for key in candidates:
if key in groups:
return [key]
return []

def _expand_dependency_groups(dependency_groups, _fail):
"""Expand the PEP 735 `include-group` entries.

Args:
dependency_groups: {type}`dict[str, list[str | dict[str, str]]]` the
`[dependency-groups]` table.
_fail: {type}`callable` the failure function.

Returns:
{type}`dict[str, list[str]] | None` normalized group names mapped to
their requirement strings, or `None` on failure.
"""
groups = {
normalize_name(name): items
for name, items in dependency_groups.items()
}

# Each round expands at least one group unless there is a cycle.
expanded = {}
for _ in range(len(groups)):
for name, items in groups.items():
if name in expanded:
continue

reqs = []
for item in items:
if types.is_string(item):
reqs.append(item)
continue

include = normalize_name(item.get("include-group", ""))
if include not in groups:
_fail("Unknown dependency group '{}' included from '{}'".format(
include,
name,
))
return None
if include not in expanded:
reqs = None
break
reqs.extend(expanded[include])

if reqs != None:
expanded[name] = reqs

if len(expanded) != len(groups):
unexpanded = [name for name in groups if name not in expanded]
_fail("Cyclic dependency group includes: {}".format(
", ".join(sorted(unexpanded)),
))
return None
return expanded
Loading
Loading