Skip to content

feat(pypi): resolve the extras of the dependencies in the whl repositories - #4523

Draft
aignas wants to merge 27 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.extras-without-hub
Draft

aignas wants to merge 27 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.extras-without-hub

Conversation

@aignas

@aignas aignas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

With #4490 the dependencies on the extras of other packages from the wheel
METADATA, e.g. Requires-Dist: bar[socks], use the @pypi//bar:[socks]
target of the hub. The hub only has the targets for the extras that are in the
lock file, so the hub passes the available extras to the whl repositories via
config.bzl and the build fails if the extra is not locked. However, lock
files often do not have the extras (e.g. uv pip compile and pip-compile
strip them by default), even though the packages of the extras are locked,
and the BUILD files of all of the whl repositories change whenever the extras
in the lock file change.

The extras of a package are only known from its own METADATA, so with this
change the whl repository of the package resolves them:

  • The whl repository of foo that depends on bar[socks] creates an
    extra_dep target that depends on @pypi//bar:_extras with a private
    requested_extra build setting set to socks via a transition.
  • The whl repository of bar creates an extra_select target named _extras
    that selects its [socks] target based on the setting and resets the
    setting on that edge, so that the targets of the extras and their
    dependencies are analyzed in the same configuration as the rest of the
    build. Only the _extras target itself is in the transitioned
    configuration.
  • The hub only forwards _extras to the whl repository of the target
    platform, like the other per-package aliases, so it does not need to know
    the extras that the packages provide.
  • An extra that the package does not provide falls back to the package
    itself, which is what pip does, so missing_extra_error and the
    available extras in config.bzl are removed.

Only the selected extra is analyzed, so the cycles are the same as with the
direct dependencies and the group members with extras from #4490 keep working
because the members are selected via the group library in the hub.

Considered alternatives:

  • Create the [extra] targets in the hub for every extra that the package
    provides. This needs the METADATA at hub creation time, i.e. fetching all
    of the wheels (or PEP 658 metadata) eagerly, including the ones for other
    platforms.
  • A single target per package that depends on all of its extras and returns a
    provider keyed by the extra. This analyzes and fetches the dependencies of
    every extra of every package reached through an extra and introduces
    spurious analysis cycles.
  • Expanding the extras in the repository rule of the dependent package by
    reading the METADATA of the dependency. This makes the whl repositories
    depend on each other at fetch time and requires combining the markers.
  • Failing lazily with a missing_package_error style message (the current
    feat(pypi): add targets for the extras of the packages #4490 behaviour) still requires the extras to be in the lock file.

The [extra] targets of the hub for the extras requested in the lock file are
unchanged.

Stacked on #4490.

Work towards #2786

The extra_dep target depends on the _extras target of a package with the
requested extra set in a private build setting and the extra_select target
selects the target of that extra and resets the setting, so that the
targets of the extras are analyzed in the same configuration. The extras
that the package does not provide fall back to the package itself, like pip.
The dependencies on the extras of other packages from the wheel METADATA,
e.g. 'Requires-Dist: bar[socks]', used the '[socks]' target of the hub,
which exists only if the extra is requested in the lock file, so the
build failed if the lock file did not have the extra.

Each whl repository now has an '_extras' target that selects the target
of the extra requested via an extra_dep target, and the hub forwards it to
the whl repository of the target platform. The extras that the package
does not provide fall back to the package, like pip does.
…ries

The whl repositories resolve the extras of their dependencies without the
hub, so the config.bzl of the hub lists only the packages again and the
missing_extra_error rule is removed. This means that the BUILD files of the
whl repositories do not change when the extras in the lock file change.
…locked

Also check that the extras that a package does not provide are ignored in
the whl_library integration test.
@aignas

aignas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

This draft PR was drafted by a Claude Code agent as a design proposal for wiring the extras of the dependencies without relying on the hub repositories or on the lock file containing the extras. The human author (@aignas) handles the review comments on this PR.

This is an agent-generated comment, automatically generated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant