Repository navigation
feat(pypi): add pip.configure.testonly to make packages testonly #4428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
aignas
wants to merge
28
commits into
bazel-contrib:main
Choose a base branch
from
aignas:aignas.feat.pip-configure-testonly
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
3ef53b2
refactor(pypi): select packages from the pyproject.toml dependency gr…
aignas 7557370
refactor(pypi): support hiding the unexposed packages of a hub
aignas 23127e5
feat(pypi): add pip.configure to select the exposed packages of a hub
aignas 6bb7f83
fix(pypi): only add the exposed packages of a configured hub to @pypi
aignas fcaac66
refactor(pypi): support testonly packages in the hub repositories
aignas 0630613
feat(pypi): add pip.configure.testonly to make packages testonly
aignas b31a734
fix(pypi): treat exposing all packages as not configuring the hub
aignas 9d97360
refactor(pypi): use explicit loops and simpler conditions in pip.conf…
aignas 35c0660
test(pypi): test the pip.default pyproject_toml with pip.configure
aignas ffa3e55
Merge branch 'aignas.feat.dep-group-selectors' into aignas.feat.pip-c…
aignas 003655c
refactor(pypi): drop the unreachable testonly check in pkg_aliases
aignas 5e4c809
refactor(pypi): compute the testonly packages once in parse_modules
aignas 41a0664
news: name the pip.configure.testonly entry after the PR
aignas 71be0c0
fix(pypi): only pass testonly to the package alias when it is set
aignas a2049ee
Merge branch 'main' into aignas.feat.dep-group-selectors
aignas 330eb71
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas 3887ace
Merge remote-tracking branch 'upstream/main' into aignas.feat.pip-con…
aignas 7765a28
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas 568c362
Merge branch 'aignas.feat.dep-group-selectors' into aignas.feat.pip-c…
aignas 1ffebb8
refactor(pypi): polish pip.configure docs, docstrings, and formatting
rickeylev a40d1d5
refactor(pypi): restore pyproject_toml attribute docstrings
rickeylev 1291919
refactor(pypi): short-circuit _fail paths with None in group selection
rickeylev 28b48d5
refactor(pypi): expand nested self-extras and refine type comments
rickeylev 0f35bd1
feat(pypi): make pip.configure a builder that does not require a pypr…
aignas 828be47
Merge remote-tracking branch 'origin/aignas.feat.dep-group-selectors'…
aignas 591ccf1
fix(pypi): fail on pip.configure.testonly without the dependency groups
aignas 7c58d7b
docs(pypi): rewrap the pip.configure docs
aignas ddd8de2
Merge remote-tracking branch 'origin/aignas.feat.dep-group-selectors'…
aignas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| (pypi) Added {obj}`pip.configure` to limit the packages that a hub exposes to | ||
| the ones listed in the dependency groups of a `pyproject.toml` file | ||
| ([#3413](https://github.com/bazel-contrib/rules_python/issues/3413)). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| (pypi) Added {attr}`pip.configure.testonly` to make the packages selected by | ||
| {obj}`pip.configure` testonly | ||
| ([#3413](https://github.com/bazel-contrib/rules_python/issues/3413)). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| """Select PyPI packages using pyproject.toml dependency groups.""" | ||
|
|
||
| load("@bazel_skylib//lib:types.bzl", "types") | ||
| load("//python/private:normalize_name.bzl", "normalize_name") | ||
| load(":pep508_requirement.bzl", "requirement") | ||
|
|
||
| def select_dependency_groups(pyproject, specs, _fail = fail): | ||
| """Select the packages named by the given dependency group specifiers. | ||
|
|
||
| Supported specifiers are: | ||
|
|
||
| * `"default"`: the `[project] dependencies`. | ||
| * `"optional:<name>"`: a `[project.optional-dependencies]` entry. | ||
| * `"group:<name>"`: a PEP 735 `[dependency-groups]` entry. | ||
| * `"optional:*"` and `"group:*"`: all entries of the table. | ||
| * `"*"`: all of the above. | ||
| * `"<name>"`: a dependency group, or an optional dependency group if | ||
| there is no dependency group with that name. | ||
| * `"transitive"`: only supported together with `"*"` and selects all | ||
| of the packages, including the transitive dependencies. | ||
|
|
||
| Args: | ||
| pyproject: {type}`struct` returned by `read_pyproject`. | ||
| specs: {type}`list[str]` the specifiers. | ||
| _fail: {type}`callable` the failure function, mainly for testing. | ||
|
|
||
| Returns: | ||
| {type}`struct | None` with the attributes: | ||
| * `packages`: {type}`list[str]` the sorted normalized package names. | ||
| * `all`: {type}`bool` whether all of the packages including the | ||
| transitive dependencies should be selected, i.e. `"*"` and | ||
| `"transitive"` were given. | ||
| Returns `None` on failure. | ||
| """ | ||
| groups = {"default": pyproject.dependencies} | ||
| for name, reqs in pyproject.optional_dependencies.items(): | ||
| groups["optional:" + normalize_name(name)] = reqs | ||
| expanded = _expand_dependency_groups(pyproject.dependency_groups, _fail) | ||
| if expanded == None: | ||
| return None | ||
| for name, reqs in expanded.items(): | ||
| groups["group:" + name] = reqs | ||
|
|
||
| if "transitive" in specs: | ||
| if "*" not in specs: | ||
| _fail("The 'transitive' dependency group specifier is only supported together with '*', which selects all of the packages, got: {}".format( | ||
| specs, | ||
| )) | ||
| return None | ||
| return struct( | ||
| # list[str]: sorted normalized package names selected by `specs`. | ||
| packages = [], | ||
| # bool: True when all hub packages are selected. | ||
| all = True, | ||
| ) | ||
|
|
||
| selected = {} | ||
| for spec in specs: | ||
| keys = _match(spec, groups) | ||
| if not keys: | ||
| _fail("Unknown dependency group '{}'. Available groups: {}".format( | ||
| spec, | ||
| ", ".join(sorted(groups)), | ||
| )) | ||
| return None | ||
| for key in keys: | ||
| selected[key] = None | ||
|
|
||
| # A self reference like `my-project[test]` selects the optional | ||
| # dependencies, including nested composite extras. | ||
| project = normalize_name(pyproject.name or "") | ||
| packages = {} | ||
| queue = list(selected) | ||
| for _ in range(len(groups)): | ||
| if not queue: | ||
| break | ||
| for req in groups[queue.pop(0)]: | ||
| req = requirement(req) | ||
| if req.name_ != project: | ||
| packages[req.name_] = None | ||
| continue | ||
| for extra in req.extras: | ||
| opt_key = "optional:" + extra | ||
| if opt_key in groups and opt_key not in selected: | ||
| selected[opt_key] = None | ||
| queue.append(opt_key) | ||
|
|
||
| return struct( | ||
| # list[str]: sorted normalized package names selected by `specs`. | ||
| packages = sorted(packages), | ||
| # bool: True when all packages including transitive ones are selected. | ||
| all = False, | ||
| ) | ||
|
|
||
| def _match(spec, groups): | ||
| """Match a dependency group specifier against normalized group keys. | ||
|
|
||
| Args: | ||
| spec: {type}`str` the dependency group specifier. | ||
| groups: {type}`dict[str, list[str]]` available groups keyed by | ||
| prefixed group name. | ||
|
|
||
| Returns: | ||
| {type}`list[str]` matching keys in `groups`. | ||
| """ | ||
| if spec == "*": | ||
| return groups.keys() | ||
| if spec in ("optional:*", "group:*"): | ||
| prefix = spec[:-1] | ||
| return [key for key in groups if key.startswith(prefix)] | ||
|
|
||
| kind, _, name = spec.rpartition(":") | ||
| name = normalize_name(name) | ||
| if kind: | ||
| candidates = [kind + ":" + name] | ||
| elif name == "default": | ||
| candidates = ["default"] | ||
| else: | ||
| candidates = ["group:" + name, "optional:" + name] | ||
|
|
||
| for key in candidates: | ||
| if key in groups: | ||
| return [key] | ||
| return [] | ||
|
|
||
| def _expand_dependency_groups(dependency_groups, _fail): | ||
| """Expand the PEP 735 `include-group` entries. | ||
|
|
||
| Args: | ||
| dependency_groups: {type}`dict[str, list[str | dict[str, str]]]` the | ||
| `[dependency-groups]` table. | ||
| _fail: {type}`callable` the failure function. | ||
|
|
||
| Returns: | ||
| {type}`dict[str, list[str]] | None` normalized group names mapped to | ||
| their requirement strings, or `None` on failure. | ||
| """ | ||
| groups = { | ||
| normalize_name(name): items | ||
| for name, items in dependency_groups.items() | ||
| } | ||
|
|
||
| # Each round expands at least one group unless there is a cycle. | ||
| expanded = {} | ||
| for _ in range(len(groups)): | ||
| for name, items in groups.items(): | ||
| if name in expanded: | ||
| continue | ||
|
|
||
| reqs = [] | ||
| for item in items: | ||
| if types.is_string(item): | ||
| reqs.append(item) | ||
| continue | ||
|
|
||
| include = normalize_name(item.get("include-group", "")) | ||
| if include not in groups: | ||
| _fail("Unknown dependency group '{}' included from '{}'".format( | ||
| include, | ||
| name, | ||
| )) | ||
| return None | ||
| if include not in expanded: | ||
| reqs = None | ||
| break | ||
| reqs.extend(expanded[include]) | ||
|
|
||
| if reqs != None: | ||
| expanded[name] = reqs | ||
|
|
||
| if len(expanded) != len(groups): | ||
| unexpanded = [name for name in groups if name not in expanded] | ||
| _fail("Cyclic dependency group includes: {}".format( | ||
| ", ".join(sorted(unexpanded)), | ||
| )) | ||
| return None | ||
| return expanded |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be: a package is testonly in the unified hub if all hubs mark it testonly
?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It has to be "any": the unified alias selects a hub by the
venvflag, buttestonlyis not configurable, so a non-testonly alias pointing at a testonly hub target fails analysis for that flag value.This is an agent-generated comment unreviewed by the user.