Skip to content

pip_parse: lockfile for build dependencies in repository rule #1325

Open

Description

@keith

Since PEP 518 projects can specify their build system requirements in their pyproject.toml file. Currently this is not included in the pip-compile and therefore can lead to non-reproducible issues. This happened in the wild today because pyyaml depends on cython but didn't pin its major version, so when cython pushed an incompatible version this broke if you were including it in bazel. One of the potential workarounds is to install the compatible cython version in the venv before installing your requirements, which I also don't believe is possible. I believe supporting this would require pip-tools to also support this option, which is potentially implemented by pypa/pip-tools#1681

Activity

  1. keith commented on Jul 18, 2023

    @keith
    MemberAuthor

    Another potentially option is multiple requirements files. I believe 1 method that would work is if there was a tooling requirements file that was installed first, and then the normal requirements file which was installed inside the venv used with the first requirements file.

  2. groodt commented on Jul 18, 2023

    @groodt
    Contributor

    There are a number of conflated issues here. One thing to note is that pip does not support pinning build dependencies. See: https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833

    Generally, the pip_parse rule is always going to be non-hermetic since it runs inside repository rules. If your dependency closure includes an sdist, pip will generally build with "build-isolation", where the build dependencies are installed into a temporary virtual environment. This happens on your machine and isn't managed by bazel. Typically, this is what you would normally want. However, sometimes there are missed dependencies specified in a dependency, or, as you've encountered, you wish to prepare the build environment ahead of time. For this, pip provides --no-build-isolation. You are able to provide this to the bazel repository rule, and you are able to prepare a build environment ahead of time on your build host. This isn't managed by bazel however.

    I suspect the above is not what you want to hear, but the flexibility and impurity of the full Python ecosystem that is comprised of Turing complete and non-declarative build specs does not fit very well with the pure and declarative / hermetic world of bazel.

    Some possible options that you may wish to consider:

    • Try rules_pycross which will run builds outside of a repository rule context
    • Make use of a private package registry / wheel-house / cheese-shop. If you only make use of pre-built wheels in your dependency closure, then bazel will not invoke the build+unpack behaviour of pip and you'll be able to use --only-binary :all: in your requirements.txt. In this configuration, bazel will simply download and unpack wheels and you will not need to be concerned with unmanaged build dependencies.
  3. changed the title [-]Support PEP 518[/-] [+]pip_parse: lockfile for build dependencies in repository rule[/+] on Jul 18, 2023
  4. keith commented on Jul 18, 2023

    @keith
    MemberAuthor

    In the linked thread the solution of using PIP_CONSTRAINT=foo.txt might actually be a good path forward for this. I verified that does work for this case.

  5. groodt commented on Jul 19, 2023

    @groodt
    Contributor

    Great! Yes, if that works for you, we may just consider that the recommended workaround and eventually close this issue.

  6. keith commented on Jul 19, 2023

    @keith
    MemberAuthor

    I'm not sure that's possible to integrate with rules_python today though

  7. github-actions commented on Jan 15, 2024

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
    Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

  8. github-actions commented on Feb 15, 2024

    @github-actions
    Contributor

    This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

  9. reopened this on Feb 16, 2024
  10. aignas commented on Feb 16, 2024

    @aignas
    Collaborator

    Reopening as the feature for including build deps in pip-tools got merged so the definition of done here could be to ensure that it is available in rules_python.

  11. added
    type: pippip/pypi integration
    and removed
    Can Close?Will close in 30 days if there is no new activity
    on Feb 16, 2024
  12. github-actions commented on Aug 14, 2024

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
    Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

  13. removed
    Can Close?Will close in 30 days if there is no new activity
    on Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions