You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PyExecutableInfo.runfiles_without_exe includes full interpreter exe with standard library #3181
This has probably been broken since it was introduced.
Description
I'd like to build container images where the interpreter + standard library is part of a base image and the "app" layer contains just my python code (and explicit data dependencies).
I tried to write a small rule that extracts the necessary info from the provider to avoid pulling in the Python interpreter itself.
Looking at the output, the runfiles include the full interpreter, which is unexpected.
In the subdirectory e2e/python, I have a clean example of what I mean with a separate MODULE.bazel and a clean reproducer. e2e/python/BUILD.bazel contains a py_binary and my custom rule for creating a "light" binary (using PyExecutableInfo). e2e/python/extract_python_files/defs.bzl contains the rule I wrote to extract only the data I want.
Here are the repro steps:
bazel build :app
find bazel-bin/app.runfiles/
🔥 Exception or Error
You can see may files that shouldn't be there, including bazel-bin/app.runfiles/rules_python++python+python_3_13_x86_64-unknown-linux-gnu/bin/python3.13
🌍 Your Environment
Operating System:
NixOS 24.11 on amd64
Output of bazel version:
Bazelisk version: development
Build label: 8.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Jun 30 16:23:40 2025 (1751300620)
Build timestamp: 1751300620
Build timestamp as int: 1751300620
Rules_python version:
bazel_dep(name = "rules_python", version = "1.5.3")
All of this makes me think that my assumptions about the intended use of PyExecutableInfo.runfiles_without_exe might be incorrect, and this is actually expected behavior. If this is the case, I'd like to turn this into a feature request instead:
For the purpose of creating container image layers (and other types of bundling), could we provide a field in some provider that contains the files of the application without the toolchain and standard library? (Bonus points if we can separate between third-party deps and normal application code). This would make it much easier for the larger ecosystem to bundle Python apps efficiently.
In general, I think it is a valid request, if users are using non-hermetic toolchains but want to build docker images, there still should be a way. Which toolchain are you using?
Ah, so you are using the hermetic toolchain, but you want to replace it during packaging? Are you concerned that the thing you are packaging is not strictly the same thing as the one you are testing?
How do you imagine the Python bootstrap to work? Do you use bootstrap_impl=script?
In theory, you could still use tar.bzl even today, where you have the mtree manifest that you can modify and just drop the runtime files and replace python with a symlink to the filesystem Python.
Ah, so you are using the hermetic toolchain, but you want to replace it during packaging?
That's correct. I want to use the same Python version in Bazel and in the container, but for efficiency, I want to put the toolchain in a separate container image layer or have it preinstalled in the base image (it doesn't change often and can be shared by multiple Python apps). I didn't think about the setting for bootstrap_impl yet.
The container image does have a "system Python" in the sense that the base image I'm layering upon contains a python interpreter.
Custom filtering with mtree manifests and tar.bzl does sound like an option. I'm currently building a more optimized way to build layers in rules_img that doesn't have any method for filtering runfiles so far, so that wouldn't work (yet).
I'm also considering just writing my own Aspect that collects srcs and data from py_library and friends, but I'm sure it's not easy to do this correctly.
The origin of runfiles_without_exe is from Google, where the entry point executable and the runtime are one in the same. The goal being, the original executable won't work as-is in the intended deployment environment, so a new one (with a different bootstrap, but same runtime) must be derived.
It's a similar intent as what you have, though.
It sounds like what you want is something like runfiles_without_runtime. You don't want the runtime, but probably want the same bootstrap.
That sounds mostly reasonable. Hm. However. The big catch is the bootstrap and runtime are fairly closely coupled when an in-build (eg hermetic) runtime is used -- the bootstrap assumes the interpreter is in the runfiles, but in your case, you want it to come from outside. Hm, now it's sounding contradictory. (Such isn't the case for a platform (non in-build) runtime, because it assumes the system provides it somehow).
What might work better is to switch the tool chain when building for your deployment environment. Switch it to a runtime_env tool chain, or equivalent, which assumes it has to get python from the environment. Though, setting that up is a bit of a pain. A transition or config change is also annoying, perf wise. Hm.
I'd be ok with adding a runfiles_without_runtime field, or changing runfiles_without_exe to also exclude the python runtime, or having separate fields (base binary runfiles, exe, py runtime). You'll probably end up needing to generate your own bootstrap, though, which is tedious.
PyExecutableInfo exposes extra info now that should make OP possible. In particular, there are separate fields for the app's runfiles and the interpreter files.
Activity
malt3 commentedon Aug 16, 2025
I tried to understand why this happens.
Runfiles containing the toolchain are collected in
_get_runtime_detailsasruntime_files:rules_python/python/private/py_executable.bzl
Lines 1247 to 1258 in cda5877
The toolchain (including
bin/pythonand the standard library) are passed into_get_base_runfiles_for_binaryasextra_common_runfiles.rules_python/python/private/py_executable.bzl
Lines 1136 to 1141 in cda5877
Those get added to
common_runfiles:rules_python/python/private/py_executable.bzl
Line 1404 in cda5877
.. which are finally used as
runfiles_without_exe:rules_python/python/private/py_executable.bzl
Line 1436 in cda5877
All of this makes me think that my assumptions about the intended use of
PyExecutableInfo.runfiles_without_exemight be incorrect, and this is actually expected behavior. If this is the case, I'd like to turn this into a feature request instead:For the purpose of creating container image layers (and other types of bundling), could we provide a field in some provider that contains the files of the application without the toolchain and standard library? (Bonus points if we can separate between third-party deps and normal application code). This would make it much easier for the larger ecosystem to bundle Python apps efficiently.
aignas commentedon Aug 17, 2025
In general, I think it is a valid request, if users are using non-hermetic toolchains but want to build docker images, there still should be a way. Which toolchain are you using?
malt3 commentedon Aug 17, 2025
This is the Python toolchain setup in
MODULE.bazel:aignas commentedon Aug 17, 2025
Ah, so you are using the hermetic toolchain, but you want to replace it during packaging? Are you concerned that the thing you are packaging is not strictly the same thing as the one you are testing?
How do you imagine the Python bootstrap to work? Do you use
bootstrap_impl=script?In theory, you could still use
tar.bzleven today, where you have themtreemanifest that you can modify and just drop the runtime files and replace python with a symlink to the filesystem Python.malt3 commentedon Aug 17, 2025
That's correct. I want to use the same Python version in Bazel and in the container, but for efficiency, I want to put the toolchain in a separate container image layer or have it preinstalled in the base image (it doesn't change often and can be shared by multiple Python apps). I didn't think about the setting for
bootstrap_implyet.The container image does have a "system Python" in the sense that the base image I'm layering upon contains a python interpreter.
Custom filtering with
mtreemanifests andtar.bzldoes sound like an option. I'm currently building a more optimized way to build layers in rules_img that doesn't have any method for filtering runfiles so far, so that wouldn't work (yet).malt3 commentedon Aug 17, 2025
I'm also considering just writing my own Aspect that collects
srcsanddatafrompy_libraryand friends, but I'm sure it's not easy to do this correctly.aignas commentedon Aug 17, 2025
FYI, The
tar.bzlandrules_ociexample shows how to put the interpreter in a separate layer.rickeylev commentedon Aug 17, 2025
The origin of runfiles_without_exe is from Google, where the entry point executable and the runtime are one in the same. The goal being, the original executable won't work as-is in the intended deployment environment, so a new one (with a different bootstrap, but same runtime) must be derived.
It's a similar intent as what you have, though.
It sounds like what you want is something like runfiles_without_runtime. You don't want the runtime, but probably want the same bootstrap.
That sounds mostly reasonable. Hm. However. The big catch is the bootstrap and runtime are fairly closely coupled when an in-build (eg hermetic) runtime is used -- the bootstrap assumes the interpreter is in the runfiles, but in your case, you want it to come from outside. Hm, now it's sounding contradictory. (Such isn't the case for a platform (non in-build) runtime, because it assumes the system provides it somehow).
What might work better is to switch the tool chain when building for your deployment environment. Switch it to a runtime_env tool chain, or equivalent, which assumes it has to get python from the environment. Though, setting that up is a bit of a pain. A transition or config change is also annoying, perf wise. Hm.
I'd be ok with adding a runfiles_without_runtime field, or changing runfiles_without_exe to also exclude the python runtime, or having separate fields (base binary runfiles, exe, py runtime). You'll probably end up needing to generate your own bootstrap, though, which is tedious.
build(deps): bump urllib3 from 2.0.7 to 2.2.2 in /tests/multiple_inpu…
FrankPortman commentedon Nov 8, 2025
@malt3 this sounds related to #3324 WDYT? I can share the heuristic I've settled on for removing the "extra stuff".
rickeylev commentedon Feb 7, 2026
PyExecutableInfo exposes extra info now that should make OP possible. In particular, there are separate fields for the app's runfiles and the interpreter files.
See the docs for more info:
https://rules-python.readthedocs.io/en/latest/api/rules_python/python/py_executable_info.html#PyExecutableInfo
docs: mention PyRuntimeInfo in PyExecutableInfo (#3573)
docs: mention PyRuntimeInfo in PyExecutableInfo (bazel-contrib#3573)