You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Congrats on the 1.8.0 release! I just tried it with venv by adding below to my .bazelrc (I am using bazel 8.5.0 on Linux):
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes
then bazel run a trial py_binary:
bazel run //util/validate_python_ml_requirements:validate_python_ml_requirements
(to see if tensorflow can finally find all cuda libraries via venv)
Thanks for the clarification!
So the "_validate_python_ml_requirements.venv" directory inside bazel-bin/util/validate_python_ml_requirements is not used? Can we not create it to avoid confusion if it is not used and has wrong symlinks in it?
Unfortunately no, we can't prevent it; it's part of how bazel creates and materializes files. I do agree their visibility is annoying and confusing (I've seen people (myself included 😅) use them, not realizing they're not technically the "correct" file path to use for their situation). (Trivia: in the past, such generated files were put in "bazel-genfiles", but that's been being phased out for a decade).
The files are used, just in an indirect way.
The bazel-bin/ files are what bazel calls the "exec root". Whenever a generate file is created, it gets materialized in the bazel exec root (i.e its the actual file that data gets written to). It's like the "backing store" for generated file content. Later, when the file is included in a binary's runfiles, it shows up as a symlink from bazel-bin/bin.runfiles/whatever.txt -> bazel-bin/whatever.txt.
I had a colleague who decided to ignore rules_python, write his own python rules, and in that process not use .runfiles and just use bazel created files.
That ended badly 😂 🤦
(one random problem was runfiles are always kept clean by bazel, e.g. no stale file, but output tree can contain any random files)
@rickeylev if the correct venv directory we should use is inside runfiles directory, can we update the runfiles_manifest file point to that venv directory instead? We depend on the runfiles_manifest file to pack the files for the py image layer tar.
can we update the runfiles_manifest file point to that venv directory instead?
I am looking at the manifest and I am not sure what you mean by that. We don't do anything in particular to the runfiles manifest and I'd be interested to learn if there is a way for us to manipulate that in some way.
While we specify the set of runfiles, we don't control the exact output format, that's handled by Bazel, so no, we can't make the runfiles manifest point anywhere specific.
we depend on runfiles_manifest to pack the files for py image layer tar
How so? It should work mostly the same as before. The main difference I can think of is the raw symlinks used by the venv. The runfiles_manifest won't carry information to distinguish "must be symlink" from "happens to be a symlink". Is that the problem?
I was doing something like this to create a bsdtar mtree spec from the runfiles_manifest. Since the manifest file now has invalid links, the bsdtar using the resulting mtree spec complains and fails.
This is an upstream issue for the tar.bzl because during the mtree manifest creation, the ruleset does not understand that the files should be preserved as relative symlinks.
At some point bazel-contrib/tar.bzl#16 (comment) used to work for me but later I switched to rules_pkg as it has better performance and it understands symlinks.
I'll close this issue as there is little we can do here.
For now, the mtree manifest has to be post-processed to support relative symlinks. The rules_img ruleset might be supporting symlinks as well as I have noted in #3117 description.
Activity
rickeylev commentedon Jan 17, 2026
The relative symlinks created within the venv are only valid relative to the runfiles location.
So the path to access it would be e.g.
bazel-bin/util/validate_python_ml_requirements/validate_python_ml_requirements.runfiles/_main/util/validate_python_ml_requirements/validate_python_ml_requirements.venv/bin/python3gfrankliu commentedon Jan 17, 2026
Thanks for the clarification!
So the "_validate_python_ml_requirements.venv" directory inside bazel-bin/util/validate_python_ml_requirements is not used? Can we not create it to avoid confusion if it is not used and has wrong symlinks in it?
rickeylev commentedon Jan 17, 2026
Unfortunately no, we can't prevent it; it's part of how bazel creates and materializes files. I do agree their visibility is annoying and confusing (I've seen people (myself included 😅) use them, not realizing they're not technically the "correct" file path to use for their situation). (Trivia: in the past, such generated files were put in "bazel-genfiles", but that's been being phased out for a decade).
The files are used, just in an indirect way.
The
bazel-bin/files are what bazel calls the "exec root". Whenever a generate file is created, it gets materialized in the bazel exec root (i.e its the actual file that data gets written to). It's like the "backing store" for generated file content. Later, when the file is included in a binary's runfiles, it shows up as a symlink frombazel-bin/bin.runfiles/whatever.txt -> bazel-bin/whatever.txt.shayanhoshyari commentedon Jan 19, 2026
I had a colleague who decided to ignore
rules_python, write his own python rules, and in that process not use.runfilesand just use bazel created files.That ended badly 😂 🤦
(one random problem was runfiles are always kept clean by bazel, e.g. no stale file, but output tree can contain any random files)
gfrankliu commentedon Jan 19, 2026
@rickeylev if the correct venv directory we should use is inside runfiles directory, can we update the runfiles_manifest file point to that venv directory instead? We depend on the runfiles_manifest file to pack the files for the py image layer tar.
aignas commentedon Jan 20, 2026
I am looking at the manifest and I am not sure what you mean by that. We don't do anything in particular to the runfiles manifest and I'd be interested to learn if there is a way for us to manipulate that in some way.
rickeylev commentedon Jan 20, 2026
While we specify the set of runfiles, we don't control the exact output format, that's handled by Bazel, so no, we can't make the runfiles manifest point anywhere specific.
How so? It should work mostly the same as before. The main difference I can think of is the raw symlinks used by the venv. The runfiles_manifest won't carry information to distinguish "must be symlink" from "happens to be a symlink". Is that the problem?
gfrankliu commentedon Jan 20, 2026
I was doing something like this to create a bsdtar mtree spec from the runfiles_manifest. Since the manifest file now has invalid links, the bsdtar using the resulting mtree spec complains and fails.
aignas commentedon Jan 21, 2026
This is an upstream issue for the
tar.bzlbecause during themtreemanifest creation, the ruleset does not understand that the files should be preserved as relative symlinks.At some point bazel-contrib/tar.bzl#16 (comment) used to work for me but later I switched to
rules_pkgas it has better performance and it understands symlinks.I'll close this issue as there is little we can do here.
For now, the
mtreemanifest has to be post-processed to support relative symlinks. Therules_imgruleset might be supporting symlinks as well as I have noted in #3117 description.