Repository navigation
fix: use JVM SourceFile attribute to map .class files to their sources - #2345
chinyeungli wants to merge 2 commits into
Conversation
* Use "pytecode" (MIT-licensed) to extract the SourceFile attribute from .class files. Signed-off-by: Chin Yeung Li <tli@nexb.com>
SourceFile attribute to map .class files to their sourcesSigned-off-by: Chin Yeung Li <tli@nexb.com>
tdruez
left a comment
There was a problem hiding this comment.
@chinyeungli Using the SourceFile attribute is the right source of truth, more reliable than parsing class names from the sources as in #2012 and #2027. Two concerns, see also my inline comments:
- Is the
pytecodedependency absolutely needed? It is a Rust extension at version 0.1.0, first released in April by a single maintainer, and we use it to read one attribute through its internal names (sourcefile_index,str_bytes). The class file format is stable and documented (JVMS §4.1 and §4.7.10): reading the constant pool and skipping to the class attributes withstructshould be a few dozen lines inpipes/jvm.py, with no new dependency to maintain. To be confirmed. map_jvm_to_classruns once per JVM language, so each unmapped.classfile is parsed again in every language step that has sources on the from/ side. On deploy jars with thousands of shaded classes without sources, that adds up. What about reading theSourceFileonce in a dedicated step and storing it inextra_data, likefind_jvm_packagesdoes for the from/ side?
| if match: | ||
| _create_jvm_binary_relations( | ||
| to_resource=to_resource, | ||
| map_type=jvm_lang.binary_map_type, | ||
| extra_data={"from_source_root": f"{from_source_root}/"}, | ||
| from_resources=from_resources, | ||
| match=match, | ||
| jvm_lang=jvm_lang, | ||
| ) | ||
| return |
There was a problem hiding this comment.
This changes the behavior for Kotlin (.kt, .kts) and Grammar (.g, .g4): before, a miss on the first extension returned without trying the second one, and a match on the first went on to the second. Now the first matching extension wins. Intended? If so, it needs its own test.
| # file can produce several class files whose basenames differ from the | ||
| # source. See https://github.com/aboutcode-org/scancode.io/issues/1875 | ||
| # and https://github.com/aboutcode-org/scancode.io/issues/1993 | ||
| # The ``SourceFile attribute is only available in JVM .class files. For |
There was a problem hiding this comment.
Typo:
| # The ``SourceFile attribute is only available in JVM .class files. For | |
| # The ``SourceFile`` attribute is only available in JVM .class files. For |
| # The ClassReader.from_file() accepts str or pathlib.Path | ||
| # Ref https://smithtrenton.github.io/pytecode/pytecode/classfile.html#ClassReader.from_file |
There was a problem hiding this comment.
Those comments describe the library API, not why. I'd drop them.
| # ``sourcefile_index`` pointing to a CONSTANT_Utf8 entry in the pool. | ||
| source_file_attribute = None | ||
| for attribute in class_file.attributes: | ||
| if hasattr(attribute, "sourcefile_index"): |
There was a problem hiding this comment.
Is there no SourceFile attribute type in pytecode for an isinstance check? Relying on attribute names such as sourcefile_index and str_bytes of a 0.1.0 library is fragile.
| break | ||
|
|
||
| if source_file_attribute is None: | ||
| return None |
There was a problem hiding this comment.
return None is redundant and not consistent with the current codebase style. Same for the other ones below.
| except Exception: | ||
| # A broad catch is intentional here: malformed or truncated class | ||
| # files must not interrupt the pipeline. ``pytecode`` raises its own | ||
| # exception types (e.g. ``MalformedClassException``) in addition to | ||
| # standard ones, so enumerating them all would be fragile. | ||
| return None |
There was a problem hiding this comment.
Keep only the ClassReader.from_file call in the try and catch the pytecode parsing exception. With the whole block under except Exception, a bug in this code or a pytecode API change (renamed str_bytes for example) returns None for every class file: the fallback is silently disabled.
| # The JVM attribute should contain a bare filename, not a path (e.g. | ||
| # "ClusterShardingQuery.scala"). Reject path traversal, separators, and | ||
| # control characters that could corrupt logs or reports downstream. | ||
| if ( | ||
| not source_file | ||
| or source_file in (".", "..") | ||
| or any(char in source_file for char in ("/", "\\", "\x00", "\n", "\r", "\t")) | ||
| ): | ||
| return None |
There was a problem hiding this comment.
The value is only used as a lookup key in pathmap.find_paths: it is never opened, stored, logged or reported. A / or .. cannot lead to a traversal, it just does not match. I'd drop this check and its test.
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| path = Path(tmpdir) / "bad.class" | ||
| path.write_bytes(b"unused") |
There was a problem hiding this comment.
from_file is mocked so the file is never read: the temporary file is not needed, any path would do. Same in the tests below.
| mock_from_file.return_value.class_info = mock.Mock( | ||
| attributes=[], | ||
| constant_pool=[], | ||
| ) |
There was a problem hiding this comment.
This mock copies pytecode internals. A small fixture class compiled without debug info (javac -g:none drops SourceFile) would test the real case.
okay, let me try to use |
There are cases where a single source file (
.javaor.scala) can produce multiple class files whose basenames differ from that of the source file. To identify the corresponding source file, we use the JVMSourceFileattribute to map.classfiles back to their sources. To do this, we use the pytecode library (https://github.com/smithtrenton/pytecode) to extract theSourceFileattribute from.classfiles.The samples listed in the issues are resolved:
#1875

#1993

Issues
.classMay Differ from Source Filename - map_deploy_to_devel (scala) #1875Changes
Use pytecode to extract the
SourceFileattribute from.classfiles and map them to their corresponding source files.Checklist