Skip to content

fix: use JVM SourceFile attribute to map .class files to their sources - #2345

Open
chinyeungli wants to merge 2 commits into
mainfrom
1875_d2d_scala_issue
Open

chinyeungli wants to merge 2 commits into
mainfrom
1875_d2d_scala_issue

Conversation

@chinyeungli

Copy link
Copy Markdown
Contributor

There are cases where a single source file (.java or .scala) can produce multiple class files whose basenames differ from that of the source file. To identify the corresponding source file, we use the JVM SourceFile attribute to map .class files back to their sources. To do this, we use the pytecode library (https://github.com/smithtrenton/pytecode) to extract the SourceFile attribute from .class files.

The samples listed in the issues are resolved:

#1875
Screenshot 2026-11-17 134705

#1993
Screenshot 2026-11-17 134745

Screenshot 2026-11-17 134806

Issues

Changes

Use pytecode to extract the SourceFile attribute from .class files and map them to their corresponding source files.

Checklist

  • I have read the contributing guidelines
  • I have linked an existing issue above
  • I have added unit tests covering the new code
  • I have reviewed and understood every line of this PR

 * Use "pytecode" (MIT-licensed) to extract the SourceFile attribute from .class files.

Signed-off-by: Chin Yeung Li <tli@nexb.com>
@chinyeungli chinyeungli changed the title Use JVM SourceFile attribute to map .class files to their sources fix: use JVM SourceFile attribute to map .class files to their sources Nov 17, 2026
Signed-off-by: Chin Yeung Li <tli@nexb.com>
@chinyeungli
chinyeungli requested a review from tdruez November 17, 2026 07:02

@tdruez tdruez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chinyeungli Using the SourceFile attribute is the right source of truth, more reliable than parsing class names from the sources as in #2012 and #2027. Two concerns, see also my inline comments:

  1. Is the pytecode dependency absolutely needed? It is a Rust extension at version 0.1.0, first released in April by a single maintainer, and we use it to read one attribute through its internal names (sourcefile_index, str_bytes). The class file format is stable and documented (JVMS §4.1 and §4.7.10): reading the constant pool and skipping to the class attributes with struct should be a few dozen lines in pipes/jvm.py, with no new dependency to maintain. To be confirmed.
  2. map_jvm_to_class runs once per JVM language, so each unmapped .class file is parsed again in every language step that has sources on the from/ side. On deploy jars with thousands of shaded classes without sources, that adds up. What about reading the SourceFile once in a dedicated step and storing it in extra_data, like find_jvm_packages does for the from/ side?

Comment thread scanpipe/pipes/d2d.py
Comment on lines +192 to +199
if match:
_create_jvm_binary_relations(
to_resource=to_resource,
map_type=jvm_lang.binary_map_type,
extra_data={"from_source_root": f"{from_source_root}/"},
from_resources=from_resources,
match=match,
jvm_lang=jvm_lang,
)
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes the behavior for Kotlin (.kt, .kts) and Grammar (.g, .g4): before, a miss on the first extension returned without trying the second one, and a match on the first went on to the second. Now the first matching extension wins. Intended? If so, it needs its own test.

Comment thread scanpipe/pipes/d2d.py
# file can produce several class files whose basenames differ from the
# source. See https://github.com/aboutcode-org/scancode.io/issues/1875
# and https://github.com/aboutcode-org/scancode.io/issues/1993
# The ``SourceFile attribute is only available in JVM .class files. For

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo:

Suggested change
# The ``SourceFile attribute is only available in JVM .class files. For
# The ``SourceFile`` attribute is only available in JVM .class files. For

Comment thread scanpipe/pipes/jvm.py
Comment on lines +302 to +303
# The ClassReader.from_file() accepts str or pathlib.Path
# Ref https://smithtrenton.github.io/pytecode/pytecode/classfile.html#ClassReader.from_file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those comments describe the library API, not why. I'd drop them.

Comment thread scanpipe/pipes/jvm.py
# ``sourcefile_index`` pointing to a CONSTANT_Utf8 entry in the pool.
source_file_attribute = None
for attribute in class_file.attributes:
if hasattr(attribute, "sourcefile_index"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no SourceFile attribute type in pytecode for an isinstance check? Relying on attribute names such as sourcefile_index and str_bytes of a 0.1.0 library is fragile.

Comment thread scanpipe/pipes/jvm.py
break

if source_file_attribute is None:
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return None is redundant and not consistent with the current codebase style. Same for the other ones below.

Comment thread scanpipe/pipes/jvm.py
Comment on lines +330 to +335
except Exception:
# A broad catch is intentional here: malformed or truncated class
# files must not interrupt the pipeline. ``pytecode`` raises its own
# exception types (e.g. ``MalformedClassException``) in addition to
# standard ones, so enumerating them all would be fragile.
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep only the ClassReader.from_file call in the try and catch the pytecode parsing exception. With the whole block under except Exception, a bug in this code or a pytecode API change (renamed str_bytes for example) returns None for every class file: the fallback is silently disabled.

Comment thread scanpipe/pipes/jvm.py
Comment on lines +337 to +345
# The JVM attribute should contain a bare filename, not a path (e.g.
# "ClusterShardingQuery.scala"). Reject path traversal, separators, and
# control characters that could corrupt logs or reports downstream.
if (
not source_file
or source_file in (".", "..")
or any(char in source_file for char in ("/", "\\", "\x00", "\n", "\r", "\t"))
):
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The value is only used as a lookup key in pathmap.find_paths: it is never opened, stored, logged or reported. A / or .. cannot lead to a traversal, it just does not match. I'd drop this check and its test.

Comment on lines +236 to +238
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / "bad.class"
path.write_bytes(b"unused")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from_file is mocked so the file is never read: the temporary file is not needed, any path would do. Same in the tests below.

Comment on lines +264 to +267
mock_from_file.return_value.class_info = mock.Mock(
attributes=[],
constant_pool=[],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mock copies pytecode internals. A small fixture class compiled without debug info (javac -g:none drops SourceFile) would test the real case.

@chinyeungli

Copy link
Copy Markdown
Contributor Author

Is the pytecode dependency absolutely needed? It is a Rust extension at version 0.1.0, first released in April by a single maintainer, and we use it to read one attribute through its internal names (sourcefile_index, str_bytes). The class file format is stable and documented (JVMS §4.1 and §4.7.10): reading the constant pool and skipping to the class attributes with struct should be a few dozen lines in pipes/jvm.py, with no new dependency to maintain. To be confirmed.

okay, let me try to use struct and remove the need of using pytecode

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants