Skip to content

Follow and resolve license references - #2237

Open
AyanSinhaMahapatra wants to merge 2 commits into
mainfrom
follow-license-references
Open

AyanSinhaMahapatra wants to merge 2 commits into
mainfrom
follow-license-references

Conversation

@AyanSinhaMahapatra

@AyanSinhaMahapatra AyanSinhaMahapatra commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Reference:

Issues

Changes

Follows license references from packages/resources like See License in License.txt, and add licenses to packages from file level license detections, try to resolve license detection issues.

Enables this for scan_codebase and all docker/rootfs related pipelines, maven specific pipeline. scan_single_package already has this functionality from scancode-toolkit.
After this, scancode.io should have feature-parity with all the scancode-toolkit license detection improvements on finding and trying to resolve unique license detection issues.

Checklist

  • I have read the contributing guidelines
  • I have linked an existing issue above
  • I have added unit tests covering the new code
  • I have reviewed and understood every line of this PR

Reference: #1497
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra
AyanSinhaMahapatra marked this pull request as draft September 21, 2026 14:47
@AyanSinhaMahapatra
AyanSinhaMahapatra marked this pull request as ready for review October 13, 2026 05:55
Also populate and resolve package/resource level license
detections from resource level license following.

Reference: #1497
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

@tdruez tdruez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AyanSinhaMahapatra See my comments, the main issue is the resolution done on the deduplicated DiscoveredLicense instead of per resource. A few higher-level points:

  • I'd make the reference resolution its own pipeline step rather than adding it to collect_and_create_license_detections, so it is visible and documented as a step.
  • The behavior differs across pipelines: in analyze_root_filesystem and the Docker pipelines, manifests are not file scanned, so the package part has no effect there.
  • scan_maven_package now also runs update_package_license_from_resource_if_missing, which serves the same purpose. We should keep a single mechanism.
  • Most of the new functions are not unit tested, only the two entry points are covered through similar end-to-end tests.

Comment thread scanpipe/models.py
Comment on lines +2992 to +2995
@property
def base_name(self):
base_name, _extension = splitext_name(self.name, is_file=self.is_file)
return base_name

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing in ScanCode.io calls this, I guess it's required by the toolkit is_legal_or_readme. A short comment saying so would keep it from being removed later.

Comment thread scanpipe/models.py
Comment on lines +3269 to +3270
Update the license expressions for a discovered package from it's license
detections.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy/paste: this is a resource, not a package. Also "its", not "it's".

Suggested change
Update the license expressions for a discovered package from it's license
detections.
Update the license expressions for this resource from its license
detections.

Comment thread scanpipe/models.py
Comment on lines +3267 to +3288
def update_license_expressions(self):
"""
Update the license expressions for a discovered package from it's license
detections.
"""
if not self.license_detections:
return

license_expressions = [
detection.get("license_expression") for detection in self.license_detections
]
self.detected_license_expression = combine_expressions(
expressions=license_expressions,
relation="AND",
unique=True,
)
self.detected_license_expression_spdx = str(
build_spdx_license_expression(
license_expression=self.detected_license_expression
)
)
self.save()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates DiscoveredPackage.update_license_expressions, only the field names change. It also relies on the caller setting license_detections on the instance beforehand, then does a full save(). What about a shared function for the expressions, and a method that takes the detections and uses update()

The str() calls are not needed: combine_expressions and build_spdx_license_expression already return a str.

Comment thread scanpipe/models.py
Comment on lines +3929 to +3950
def update_license_expressions(self):
"""
Update the license expressions for a discovered package from it's license
detections.
"""
if not self.license_detections:
return

license_expressions = [
detection.get("license_expression") for detection in self.license_detections
]
self.declared_license_expression = combine_expressions(
expressions=license_expressions,
relation="AND",
unique=True,
)
self.declared_license_expression_spdx = str(
build_spdx_license_expression(
license_expression=self.declared_license_expression
)
)
self.save()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above. Duplicated.

Comment on lines 46 to +47
cls.scan_for_files,
cls.scan_for_application_packages,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change? Manifests are now file scanned too, which is where the new data in the expected *_scan_codebase.json files comes from. Resources flagged by scan_for_files as scanned-with-error (timeout) or ignored-by-max-file-size are also no longer scanned for packages. A package-lock.json over SCAN_MAX_FILE_SIZE would lose all its dependencies. The class docstring ("does not further scan the files contained in a package") is out of sync as well.

Comment on lines +1291 to +1294
"""
Try to find and return a referenced resource using database calls instead of a
codebase walk first and fall back only if a resource is not found.
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring does not match the code: there is no fallback to a codebase walk.

if referenced_resource:
return referenced_resource

for package in resource.discovered_packages.all():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike the toolkit find_referenced_resource, the codebase root is not looked up. Is that intended? The PR description mentions feature parity with the toolkit.

self.assertEqual("package", file1.parent_path)

def test_scanpipe_scancode_follow_and_resolve_referenced_licenses_resources(self):
project1 = Project.objects.create(name="Utility: PurlDB")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover? The project name looks copy/pasted.

Comment on lines +752 to +762
archive_location = (
self.data / "scancode" / "unknown-ref-to-key-file-root.tar.xz"
)
copy_input(archive_location, project1.codebase_path)
scancode.extract_archives(project1.codebase_path)
collect_and_create_codebase_resources(project1)
scancode.scan_for_files(project1)
scancode.scan_for_application_packages(project1, assemble=True)
scancode.collect_and_create_license_detections(project1)
scancode.follow_and_resolve_referenced_licenses_in_resources(project1)
scancode.follow_and_resolve_referenced_licenses_in_packages(project1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same 8-line setup is repeated in the 4 tests. I'd move it to a helper taking the archive name and the assemble value.

file1 = project1.codebaseresources.get(
path="unknown-ref-to-key-file-root.tar.xz-extract/unknown-ref-to-key-file-root/esprima-compare.js"
)
self.assertEqual(file1.detected_license_expression, "mit")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expected value first, as in the rest of this module. Same for the other new assertions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mass-scan: Design and create code to spot incorrect licenses

2 participants