Repository navigation
Follow and resolve license references - #2237
AyanSinhaMahapatra wants to merge 2 commits into
Conversation
Reference: #1497 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
Also populate and resolve package/resource level license detections from resource level license following. Reference: #1497 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
9892500 to
959c2f6
Compare
tdruez
left a comment
There was a problem hiding this comment.
@AyanSinhaMahapatra See my comments, the main issue is the resolution done on the deduplicated DiscoveredLicense instead of per resource. A few higher-level points:
- I'd make the reference resolution its own pipeline step rather than adding it to
collect_and_create_license_detections, so it is visible and documented as a step. - The behavior differs across pipelines: in
analyze_root_filesystemand the Docker pipelines, manifests are not file scanned, so the package part has no effect there. scan_maven_packagenow also runsupdate_package_license_from_resource_if_missing, which serves the same purpose. We should keep a single mechanism.- Most of the new functions are not unit tested, only the two entry points are covered through similar end-to-end tests.
| @property | ||
| def base_name(self): | ||
| base_name, _extension = splitext_name(self.name, is_file=self.is_file) | ||
| return base_name |
There was a problem hiding this comment.
Nothing in ScanCode.io calls this, I guess it's required by the toolkit is_legal_or_readme. A short comment saying so would keep it from being removed later.
| Update the license expressions for a discovered package from it's license | ||
| detections. |
There was a problem hiding this comment.
Copy/paste: this is a resource, not a package. Also "its", not "it's".
| Update the license expressions for a discovered package from it's license | |
| detections. | |
| Update the license expressions for this resource from its license | |
| detections. |
| def update_license_expressions(self): | ||
| """ | ||
| Update the license expressions for a discovered package from it's license | ||
| detections. | ||
| """ | ||
| if not self.license_detections: | ||
| return | ||
|
|
||
| license_expressions = [ | ||
| detection.get("license_expression") for detection in self.license_detections | ||
| ] | ||
| self.detected_license_expression = combine_expressions( | ||
| expressions=license_expressions, | ||
| relation="AND", | ||
| unique=True, | ||
| ) | ||
| self.detected_license_expression_spdx = str( | ||
| build_spdx_license_expression( | ||
| license_expression=self.detected_license_expression | ||
| ) | ||
| ) | ||
| self.save() |
There was a problem hiding this comment.
This duplicates DiscoveredPackage.update_license_expressions, only the field names change. It also relies on the caller setting license_detections on the instance beforehand, then does a full save(). What about a shared function for the expressions, and a method that takes the detections and uses update()
The str() calls are not needed: combine_expressions and build_spdx_license_expression already return a str.
| def update_license_expressions(self): | ||
| """ | ||
| Update the license expressions for a discovered package from it's license | ||
| detections. | ||
| """ | ||
| if not self.license_detections: | ||
| return | ||
|
|
||
| license_expressions = [ | ||
| detection.get("license_expression") for detection in self.license_detections | ||
| ] | ||
| self.declared_license_expression = combine_expressions( | ||
| expressions=license_expressions, | ||
| relation="AND", | ||
| unique=True, | ||
| ) | ||
| self.declared_license_expression_spdx = str( | ||
| build_spdx_license_expression( | ||
| license_expression=self.declared_license_expression | ||
| ) | ||
| ) | ||
| self.save() |
| cls.scan_for_files, | ||
| cls.scan_for_application_packages, |
There was a problem hiding this comment.
Why this change? Manifests are now file scanned too, which is where the new data in the expected *_scan_codebase.json files comes from. Resources flagged by scan_for_files as scanned-with-error (timeout) or ignored-by-max-file-size are also no longer scanned for packages. A package-lock.json over SCAN_MAX_FILE_SIZE would lose all its dependencies. The class docstring ("does not further scan the files contained in a package") is out of sync as well.
| """ | ||
| Try to find and return a referenced resource using database calls instead of a | ||
| codebase walk first and fall back only if a resource is not found. | ||
| """ |
There was a problem hiding this comment.
The docstring does not match the code: there is no fallback to a codebase walk.
| if referenced_resource: | ||
| return referenced_resource | ||
|
|
||
| for package in resource.discovered_packages.all(): |
There was a problem hiding this comment.
Unlike the toolkit find_referenced_resource, the codebase root is not looked up. Is that intended? The PR description mentions feature parity with the toolkit.
| self.assertEqual("package", file1.parent_path) | ||
|
|
||
| def test_scanpipe_scancode_follow_and_resolve_referenced_licenses_resources(self): | ||
| project1 = Project.objects.create(name="Utility: PurlDB") |
There was a problem hiding this comment.
Leftover? The project name looks copy/pasted.
| archive_location = ( | ||
| self.data / "scancode" / "unknown-ref-to-key-file-root.tar.xz" | ||
| ) | ||
| copy_input(archive_location, project1.codebase_path) | ||
| scancode.extract_archives(project1.codebase_path) | ||
| collect_and_create_codebase_resources(project1) | ||
| scancode.scan_for_files(project1) | ||
| scancode.scan_for_application_packages(project1, assemble=True) | ||
| scancode.collect_and_create_license_detections(project1) | ||
| scancode.follow_and_resolve_referenced_licenses_in_resources(project1) | ||
| scancode.follow_and_resolve_referenced_licenses_in_packages(project1) |
There was a problem hiding this comment.
The same 8-line setup is repeated in the 4 tests. I'd move it to a helper taking the archive name and the assemble value.
| file1 = project1.codebaseresources.get( | ||
| path="unknown-ref-to-key-file-root.tar.xz-extract/unknown-ref-to-key-file-root/esprima-compare.js" | ||
| ) | ||
| self.assertEqual(file1.detected_license_expression, "mit") |
There was a problem hiding this comment.
Expected value first, as in the rest of this module. Same for the other new assertions.
Reference:
Issues
Changes
Follows license references from packages/resources like See License in License.txt, and add licenses to packages from file level license detections, try to resolve license detection issues.
Enables this for scan_codebase and all docker/rootfs related pipelines, maven specific pipeline.
scan_single_packagealready has this functionality from scancode-toolkit.After this, scancode.io should have feature-parity with all the scancode-toolkit license detection improvements on finding and trying to resolve unique license detection issues.
Checklist