You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some of our files contain generator of source code.
Looks like this:
def printHeader(dest):
year = datetime.datetime.now().year
print("# Copyright (C) %s, ACME, All Right Reserved." % year, file=dest)
print("#", file=dest)
print("# This file is subject to the terms and conditions of", file=dest)
print("# ACME Software License Agreement", file=dest)
print("# SPDX-License-Identifier: LicenseRef-ACME-Proprietary", file=dest)
print("#", file=dest)
print("# DO NOT EDIT MANUALLY", file=dest)
print("# This file was autogenerated by mighty_generate.py", file=dest)
The 5th line got detected by the spdx matcher, but the detected SPDX license is unknown
We do add LicenseRef-ACME-Proprietary in our custom SPDX license, but it is not recognised, probably the end of the line is taken as part of the full SPDX ID.
So LicenseRef-ACME-Proprietary", file=dest) will be reported as something unknown
But So LicenseRef-ACME-Proprietary would be reported as something unknown
Some solution elements:
fix the false positive detection for the few patterns of this code generator with a false positive rule? ... ? not sure.
do not collect SPDX-License-Identifiers when there is the word "print" just before or something that could make this code-like looking , but that could be dangerous
we need to find a way to report LicenseRef-something correctly when this is not a known license in scancode license dataset
provide a way to "fix" certain matches in a local, specific way with some config file. Here for instance
hi @pombredanne thanks for detalled comments. makes a lot of sense
Some comments:
But So LicenseRef-ACME-Proprietary would be reported as something unknown
LicenseRef-ACME-Proprietary is not reported as something unknown in my instance as I have custom rules that defines it.(a fix for #2471 that is not yet ready for upstream). I think this is the best way to solve it for me. This ensures that all SPDX licenses are correctly identified by the IP Management group.
We could report it as expected, but we would still need some kind unknown_spdx flag so that it can be reviewed.
fix the false positive detection for the few patterns of this code generator with a false positive rule?
I think this fits my problem and I will experiment on it for short term.
do not collect SPDX-License-Identifiers when there is the word "print"
I had a similar idea, but print is not universal. what is more universal are to me are quotes
I was thinking about skipping the identification if:
there are two of the same quotes (",',` ) in the line
spdx_expression = 'LicenseRef-ACME-Proprietary", file=dest)'
line = 'print("# SPDX-License-Identifier: LicenseRef-ACME-Proprietary", file=dest)'
The mere presence of a single quote may be enough to flag this as a false positive IMHO... BUT there could be cases (such as formatted markdown?) where this may not be a false positive?
Although it is not a license registered in SPDX, is there a way to print the License ID ("Sample-Proprietary" in the example above) if it is written according to the notation of the SPDX License?
If "key" extracted as "unknown-spdx", how about outputting "identifier" in "matched_rule" as License ID written?
Activity
pombredanne commentedon Apr 23, 2021
@tardyp thanks.
SPDX ids are detected if they start in the first few words of a line per https://github.com/nexB/scancode-toolkit/blob/9ff0880379aa68cc8d1c4e31cc9c206c6b8e9932/src/licensedcode/query.py#L370
So
print("# SPDX-License-Identifier: LicenseRef-ACME-Proprietary", file=dest)would be collected and detectedwe assume that everything after
SPDX-License-Identifier:is an SPDX license expression up to the end of the linehttps://github.com/nexB/scancode-toolkit/blob/9ff0880379aa68cc8d1c4e31cc9c206c6b8e9932/src/licensedcode/query.py#L390
when doing the detection there are a few cleanups applied to deal with some well known weird and common issues, and then the expression is parsed.... For now we do not consider unknown
LicenseRefas a valid license. Furthermore there is some trailing "jink" in file=destSee https://github.com/nexB/scancode-toolkit/blob/9ff0880379aa68cc8d1c4e31cc9c206c6b8e9932/src/licensedcode/match_spdx_lid.py
So
LicenseRef-ACME-Proprietary", file=dest)will be reported as something unknownBut So
LicenseRef-ACME-Proprietarywould be reported as something unknownSome solution elements:
tardyp commentedon Apr 23, 2021
hi @pombredanne thanks for detalled comments. makes a lot of sense
Some comments:
LicenseRef-ACME-Proprietary is not reported as something unknown in my instance as I have custom rules that defines it.(a fix for #2471 that is not yet ready for upstream). I think this is the best way to solve it for me. This ensures that all SPDX licenses are correctly identified by the IP Management group.
We could report it as expected, but we would still need some kind unknown_spdx flag so that it can be reviewed.
I think this fits my problem and I will experiment on it for short term.
I had a similar idea, but print is not universal. what is more universal are to me are quotes
I was thinking about skipping the identification if:
",',`) in the lineSPDX-License-Identifieris between those quotespombredanne commentedon Apr 23, 2021
Yes, quotes would be better... we are mostly ignoring quotes.
Except for SPDX License Identifiers, where we can access to the raw text. In https://github.com/nexB/scancode-toolkit/blob/bb044200ae86770f9bb01560c0033037ee18b947/src/licensedcode/query.py#L441 we could likely add a test on the content.
If I use this one liner as a test file
print("# SPDX-License-Identifier: LicenseRef-ACME-Proprietary", file=dest)Then in https://github.com/nexB/scancode-toolkit/blob/bb044200ae86770f9bb01560c0033037ee18b947/src/licensedcode/query.py#L441
we have these values to use as needed:
The mere presence of a single quote may be enough to flag this as a false positive IMHO... BUT there could be cases (such as formatted markdown?) where this may not be a false positive?
soimkim commentedon Jun 19, 2021
I also have a similar issue.
I want to print the items marked with SPDX License, but the ScanCode json result is output as follows.
Information written to the file:
# SPDX-License-Identifier: LicenseRef-Sample-ProprietaryScanCode json result :
"licenses": [ { "key": "unknown-spdx", "score": 83.33, "name": "Unknown SPDX license detected but not recognized", "short_name": "unknown SPDX", "category": "Unstated License", "is_exception": false, "owner": "Unspecified", "homepage_url": null, "text_url": "", "reference_url": "https://scancode-licensedb.aboutcode.org/unknown-spdx", "scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/unknown-spdx.LICENSE", "scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/unknown-spdx.yml", "spdx_license_key": "LicenseRef-scancode-unknown-spdx", "spdx_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/unknown-spdx.LICENSE", "start_line": 3, "end_line": 3, "matched_rule": { "identifier": "spdx-license-identifier: unknown-spdx", "license_expression": "unknown-spdx", "licenses": [ "unknown-spdx" ], ...Although it is not a license registered in SPDX, is there a way to print the License ID ("Sample-Proprietary" in the example above) if it is written according to the notation of the SPDX License?
If "key" extracted as "unknown-spdx", how about outputting "identifier" in "matched_rule" as License ID written?
pombredanne commentedon Jun 19, 2021
This is not what you ask but at least if you use the option
--license-textyou will also see the original expression:soimkim commentedon Jun 21, 2021
Dear @pombredanne ,
Thanks for the quick reply.
In my case, I just load the value of matched_text.
pombredanne commentedon Aug 20, 2021
@soimkim I entered a new issue for your report as I think it may be best tracked separately. Please see #2650