Repository navigation
Description
Context
We are reporting too many false positive licenses. We need to fix this!
Problem
There are several false cases, yet they boil down to these types:
-
False detection of very short and weak license detection rules detected exactly such as:
- a URL or a project name such as a URL to a well known AGPL-licensed which is not always a sign of AGPL as in False positive AGPL detection from a mere URL #2877
- the detection of the word
GPLin a binary Tracing "Start Line" of ScanCode report back to the Binary file. #2874 - the detection of longer
may not be modifiedin False-positiveproprietary-licensefinding in Guava source code #2865
-
Detection of a license text or notice fragment which is too weak to represent a bona fide license detection alone.
-
Detection of longer unknown license references such as
- a "license introduction" (as in "This is licensed under....") that may be noisy when followed by a bona fide license notice or text.
- a license reference to the license in a file (as in "See file COPYING for license") where we can follow the reference
-
Lack of proper detection of a structured license tag found in a package manifest which is returned as an unknown license
-
When fragments of the same license are detected with only copyrights added in between as in license detection: Add the nunit license #2859
-
When sequence of SPDX licenses id are found in license detection tools
-
Please add yours!
Solution elements
We could treat and report separately mere clues such as this one: they could be an interesting insight in some cases, but alone they are too weak to be considered a license detection
The upcoming two-step process where license matches are grouped in a license detection is another way to consider. We could detect patterns of license matches that could be resolved in a detection. For instance a license intro followed by a license notice.
The scancode-analyzer heuristics and ML-based detection of false positive is another way
Activity
porsche-rishisaxena commentedon Feb 28, 2022
Hi Philippe,
In reference to our collective ORT community meeting, we touch base on the false positive license detection 2 weeks ago on version v30.1.0 where Porsche AG OSO also consolidated a report of false-positive cases. Please find attached the report for your kind reference and review.
report_false_positives.xlsx
CC: @sschuberth
PatteSI commentedon Feb 28, 2022
Thank you for taking action here.
I will now have a deeper look into our false positive findings as well.
EDIT: I forgot to mention that everything I mention below was found using scanCode 30.1.0
At first glance it seems that many LicenseRef-scancode-free-unknown and LicenseRef-scancode-unknown-license-reference findings in our Java projects are actually found in META-INF/LICENSE files created by Maven inside the JARs Example: https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.13.1/jackson-dataformat-yaml-2.13.1.jar
In this case line 3: "Jackson is a high-performance, Free/Open Source JSON processing library." and line 11-13: "Jackson core and extension components may be licensed under different licenses.
To find the details that apply to this artifact see the accompanying LICENSE file.
For more information, including possible other licensing options, contact"
The latter one would probably fall into point 3 mentioned above, a reference to another license file. No idea about the first one though. Not sure why they are only found in the JAR binary and not also in the actual source repo with the same text: https://github.com/FasterXML/jackson-dataformats-text/blob/2.14/properties/src/main/resources/META-INF/NOTICE
Another interesting example is okhttp3 because the false positive that is found was actually introduced by yourself @pombredanne ;-) : lysine-dev/okhttp#4569 , The current file in my example is this one: https://github.com/square/okhttp/blob/parent-5.0.0-alpha.3/okhttp/src/main/resources/okhttp3/internal/publicsuffix/NOTICE . The license LicenseRef-scancode-unknown-license-reference is found in line 4: "It is subject to the terms of the Mozilla Public License, v. 2.0:" I don't understand why in this case MPL v.2.0 is not recognized correctly.
PatteSI commentedon Mar 1, 2022
I created a python parser that can parse the evaluated-model.json file create by the ORT Reporter.
It is currently scanning a list of problematic licenseRefscan codes which are mostly (always?) causing false positives: https://gist.github.com/PatteSI/5904f4bdfb149dc1ce8c73da53e2f6ae
I parse a couple of our component and this is the result. Of course it still contains a lot of duplicates (it's a json file but github won't let me upload it as .json):
falsePosFindingFinal.txt
pombredanne commentedon Mar 3, 2022
@porsche-rishisaxena Thank you ++ for the list of false positive in #2878 (comment) ... this is great and actionable!
pombredanne commentedon Mar 3, 2022
@PatteSI re: #2878 (comment)
These tow look like basic license-related clues, but are not real license statement alright.
Here is the detection I get:
This is weird and I got them the same way in both case. Could it be ORT handling things differently in these cases?
Oh well.... as the saying goes, "no good deed goes unpunished!"
https://raw.githubusercontent.com/square/okhttp/parent-5.0.0-alpha.3/okhttp/src/main/resources/okhttp3/internal/publicsuffix/NOTICE scans this way:
You will note that I am using these command line options:
--license --license-text --license-text-diagnostics --yaml-which means: license, with actual license text, but limited to the exact portion of text that was matched and reported as yaml directly on screen (with the dash) rather than to a file.This overall looks like a case of where we could merge the license intro "subject to the terms" with a following notice.
There are also some missing rules separately that will help catch more of the MPL URL and more of the MPL details in general.
pombredanne commentedon Mar 3, 2022
@PatteSI re: #2878 (comment)
This is great! Ideally what I would need is a script that would fetch the code. With that I could run
extractcodeto extract any archive and run a scan to get the actual details. I think this can be derived from you JSON.pombredanne commentedon Mar 3, 2022
@porsche-rishisaxena re: #2878 (comment)
The CSV is super useful and I can derive a script to automate re scanning from this too.
In your case and @PatteSI case, creating these data required a lot of (useful) work.
I am wondering what could be the tools that would make it easier to help you report these false positive.
sschuberth commentedon Mar 3, 2022
ORT is not handling findings in binaries or sources differently per se, and is taking ScanCode findings mostly as-is (except some post-processing to remedy #2873). But it might be that some project-specific path excludes were applied in that particular case.
sschuberth commentedon Mar 3, 2022
For ORT, if false-positives were addressed via package configurations, we could quite easily extract the
detected_licensevs. theconcluded_license.@fviernau, is that something that could be done from HERE's (probably massive) amount of package configurations?
pombredanne commentedon Mar 3, 2022
@fviernau
If there is something that can be shared, that could be used to fix massively some of these false positive! :)
pombredanne commentedon Mar 5, 2022
Here are some related issues:
proprietary-licensefinding in Guava source code #2865 by @sschuberth and @PatteSILicenseRef-scancode-elastic-license-2018#2815 by @sschuberthThese ones can likely be fixed with the new key phrases feature:
These could help with the diagnostic false positive:
This is an example of an weak detection for a new license:
This may help with some false positives:
@rspier ping too
I think we should have a live call to discuss the options to fix these. What do you think?
bennati commentedon Mar 7, 2022
@pombredanne I attach the false positives from a bunch of HERE curations, as produced by @PatteSI 's script.
Hope this helps,
falsepositives.txt
sschuberth commentedon Mar 8, 2022
To be frank, I believe having a live call with all reporters of false-positive mentioned here would be overkill. Also, I guess most people don't care too much how their issue is fixed as long as it is fixed.
From my side, however, I'd strongly vote against hard-coding just the reported cases as false-positives. Instead, we should
10 remaining items
PatteSI commentedon Mar 23, 2022
I think this discussion is a bit deviation from the original problem here. It doesn't matter what anyone thinks "unkown license" means. We are discussing how the heuristics could be improved and ways to give the end users more options to evaluate findings. I am talking here as an end-user of ORT, which is using ScanCode as a scanner. Now we started to migrate away from NexusIQ to ORT and we see sometime hundreds of those "unknown-license" findings in big projects. There are many examples of trivial finding where the heuristics/rules used in ScanCode are just to broad and get triggered for simple comments using the word "license". We are not only discussing the general problem here of how to improve the rule based findings. There are many example given in the first post. It's not only about "unknown" licenses.
pombredanne commentedon Mar 26, 2022
I have attached a presentation to better grap a summary of the issue:
ScanCode-licenses-false-positive-2022-03.pptx.pdf
pombredanne commentedon Mar 26, 2022
@richardfontana I would be interested to get some feedback too
@opensourcepilot I was re-reading your (thought-provoking) article in https://opensource.com/article/21/7/open-source-scanning-error and we are trying to fix false positives license detections for ScanCode with this issue. Your insights would be much appreciated!
richardfontana commentedon Mar 26, 2022
@sutula may find this of interest
alext34ms commentedon Mar 28, 2022
On the topic of making the current rule set a bit more stringent:
The feature of making certain tokens required by putting them in
{{}}is an awesome tool to sharpen SCTK even more.The challenge is that we have 31.000+ rules. But that should not stop us. 😉
Proposal: doing an "automated" retro-fit of all rules to include SPDX identifiers in
{{}}.I made a very KISS "one liner" that does just that. The result seems quite OK when looking at some sample rules.
Some thoughts for this update:
false-positive*) rules and what are those even?spdx_identifier.list
Above one liner makes changes to about 6400/31000 rules.
pombredanne commentedon Mar 29, 2022
@alext34ms re:
Sleek! very smart. I like it
I do not think there it would have any impact.
These should be left alone. These are rules that can be matched only exactly and that are about licenses but are NOT license notices or texts. They should be used sparingly as a last resort. For instance, this text:
is NOT a GPL-related notice, but some commentary about the GPL license and would be a typical case for a "false positive" rule.
It should, but it may also degrade and miss some matches in a few corner cases. These could be caught separately by the
--unknow-licenseoption though.It will likely make some fail.
I think the approach could be refined using a Python script as we have code that handle the RULEs and has all SPDX licenses alright and we could also expand this to a few more things:
Some scripts examples to use as a base are in https://github.com/nexB/scancode-toolkit/blob/develop/etc/scripts/licenses/
pombredanne commentedon May 12, 2022
@vargenau Another case to track here #2905
borisbaldassari commentedon May 17, 2022
Hi Philippe, all,
First things first: thanks for the good work people! You're great!
I'd like to contribute some feedback regarding false positives (/wrong license) we get at the Eclipse Foundation. One of our main issues is with the canonical headers used with the EPL-2.
Some lines (typically {4,5}) are recognised as
LicenseRef-scancode-unknown-license-referenceeven with the SPDX tag sitting right behind. It seems that another license text (license-intro_29.RULE) is matched before the EPL-2.0 text, so even adding various variations of the headers (line ends, etc.) doesn't help. Setting the license-score to 100 helps a bit (i.e. less wrong violations), but still not enough to make this case go away.Would it be helpful to provide a list of false positives / wrong identifications? I'll be happy to provide one if so.
AyanSinhaMahapatra commentedon May 17, 2022
@borisbaldassari Thanks for your feedback and report.
We are working on this and this specific issue of a
license_introbeing present before detections is going to be fixed, this is WIP and hasn't landed yet.This would be extremely helpful, we will use this for testing this new feature extensively, as we are using the other lists contributed here. Thanks a lot!
borisbaldassari commentedon May 18, 2022
Hi @AyanSinhaMahapatra Thanks for the head-up! I'll wait for the landing and give it a try. :-)
Please find below a list of unknown-license false positives found in a few Eclipse projects (Che, JGit, CDT, Tycho). If needed I can analyse more projects -- but since we're using ORT we don't have direct access to the scancode output, so I need to run it separately (and manually).
scancode_fp_eclipse.tar.gz
borisbaldassari commentedon May 18, 2022
Please also find attached the Python script used to generate the csv's, if it's useful.
extract_false_positives.py.tar.gz
pombredanne commentedon May 20, 2022
@borisbaldassari Thank you ++
pombredanne commentedon May 23, 2022
Another short SSPL false positive #2975
PatteSI commentedon Aug 2, 2023
As @pombredanne also asked in my initial issue for a list of false positives I just wanted to mention that the ORT community also started sharing curations for false positives. I guess ScanCode ist still one of the most widely used scanning component in ORT so they might all be relevant for you. Check out their curantions and package configurations: https://github.com/oss-review-toolkit/ort-config