You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the SPDX code, we have multiple times the same code, for example: ...
I assume here (may be incorrectly) that when you say " we have multiple times the same code" you mean that the "Se details ... " comment is showing up twice?
It is always best to run with --license-text with an SPDX output to ensure that the extracted text is populated.
Otherwise, you have a boilerplate ExtractedText: <text>See details at https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/unknown-license-reference.yml </text>
the text is damaged with mojibake. This is eventually making matching a bit less accurate
$ chardet3 LICENSE.txt
LICENSE.txt: windows-1252 with confidence 0.73
$ file LICENSE.txt
LICENSE.txt: Non-ISO extended-ASCII text, with very long lines, with CRLF line terminators
the license detection could be better and return only two matches rather than three
we are working on refinements to eventually merge multiple matches in a single detection
NB: If you are interested in container scans, check out also the companion server project http://scancode.io/
headers:
- tool_name: scancode-toolkit
tool_version: 31.0.0b1
options:
input:
- LICENSE.txt
--license: yes
--license-text: yes
--license-text-diagnostics: yes
--yaml: '-'
notice: |
Generated with ScanCode and provided on an "AS IS" BASIS, WITHOUT WARRANTIES
OR CONDITIONS OF ANY KIND, either express or implied. No content created from
ScanCode should be considered or used as legal advice. Consult an Attorney
for any legal advice.
ScanCode is a free software code scanning tool from nexB Inc. and others.
Visit https://github.com/nexB/scancode-toolkit/ for support and download.
start_timestamp: '2022-03-31T185132.281914'
end_timestamp: '2022-03-31T185135.025805'
output_format_version: 2.0.0
duration: '2.743912696838379'
message:
errors: []
extra_data:
spdx_license_list_version: '3.16'
files_count: 1
files:
- path: LICENSE.txt
type: file
licenses:
- key: bsd-simplified
score: '100.0'
name: BSD-2-Clause
short_name: BSD-2-Clause
category: Permissive
is_exception: no
is_unknown: no
owner: Regents of the University of California
homepage_url: http://www.opensource.org/licenses/BSD-2-Clause
text_url: http://opensource.org/licenses/bsd-license.php
reference_url: https://scancode-licensedb.aboutcode.org/bsd-simplified
scancode_text_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.LICENSE
scancode_data_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.yml
spdx_license_key: BSD-2-Clause
spdx_url: https://spdx.org/licenses/BSD-2-Clause
start_line: 5
end_line: 5
matched_rule:
identifier: bsd-simplified_226.RULE
license_expression: bsd-simplified
licenses:
- bsd-simplified
referenced_filenames: []
is_license_text: no
is_license_notice: yes
is_license_reference: no
is_license_tag: no
is_license_intro: no
has_unknown: no
matcher: 2-aho
rule_length: 13
matched_length: 13
match_coverage: '100.0'
rule_relevance: 100
matched_text: "The BSD-2 license (the \x93License\x94) set forth below applies\
\ to all parts"
- key: bsd-simplified
score: '50.0'
name: BSD-2-Clause
short_name: BSD-2-Clause
category: Permissive
is_exception: no
is_unknown: no
owner: Regents of the University of California
homepage_url: http://www.opensource.org/licenses/BSD-2-Clause
text_url: http://opensource.org/licenses/bsd-license.php
reference_url: https://scancode-licensedb.aboutcode.org/bsd-simplified
scancode_text_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.LICENSE
scancode_data_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.yml
spdx_license_key: BSD-2-Clause
spdx_url: https://spdx.org/licenses/BSD-2-Clause
start_line: 5
end_line: 5
matched_rule:
identifier: bsd-simplified_275.RULE
license_expression: bsd-simplified
licenses:
- bsd-simplified
referenced_filenames: []
is_license_text: no
is_license_notice: yes
is_license_reference: no
is_license_tag: no
is_license_intro: no
has_unknown: no
matcher: 3-seq
rule_length: 28
matched_length: 14
match_coverage: '50.0'
rule_relevance: 100
matched_text: "License\x94) [set] [forth] [below] [applies] [to] [all] [parts]\
\ [of] [the] [Tern] project. You may not use this file except in compliance\
\ with the License."
- key: bsd-simplified
score: '100.0'
name: BSD-2-Clause
short_name: BSD-2-Clause
category: Permissive
is_exception: no
is_unknown: no
owner: Regents of the University of California
homepage_url: http://www.opensource.org/licenses/BSD-2-Clause
text_url: http://opensource.org/licenses/bsd-license.php
reference_url: https://scancode-licensedb.aboutcode.org/bsd-simplified
scancode_text_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.LICENSE
scancode_data_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.yml
spdx_license_key: BSD-2-Clause
spdx_url: https://spdx.org/licenses/BSD-2-Clause
start_line: 5
end_line: 7
matched_rule:
identifier: bsd-simplified_53.RULE
license_expression: bsd-simplified
licenses:
- bsd-simplified
referenced_filenames: []
is_license_text: no
is_license_notice: no
is_license_reference: yes
is_license_tag: no
is_license_intro: no
has_unknown: no
matcher: 2-aho
rule_length: 3
matched_length: 3
match_coverage: '100.0'
rule_relevance: 100
matched_text: "License. \r\n\r\nBSD-2"
- key: bsd-simplified
score: '100.0'
name: BSD-2-Clause
short_name: BSD-2-Clause
category: Permissive
is_exception: no
is_unknown: no
owner: Regents of the University of California
homepage_url: http://www.opensource.org/licenses/BSD-2-Clause
text_url: http://opensource.org/licenses/bsd-license.php
reference_url: https://scancode-licensedb.aboutcode.org/bsd-simplified
scancode_text_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.LICENSE
scancode_data_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/bsd-simplified.yml
spdx_license_key: BSD-2-Clause
spdx_url: https://spdx.org/licenses/BSD-2-Clause
start_line: 7
end_line: 12
matched_rule:
identifier: bsd-simplified_169.RULE
license_expression: bsd-simplified
licenses:
- bsd-simplified
referenced_filenames: []
is_license_text: yes
is_license_notice: no
is_license_reference: no
is_license_tag: no
is_license_intro: no
has_unknown: no
matcher: 2-aho
rule_length: 184
matched_length: 184
match_coverage: '100.0'
rule_relevance: 100
matched_text: "License \r\n\r\nRedistribution and use in source and binary forms,\
\ with or without modification, are permitted provided that the following\
\ conditions are met:\r\n\x95\tRedistributions of source code must retain\
\ the above copyright notice, this list of conditions and the following\
\ disclaimer.\r\n\x95\tRedistributions in binary form must reproduce the\
\ above copyright notice, this list of conditions and the following disclaimer\
\ in the documentation and/or other materials provided with the distribution.\r\
\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"\
AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED\
\ TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\
\ PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS\
\ BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\
\ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE\
\ GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\
\ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,\n WHETHER IN CONTRACT,\
\ STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING\
\ IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY\
\ OF SUCH DAMAGE."
license_expressions:
- bsd-simplified
- bsd-simplified
- bsd-simplified
- bsd-simplified
percentage_of_license_text: '94.64'
scan_errors: []
wrt. to reuse: https://github.com/fsfe/reuse-tool/tree/master/src/reuse/resources contains long lists of SPDX licenses that are real license mentions but false positives since this is a tool that is license-related.
The latest develop branch has several fixes in this area and many more planned in #2878 but this is still showing up in this case.
In general, note that ScanCode is not optimized to scan tools that are themselves license detection tools, so you can expect a lot of matches in these cases.
In the SPDX code, we have multiple times the same code, for example: ...
I assume here (may be incorrectly) that when you say " we have multiple times the same code" you mean that the "Se details ... " comment is showing up twice?
It is always best to run with --license-text with an SPDX output to ensure that the extracted text is populated. Otherwise, you have a boilerplate ExtractedText: <text>See details at https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/unknown-license-reference.yml </text>
Hi Philippe,
Sorry if I was not clear.
In the tern.spdx SPDX file, you have the following:
In the SPDX code, we have multiple times the same code, for example: ...
I assume here (may be incorrectly) that when you say " we have multiple times the same code" you mean that the "Se details ... " comment is showing up twice?
It is always best to run with --license-text with an SPDX output to ensure that the extracted text is populated. Otherwise, you have a boilerplate ExtractedText: <text>See details at https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/unknown-license-reference.yml </text>
Hi Philippe,
As you recommended, I have used the --license-text option.
unknown-license-reference is a special case where ScanCode detects elements of what may be a license. The LicenseID values for unknown-license detections are generated for consistency in the output data - not for use in an SPDX document. There is major rework pending on the handling of unknown-license-reference - see also #2878
@vargenau I am revisiting this as we start some major work on false positive:
You are scanning tern and reuse and they have quite a few licenses in them making it not the best example as they are license tools. That said we should still scan them correctly
There is work on a the new thing called "License Detection" to eventually group multiple license matches in a detection that should generally help cope with some of the issues you brought up here
@pombredanne that file comes directly from GitHub when you choose a license for the project. Do you have a suggestion for a more parse-able/standard license text we can use to communicate BSD-2?
Thanks for bringing this to my attention, I wasn't aware. Happy to update!
that file comes directly from GitHub when you choose a license for the project. Do you have a suggestion for a more parse-able/standard license text we can use to communicate BSD-2?
Thanks @pombredanne. The license file in Tern was created 5 years ago so it's good you're bringing this up. I opened a PR to fix this in Tern. Could you have a look?
Activity
pombredanne commentedon Mar 31, 2022
@vargenau Thank you for the report!
You wrote:
I assume here (may be incorrectly) that when you say " we have multiple times the same code" you mean that the "Se details ... " comment is showing up twice?
It is always best to run with
--license-textwith an SPDX output to ensure that the extracted text is populated.Otherwise, you have a boilerplate
ExtractedText: <text>See details at https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/unknown-license-reference.yml </text>pombredanne commentedon Mar 31, 2022
I ran a single scan on https://raw.githubusercontent.com/tern-tools/tern/cdc6732eda7de1e5e1f9e1298a6db2e073ec48fc/LICENSE.txt
Of note:
NB: If you are interested in container scans, check out also the companion server project http://scancode.io/
pombredanne commentedon Mar 31, 2022
wrt. to reuse: https://github.com/fsfe/reuse-tool/tree/master/src/reuse/resources contains long lists of SPDX licenses that are real license mentions but false positives since this is a tool that is license-related.
The latest develop branch has several fixes in this area and many more planned in #2878 but this is still showing up in this case.
In general, note that ScanCode is not optimized to scan tools that are themselves license detection tools, so you can expect a lot of matches in these cases.
vargenau commentedon Apr 1, 2022
Hi Philippe,
Sorry if I was not clear.
In the tern.spdx SPDX file, you have the following:
(see bigger extract in the initial report)
Why do we have the same information 3 times for the same file?
vargenau commentedon Apr 1, 2022
Hi Philippe,
As you recommended, I have used the
--license-textoption.I now get:
So we have the same LicenseID with a different ExtractedText. This seems illegal for me.
The SPDX spec says:
Provide a locally unique identifier to refer to licenses that are not found on the SPDX License List.
What do you think?
pombredanne commentedon Apr 1, 2022
This "scancode" as used in the
LicenseRefis called a license namespace and this is registered here https://tools.spdx.org/app/archive_namespace_requests/2/So this is global and not local.
mjherzog commentedon Apr 1, 2022
unknown-license-reference is a special case where ScanCode detects elements of what may be a license. The LicenseID values for unknown-license detections are generated for consistency in the output data - not for use in an SPDX document. There is major rework pending on the handling of unknown-license-reference - see also #2878
pombredanne commentedon May 12, 2022
@vargenau I am revisiting this as we start some major work on false positive:
@rnjudge see https://raw.githubusercontent.com/tern-tools/tern/cdc6732eda7de1e5e1f9e1298a6db2e073ec48fc/LICENSE.txt which is your damaged and not-really-standard license text and notice. The main issue is mojibake
rnjudge commentedon May 12, 2022
@pombredanne that file comes directly from GitHub when you choose a license for the project. Do you have a suggestion for a more parse-able/standard license text we can use to communicate BSD-2?
Thanks for bringing this to my attention, I wasn't aware. Happy to update!
pombredanne commentedon May 12, 2022
@rnjudge you wrote:
It may have bee this way, but this seems to be no longer the case:
https://raw.githubusercontent.com/pombredanne/test-bsd2/main/LICENSE
Any BSD text that scancode detects works! (I will be adding yours as a new rule FWIW) .... using https://scancode-licensedb.aboutcode.org/bsd-simplified.html will surely work perfectly . This https://opensource.org/licenses/bsd-license.php and this too https://spdx.org/licenses/BSD-2-Clause will be fine.
Fix license file formatting to be machine readable
rnjudge commentedon May 13, 2022
Thanks @pombredanne. The license file in Tern was created 5 years ago so it's good you're bringing this up. I opened a PR to fix this in Tern. Could you have a look?
Fix license file formatting to be machine readable
Fix license file formatting to be machine readable
vargenau commentedon Apr 11, 2024
In this issue, we have in fact two different issues reported in:
Closing this issue as a duplicate.