Skip to content

Add a utility to map originator defined license strings to SPDX license format #106

Closed

Description

@nishakm

Package managers or text files may declare a license string that may not be an SPDX license format. For example, many projects declare their license as BSD but it is unclear which BSD.
oss-review-toolkit has an implementation that comes reasonably close.

https://github.com/heremaps/oss-review-toolkit/blob/490c2da69435182db19c6dbdebad863de41a7108/spdx-utils/src/main/kotlin/SpdxLicenseAliasMapping.kt

https://github.com/heremaps/oss-review-toolkit/blob/518e17c0b1385cc403960cfdfdff69e76240cb27/spdx-utils/src/main/kotlin/SpdxDeclaredLicenseMapping.kt

This is written in kotlin but it looks reasonably straightforward to convert it to python2/3

This would be useful for tools that are reading user-declared licenses.

cc @tsteenbe @goneall @kestewart

Activity

  1. pombredanne commented on Jun 7, 2019

    @pombredanne
    Member

    @nishakm Thank you! This is indeed useful but such a mapping would be most useful in the context of a certain package manifest and type and its actual raw license declaration data and IMHO in the context of license detection. As such the mappings from ORT are generic and would better if they were package-type specific (e.g. capturing the conventions of npms, Maven, etc).

    But my main point is that I am not sure there is a place in this library where I could make use of this data. When you craft SPDX documents, you need to have already the proper normalized licenses ids and license expressions so there would not be a place to plug this in. This would have to be done/used before.

    Therefore I think this is something that could be best used in two places:

    1. as a list of aliases when parsing license expressions with the license-expression(https://github.com/nexB/license-expression/) library. This is something that is explicitly supported there and it would be great to have a list of aliases. This could be a list of generic aliases alright

    2. as mappings used when you parse the declared licenses of package manifests in the scancode-toolkit. This probably would be another good place.

    Feedback welcomed!

  2. pombredanne commented on Jun 7, 2019

    @pombredanne
    Member

    @sschuberth btw, this mapping is not right: https://github.com/heremaps/oss-review-toolkit/blob/518e17c0b1385cc403960cfdfdff69e76240cb27/spdx-utils/src/main/kotlin/SpdxDeclaredLicenseMapping.kt#L120
    "CDDL v1.0 / GPL v2 dual license" to (CDDL_1_0 and GPL_2_0_ONLY), should be instead an OR and not and AND IMHO

  3. sschuberth commented on Jun 7, 2019

    @sschuberth
    Member

    Thanks @pombredanne for the hint, @mnonnenmacher any comments?

  4. nishakm commented on Jun 8, 2019

    @nishakm
    Author

    @nishakm Thank you! This is indeed useful but such a mapping would be most useful in the context of a certain package manifest and type and its actual raw license declaration data and IMHO in the context of license detection. As such the mappings from ORT are generic and would better if they were package-type specific (e.g. capturing the conventions of npms, Maven, etc).

    Agreed. It would be a much larger undertaking then :)

    But my main point is that I am not sure there is a place in this library where I could make use of this data. When you craft SPDX documents, you need to have already the proper normalized licenses ids and license expressions so there would not be a place to plug this in. This would have to be done/used before.
    I was going to go off and create an independent python module for this purpose, but I was told by the SPDX folks that this repo would be a good central location for such a module.

    Therefore I think this is something that could be best used in two places:

    1. as a list of aliases when parsing license expressions with the license-expression(https://github.com/nexB/license-expression/) library. This is something that is explicitly supported there and it would be great to have a list of aliases. This could be a list of generic aliases alright

    The use case is basically translating what looks like the appropriate license declared somewhere in the artifact into the license expression. So I'm not sure how a license expression parser would help here.

    1. as mappings used when you parse the declared licenses of package manifests in the scancode-toolkit. This probably would be another good place.

    I'd like it to be independent of scancode-toolkit because other projects who want the same thing can use it as well. But if there are already mappings in here, where in the project might I find it?

    Feedback welcomed!

  5. sschuberth commented on Sep 25, 2019

    @sschuberth
    Member

    @nishakm how about creating language-agnostic license mappings in JSON / YAML format that are initially populated with the existing mappings from ORT / the ones that nexB has, and put them in a "neutral" place like probably a new repository at https://github.com/spdx?

  6. nishakm commented on Sep 25, 2019

    @nishakm
    Author

    @nishakm how about creating language-agnostic license mappings in JSON / YAML format that are initially populated with the existing mappings from ORT / the ones that nexB has, and put them in a "neutral" place like probably a new repository at https://github.com/spdx?

    This sounds good to me if @kestewart and @goneall are OK with creating an independent repo under the spdx github namespace. I am not a license geek but I already feel like a 1:1 mapping is not going to get all the way there. There needs to be some kind of string formatting or downstream processing from there. But a 1:1 mapping to start off would be great!

  7. pombredanne commented on Sep 25, 2019

    @pombredanne
    Member

    @sschuberth re:

    how about creating language-agnostic license mappings in JSON / YAML format that are initially populated with the existing mappings from ORT / the ones that nexB has, and put them in a "neutral" place like probably a new repository at https://github.com/spdx?

    I was exactly talking about this yesterday about this very ticket of @nishakm

    @nishakm re:

    I am not a license geek but I already feel like a 1:1 mapping is not going to get all the way there. There needs to be some kind of string formatting or downstream processing from there. But a 1:1 mapping to start off would be great!

    The right approach would be indeed not to have a 1:1 mapping but something imho which would be this way:
    Given these:

    • a license string and or structured data snippet (to account for npm old styles and Maven structures) as found in a package manifest.
    • a package manager type (e.g. a Package URL type)
      Then we map to:
    • a license expression
    • some indication of confidence (say between 0 and 100) for the accuracy of this mapping
    • some optional notes

    Let me create the repo :)

  8. pombredanne commented on Sep 25, 2019

    @pombredanne
    Member

    @nishakm @sschuberth there it is: https://github.com/spdx/package-licenses-mapping
    @nishakm I invited you there as a committer too.

  9. pombredanne commented on Sep 25, 2019

    @pombredanne
    Member

    See spdx/package-licenses-mapping#1 which is the continuation for this ticket

  10. pombredanne commented on Sep 25, 2019

    @pombredanne
    Member

    @nishakm one of the reason that having a separate repo is better is that it could be reused in many places beyond this Python tool repo.

  11. nishakm commented on Sep 25, 2019

    @nishakm
    Author

    It's what I asked for initially. It was suggested that I file an issue here :)

  12. pombredanne commented on Sep 25, 2019

    @pombredanne
    Member

    @nishakm that's fine, you could have made it clear in the ticket

  13. pombredanne commented on Sep 25, 2019

    @pombredanne
    Member

    @nishakm re:

    The use case is basically translating what looks like the appropriate license declared somewhere in the artifact into the license expression. So I'm not sure how a license expression parser would help here.

    This would be important to validate that the license expressions are correct and in a canonical form. That's a something to add as a test of sorts

  14. goneall commented on Sep 28, 2019

    @goneall
    Member

    Just catching up on the issue .

    I recall discussing this on one of the SPDX calls and I do recall talking about adding the issue to the tools.

    I also agree with the above comments that we should have a separate data mapping repo along with tools implementations that support the mapping. I don't recall the discussion precisely, but I don't think anyone had a concern with a neutral mapping repo - just a concern about adding it to the spec since the mapping may be updated quite frequently.

    I like the idea of the mapping repo and can use this in the SPDX Java Tools as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions