Skip to content

fast-scan: Apply performance improvements to code: batch 4 (from benchmark results) #4066

Closed

Description

@pombredanne

This is to apply more performance improvements from benchmark results.

Activity

  1. JonoYang commented on Jul 31, 2025

    @JonoYang
    Member

    Following the findings from #4059, where we found out that we spend a lot of time in licensedcode.seq, we decided to create a Cython version of it to speed things up. We modified the original seq.py using hints and bits from CyDifflib (https://github.com/rapidfuzz/CyDifflib) to apply similar modifications to the initial seq.py and have released it as a new library, cyseq (https://github.com/aboutcode-org/cyseq) and merged it into scancode-toolkit at #4500. The improvements are only available on linux for now.

    Using the file meta-quest-oss-notice.md as a test for license scanning, we see ~3 second improvement:

    (venv) foo@bar:~/scancode-toolkit
    $ scancode --version
    ScanCode version: v32.4.1
    ScanCode Output Format version: 4.1.0
    SPDX License list version: 3.27
    (venv) foo@bar:~/scancode-toolkit
    $ scancode -l ~/meta-quest-oss-notice.md --json-pp ~/tmp/out.json -n -1
    Disabling multi-processing and multi-threading for debugging.
    Setup plugins...
    Collect file inventory...
    Scan files for: licenses with -1 process(es)...
    [####################] 2                                  
    Scanning done.
    Summary:        licenses with -1 process(es)
    Errors count:   0
    Scan Speed:     0.10 files/sec. 
    Initial counts: 1 resource(s): 1 file(s) and 0 directorie(s) 
    Final counts:   1 resource(s): 1 file(s) and 0 directorie(s) 
    Timings:
      scan_start: 2025-07-31T225857.388982
      scan_end:   2025-07-31T225908.744253
      setup_scan:licenses: 1.55s
      setup: 1.55s
      scan: 9.81s
      total: 11.36s
    Removing temporary files...done.
    

    after using cyseq:

    (venv) foo@bar:~/scancode-toolkit
    $ scancode --version
    ScanCode version: v32.4.1-20-g4556603e32
    ScanCode Output Format version: 4.1.0
    SPDX License list version: 3.27
    (venv) foo@bar:~/scancode-toolkit
    $ scancode -l ~/meta-quest-oss-notice.md --json-pp ~/tmp/out.json -n -1
    Disabling multi-processing and multi-threading for debugging.
    Setup plugins...
    Collect file inventory...
    Scan files for: licenses with -1 process(es)...
    [####################] 2                                  
    Scanning done.
    Summary:        licenses with -1 process(es)
    Errors count:   0
    Scan Speed:     0.15 files/sec. 
    Initial counts: 1 resource(s): 1 file(s) and 0 directorie(s) 
    Final counts:   1 resource(s): 1 file(s) and 0 directorie(s) 
    Timings:
      scan_start: 2025-07-31T230109.417642
      scan_end:   2025-07-31T230117.686260
      setup_scan:licenses: 1.52s
      setup: 1.52s
      scan: 6.74s
      total: 8.28s
    Removing temporary files...done.
    
  2. pombredanne commented on Aug 9, 2025

    @pombredanne
    MemberAuthor

    For background, the seq.py code is originally based on a heavily modified subset of the Python standard difflib utility at https://github.com/python/cpython/blob/main/Lib/difflib.py focusing strictly on sequence matching and alignment and specialized to process sequences of integers as input/output (as opposed to arbitrary sequences) and dealing with difflib's "junk" strings concept differently, based on the an integer being smaller or bigger than a threshold (which our notion of important vs. less important tokens for ScanCode license words matching).

    We also used as inputs to help with the Cython port a port of difflib to Cython ( CyDifflib at https://github.com/rapidfuzz/CyDifflib) to bring in similar modifications we made to the initial difflib.py.

    This has released as a new library, we call cyseq (https://github.com/aboutcode-org/cyseq) has using cyseq has been into scancode-toolkit at:

    These improvements were running initially only on linux, and later we managed to get the Cython build to work on Linux, Windows and macOS.

    See these published release:

    License-wise, there is a need for some clarification (something tracked further with: aboutcode-org/cyseq#6 )

  3. pombredanne commented on Aug 9, 2025

    @pombredanne
    MemberAuthor

    The observed performance boost is roughly cutting 30% of the license scan time.

  4. moved this from Done to Validated in 00-AboutCodePlanneron Aug 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions