You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Following the findings from #4059, where we found out that we spend a lot of time in licensedcode.seq, we decided to create a Cython version of it to speed things up. We modified the original seq.py using hints and bits from CyDifflib (https://github.com/rapidfuzz/CyDifflib) to apply similar modifications to the initial seq.py and have released it as a new library, cyseq (https://github.com/aboutcode-org/cyseq) and merged it into scancode-toolkit at #4500. The improvements are only available on linux for now.
Using the file meta-quest-oss-notice.md as a test for license scanning, we see ~3 second improvement:
For background, the seq.py code is originally based on a heavily modified subset of the Python standard difflib utility at https://github.com/python/cpython/blob/main/Lib/difflib.py focusing strictly on sequence matching and alignment and specialized to process sequences of integers as input/output (as opposed to arbitrary sequences) and dealing with difflib's "junk" strings concept differently, based on the an integer being smaller or bigger than a threshold (which our notion of important vs. less important tokens for ScanCode license words matching).
We also used as inputs to help with the Cython port a port of difflib to Cython ( CyDifflib at https://github.com/rapidfuzz/CyDifflib) to bring in similar modifications we made to the initial difflib.py.
Activity
JonoYang commentedon Jul 31, 2025
Following the findings from #4059, where we found out that we spend a lot of time in
licensedcode.seq, we decided to create a Cython version of it to speed things up. We modified the original seq.py using hints and bits from CyDifflib (https://github.com/rapidfuzz/CyDifflib) to apply similar modifications to the initial seq.py and have released it as a new library,cyseq(https://github.com/aboutcode-org/cyseq) and merged it into scancode-toolkit at #4500. The improvements are only available on linux for now.Using the file meta-quest-oss-notice.md as a test for license scanning, we see ~3 second improvement:
after using cyseq:
pombredanne commentedon Aug 9, 2025
For background, the seq.py code is originally based on a heavily modified subset of the Python standard difflib utility at https://github.com/python/cpython/blob/main/Lib/difflib.py focusing strictly on sequence matching and alignment and specialized to process sequences of integers as input/output (as opposed to arbitrary sequences) and dealing with difflib's "junk" strings concept differently, based on the an integer being smaller or bigger than a threshold (which our notion of important vs. less important tokens for ScanCode license words matching).
We also used as inputs to help with the Cython port a port of difflib to Cython ( CyDifflib at https://github.com/rapidfuzz/CyDifflib) to bring in similar modifications we made to the initial difflib.py.
This has released as a new library, we call
cyseq(https://github.com/aboutcode-org/cyseq) has using cyseq has been into scancode-toolkit at:These improvements were running initially only on linux, and later we managed to get the Cython build to work on Linux, Windows and macOS.
See these published release:
License-wise, there is a need for some clarification (something tracked further with: aboutcode-org/cyseq#6 )
pombredanne commentedon Aug 9, 2025
The observed performance boost is roughly cutting 30% of the license scan time.