Skip to content

fast-scan: Create scan benchmark for package detection in ScanCode Toolkit #4061

Closed

Description

@pombredanne

The package detection benchmark should be based on real case and not made up data. We should have at least a package for each type, and cases with and without nested package files, deep dependencies and complex licensing.

We would need to have:

  • Details of data/repos/files used for benchmarking
  • Simple instructions to run the benchmark
  • Document the issues we found

Activity

  1. AyanSinhaMahapatra commented on Feb 28, 2025

    @AyanSinhaMahapatra
    Member
  2. chinyeungli commented on Jul 18, 2025

    @chinyeungli
    Contributor

    fast-scan-package_scans_benchmark-list.csv

    @AyanSinhaMahapatra I have created a list that could potentially used as the benchmarks for package scan. Please take a quick look to see if it's alright and I'll do the download and launch the package scan.

  3. moved this from Todo ready to In progress in 00-AboutCodePlanneron Jul 24, 2025
  4. chinyeungli commented on Jul 30, 2025

    @chinyeungli
    Contributor

    @pombredanne please check fast-scan-package_scans_benchmark-list.csv

    (Also at
    https://docs.google.com/spreadsheets/d/1A4g6LvP1s-DuBI1_Ev_hJuQ81bTTkMzL/edit?usp=drive_link&ouid=111292578521869099532&rtpof=true&sd=true )

    if these are the things that we want to look at. There are basically 3 items that I look at:

    1. "for_packages" coverage
    2. detected license
    3. different purls in same repo

    Since these tasks are entirely manual, perhaps doing a few of each type is sufficient?

  5. chinyeungli commented on Aug 11, 2025

    @chinyeungli
    Contributor

    @AyanSinhaMahapatra you said you have a script to run scans, and get summaries and look for issues with --todo , feel free to let me know if I can help with anything

  6. moved this from In progress to In review in 00-AboutCodePlanneron Aug 14, 2025
  7. pombredanne commented on Sep 3, 2025

    @pombredanne
    MemberAuthor

    @chinyeungli this make sense. At some point of time, creating a pipeline can work too.

  8. moved this from In review to In progress in 00-AboutCodePlanneron Sep 3, 2025
  9. AyanSinhaMahapatra commented on Sep 3, 2025

    @AyanSinhaMahapatra
    Member

    @chinyeungli @pombredanne update on this:

    I had an incomplete report, see for details:
    projects_review_v1.csv

    The scan was stuck because of some repos like https://github.com/jakub014/CG-dependency-analyzer which are not really great examples and have lots of manifests (at https://github.com/jakub014/CG-dependency-analyzer/tree/master/src/main/resources/poms) which cause issues and large scan times. We can remove these kind of repos from the benchmark, we should only use: popular packages with the useful type of manifests and which are packages/libraries and not data repos/other projects.

    I could not finish the scans as I was looking to run the scans inside somekind of timeout using src/scancode/interrupt.py (for the package assembly step) so we are not stuck there. I still have to update the script and rerun this so we finish scanning all the packages here.

    At some point of time, creating a pipeline can work too.

    Yes, but this is to measure the times and identify performance issues in scancode-toolkit (and we're using https://github.com/aboutcode-org/scancode-toolkit/tree/package-assembly-measure which I'm not sure we can merge) so we need the script in the short term atleast. But yes we do support running pipelines with SCIO for multiple inputs using batch-create from a CSV which we can enhance.

    Attaching the script also for reference:

    fetch-scan-review.py

  10. chinyeungli commented on Sep 3, 2025

    @chinyeungli
    Contributor

    @AyanSinhaMahapatra let me know if there's anything that I can help here.

  11. removed their assignment
    on Feb 24, 2026
  12. pombredanne commented on May 14, 2026

    @pombredanne
    MemberAuthor

    This is partially done:

    @AyanSinhaMahapatra what's left to do?

  13. AyanSinhaMahapatra commented on Jun 26, 2026

    @AyanSinhaMahapatra
    Member

    I added a timeout to the plugins and process codebase step with a new CLI option, see:

    Restarted the benchmarking scans with this. Will do the same for scancode.io once this finishes.

  14. AyanSinhaMahapatra commented on Aug 4, 2026

    @AyanSinhaMahapatra
    Member

    See #4061 (comment) above for more details on the benchmarking methods.

    See package scan and package assembly step execution times.

    Execution script has been improved to:

    • write current scan processed to debug memory failures/tmux crashes
    • append CSV report after every scan to avoid failures
    • scan resume after scan failures based on scan outputs
    • remove problematic inputs from scanning failures

    We should improve the inputs to include more popular repos in addition to covering different manifests/workspaces/monorepos structures.

  15. moved this from In progress to Done in 00-AboutCodePlanneron Aug 4, 2026
  16. moved this from Done to Reviewed in 00-AboutCodePlanneron Aug 6, 2026
  17. moved this from Reviewed to Validated in 00-AboutCodePlanneron Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions