Skip to content

fix: skip matched extensions in match_unmapped_resources - #2339

Open
AradhyaTiwari10 wants to merge 1 commit into
aboutcode-org:mainfrom
AradhyaTiwari10:fix-match-unmapped-resources-extensions
Open

AradhyaTiwari10 wants to merge 1 commit into
aboutcode-org:mainfrom
AradhyaTiwari10:fix-match-unmapped-resources-extensions

Conversation

@AradhyaTiwari10

Copy link
Copy Markdown

Issues

Changes

match_unmapped_resources() called to_unmapped.exclude(extension__in=matched_extensions) without assigning the result, so the matched_extensions filter was a no-op and files already looked up in PurlDB by match_resources_to_purldb (for example .class files) were sent to PurlDB again. The result is now assigned back to to_unmapped.

Those files still end up with the requires-review status at the end of the step, as before; only the second PurlDB lookup is skipped.

Added test_match_unmapped_resources_skips_matched_extensions, which fails on main (Mapping 2 to/ resources ...) and passes with the fix. manage.py test scanpipe.tests.pipes.test_d2d scanpipe.tests.test_pipelines: 186 tests OK. ruff check and ruff format --check pass.

AI usage: I used Claude Code to help locate the bug and write the fix and the test; I reproduced the failure and ran the tests locally.

Checklist

  • I have read the contributing guidelines
  • I have linked an existing issue above
  • I have added unit tests covering the new code
  • I have reviewed and understood every line of this PR

…g#2338

The result of to_unmapped.exclude(extension__in=matched_extensions) was
discarded, so files with an extension already looked up in PurlDB were
sent to PurlDB a second time.

Signed-off-by: Aradhya Tiwari <82563152+AradhyaTiwari10@users.noreply.github.com>
@tdruez
tdruez marked this pull request as ready for review November 17, 2026 08:08

@tdruez tdruez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AradhyaTiwari10 Looks good, see my two minor comments on the test.

self.assertEqual(flag.IGNORED_MEDIA_FILE, media_file.status)

@mock.patch("scanpipe.pipes.d2d._match_purldb_resources")
def test_match_unmapped_resources_skips_matched_extensions(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test names in this module use the test_scanpipe_pipes_d2d_ prefix.

Suggested change
def test_match_unmapped_resources_skips_matched_extensions(
def test_scanpipe_pipes_d2d_match_unmapped_resources_skips_matched_extensions(

Comment on lines +1518 to +1522
to_resources = mock_match_purldb_resources.call_args.kwargs["to_resources"]
self.assertEqual(
["to/project.tar.zst-extract/expected.json"],
[resource.path for resource in to_resources],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works only because if to_unmapped: evaluated and cached the QuerySet before the call. Re-evaluated, it would be empty since all resources are flagged at this point, so a switch to .exists() in the pipe would break this test for the wrong reason. The log assertion above already covers the exclusion, I'd drop this one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

match_unmapped_resources ignores matched_extensions and re-sends those files to PurlDB

2 participants