Skip to content

minor: Move shuffle logic from CometExecRule to CometShuffleExchangeExec serde implementation - #2853

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:exec-rule-move-shuffle-code
Dec 5, 2025
Merged

andygrove merged 2 commits into
apache:mainfrom
andygrove:exec-rule-move-shuffle-code

Conversation

@andygrove

@andygrove andygrove commented Dec 5, 2025 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

CometExecRule contains shuffle logic that would be better located in CometShuffleExchangeExec.

What changes are included in this PR?

  • CometShuffleExchangeExec now implements CometSink
  • CometExecRule uses serde framework to serialize CometShuffleExchangeExec

How are these changes tested?

@codecov-commenter

codecov-commenter commented Dec 5, 2025 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.95455% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.17%. Comparing base (f09f8af) to head (7397c6e).
⚠️ Report is 738 commits behind head on main.

Files with missing lines Patch % Lines
...t/execution/shuffle/CometShuffleExchangeExec.scala 83.81% 14 Missing and 14 partials ⚠️
...n/scala/org/apache/comet/rules/CometExecRule.scala 33.33% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2853      +/-   ##
============================================
+ Coverage     56.12%   59.17%   +3.04%     
- Complexity      976     1339     +363     
============================================
  Files           119      167      +48     
  Lines         11743    15308    +3565     
  Branches       2251     2549     +298     
============================================
+ Hits           6591     9058    +2467     
- Misses         4012     4952     +940     
- Partials       1140     1298     +158     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@andygrove
andygrove marked this pull request as ready for review December 5, 2025 20:34
// try native shuffle first, then columnar shuffle, then fall back to Spark
// if neither are supported
tryNativeShuffle(s).orElse(tryColumnarShuffle(s)).getOrElse(s)
convertToComet(s, CometShuffleExchangeExec).getOrElse(s)

@andygrove andygrove Dec 5, 2025 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CometExecRule now delegates to CometShuffleExchangeExec for serde

@andygrove andygrove changed the title minor: Move shuffle logic to CometShuffleExchangeExec minor: Move shuffle logic from CometExecRule to CometShuffleExchangeExec serde implementation Dec 5, 2025

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove this is LGTM

@andygrove
andygrove merged commit 5c6a027 into apache:main Dec 5, 2025
117 checks passed
@andygrove
andygrove deleted the exec-rule-move-shuffle-code branch December 5, 2025 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants