Skip to content

Wrap in CoalesceBatchesExec more operators #2187

Open

Description

@akupchinskiy

What is the problem the feature request solves?

Datafusion optimizer wraps some operators which might produce multiple small batches with CoalesceBatchesExec node. In Comet, this technique is already applied to SMJ with filter.

Describe the potential solution

Shouldn't it be extended to other plans (Filter/HashJoin)? It might be especially beneficial for workloads where sort merge join -> hash join replacement is enforced.

Additional context

No response

Activity

  1. andygrove commented on Aug 19, 2025

    @andygrove
    Member

    I did experiment with this in the past (#495). My conclusion at the time was that when we write to a shuffle file, we are effectively coalescing batches as part of the process, so it was redundant to add additional CoalesceBatchesExec operators.

  2. akupchinskiy commented on Aug 19, 2025

    @akupchinskiy
    ContributorAuthor

    Ahh, make sense. I faced degradation where hash join has been chained with another comet operator (broadcast exchange), not the shuffle writer.
    Image

  3. akupchinskiy commented on Aug 19, 2025

    @akupchinskiy
    ContributorAuthor

    It is q14 from tpdcs benchmark with hash join over smj preference config on

  4. akupchinskiy commented on Aug 20, 2025

    @akupchinskiy
    ContributorAuthor

    I am thinking of making this logic conditional. We can infer chain patterns during CometExecRule optimization phase and pass the flag to the native planner to do coalesce wrapping only in cases like above. @andygrove what do you think?

  5. andygrove commented on Aug 21, 2025

    @andygrove
    Member

    I'd like to bring @mbutrovich into the discussion since we have been discussing where optimization rules should live. I think we are leaning towards keeping the scala plan representation as close to the Spark plan as possible and handle adding additional nodes on the native side.

    I do think we should continue exploring adding coalesce execs similar to how DataFusion does this in a physical optimizer rule.

  6. self-assigned this
    on Mar 29, 2026
  7. added this to the 0.15.0 milestone on Mar 29, 2026
  8. modified the milestones: 0.15.0, 0.16.0 on Apr 25, 2026
  9. andygrove commented on Oct 28, 2026

    @andygrove
    Member

    I am no longer working on this issue, so I have unassigned myself. It is open for another contributor to pick up.

  10. removed their assignment
    on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions