Skip to content

perf: coalesce small native shuffle batches before broadcast serialization #7257

Closed

Description

@sunchao

What is the problem the feature request solves?

A broadcast build that reads many small local native shuffle blocks serializes an Arrow stream for each decoded batch. The driver collects all these streams before combining them, so tiny inputs multiply Arrow export, serialization, and stream-opening work.

Describe the potential solution

Add opt-in bounded grouping of small decoded batches before Arrow export and broadcast serialization. Start with direct local Comet native shuffle inputs and INT, BIGINT, BOOLEAN, STRING, and BINARY columns. Preserve AQE partition ranges and other consumers of a reused shuffle. Bound grouped inputs by rows, retained/copy bytes, and input count; pass large batches through without a merge copy.

Willingness to contribute

I can contribute the implementation, regression tests, and a reproducible producer benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions