You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: coalesce small native shuffle batches before broadcast serialization #7257
A broadcast build that reads many small local native shuffle blocks serializes an Arrow stream for each decoded batch. The driver collects all these streams before combining them, so tiny inputs multiply Arrow export, serialization, and stream-opening work.
Describe the potential solution
Add opt-in bounded grouping of small decoded batches before Arrow export and broadcast serialization. Start with direct local Comet native shuffle inputs and INT, BIGINT, BOOLEAN, STRING, and BINARY columns. Preserve AQE partition ranges and other consumers of a reused shuffle. Bound grouped inputs by rows, retained/copy bytes, and input count; pass large batches through without a merge copy.
Willingness to contribute
I can contribute the implementation, regression tests, and a reproducible producer benchmark.
Activity