You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Slice arrays before normalizing to avoid allocating over the whole backing buffer #5708
Reported during review of #5262: reading two visible values out of a sliced array allocated approximately 8 MiB, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes.
This is a component allocation measurement, not an end-to-end query timing.
Describe the potential solution
Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected.
Additional context
Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work.
Activity
andygrove commentedon Sep 5, 2026
Tracked under #5709 ([EPIC] DataFusion 55 upgrade follow-ups).
Avoid normalizing hidden values in sliced list set operations (apache…
Avoid normalizing hidden values in sliced list set operations (apache…
Avoid normalizing hidden values in sliced list set operations (apache…