Skip to content

Slice arrays before normalizing to avoid allocating over the whole backing buffer #5708

Open

Description

@andygrove

Describe the problem

Reported during review of #5262: reading two visible values out of a sliced array allocated approximately 8 MiB, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes.

This is a component allocation measurement, not an end-to-end query timing.

Describe the potential solution

Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected.

Additional context

Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work.

Activity

  1. andygrove commented on Sep 5, 2026

    @andygrove
    MemberAuthor

    Tracked under #5709 ([EPIC] DataFusion 55 upgrade follow-ups).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationenhancementNew feature or requestpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions