Skip to content

Add avg_byte_size column-statistic #19973

Open

Description

@gene-bordegaray

Is your feature request related to a problem or challenge?

Right now Datafusion relies heavily on row count and total size statistics to estimate costs during planning. The statistics used come at two levels:

  1. Relational (modeled by Statistics): contains num_rows (amount of rows to be scanned) and total_byte_size (total output bytes)
  2. Column (modeled by ColumnStatistics): contains byte_size (estimated size of column in bytes)

While these are useful metrics for stable operations, like reading files, they become unstable and inaccurate during query execution.

To show how, let's walk through an example.

A query initially reads 1 million rows from a Parquet file. On the initial read it finds that the `byte_size` for col_a is 1GB.

Now there is a filter that reduces the row count to 10 rows. 
DataFusion now has to guess the new `byte_size` and because it lacks a stable metric it often loses this information or a linear scaling.

This issue can be solved by introducing an average row size statistic. 
This way, on a file read we mark the average row size in the column, which no matter how the column is filtered, joined, etc will remain stable. 
In our example, knowing that we have 10 rows after a filter, we can just calculate the column data size via `num_rows * avg_byte_size`

Describe the solution you'd like

I would like to introduce a new avg_byte_size: Precision<usize> column to the ColumnStatistics struct which tracks the average size of a row in a given column. This field would give DataFusion a new model of statistic: "data density" compared to the existing "data totals".

This field would allow ever operator in the plan to recalculate output size accurately via:

$$Estimated Output Bytes = Row Count * Sum(Average Byte Size)$$

The first PR to tackle this issue is just adding the avg_byte_size field to ColumnStatistics, basic supporting methods and functionality: merging partitions' avg_byte_size fields and reconstructing total byte size for a relation through avg_byte_size, and updating parquet datasources to derive and calculate this field.

There will follow-up work to propagate and utilize these statistics.

Describe alternatives you've considered

I considered just using byte_size / num_rows because it's "the same thing" but this is not true. While this can be calculated on a file scan, it is a fragile value that can easily be corrupt. To show why here are some examples:

  • When a filter reduces the row count, we have to "guess" how byte_size will scale. Often, this cannot guarantee uniformity and will set it to Absent. Once byte_size is lost, there is no recovery and can't derive these statistics for subsequent operators.
  • Re-calculating the average from two unstable totals at every operator is redundant and more importantly introduces cascading rounding errors.

Additional context

This statistics will be impactful for both single-node DataFusion and distributed DataFusion.

Single-Node: We will be able to more reliably and consistently calculate data size and get more accurate results. This will have rippling effects in how query are planned.

Distributed: This will reap all the benefits of better single-node statistics and planning. It will also open up the doors to make better network cost decisions.

There is ongoing effort to improve statistics:
datafusion-contrib/datafusion-distributed#296
#15265

Activity

  1. gene-bordegaray commented on Jan 24, 2026

    @gene-bordegaray
    Author
  2. gene-bordegaray commented on Jan 24, 2026

    @gene-bordegaray
    Author

    take

  3. gabotechs commented on Feb 9, 2026

    @gabotechs
    Contributor

    I considered just using byte_size / num_rows because it's "the same thing" but this is not true. While this can be calculated on a file scan, it is a fragile value that can easily be corrupt

    Aren't all the challenges you are describing here also applicable to avg_byte_size?

    in other words: what would it mean if avg_byte_size is different from byte_size / num_rows?

  4. gabotechs commented on Feb 9, 2026

    @gabotechs
    Contributor

    Answering myself: there do is several places that forget to do the manual upscaling/downscaling of byte_size. For example:

    // statistics which might yield subpar results (although it is
    // true, esp regarding min/max). For a better estimation, we need
    // filter selectivity analysis first.
    column_statistics: left_stats
    .column_statistics
    .into_iter()
    .chain(right_stats.column_statistics)
    .collect(),
    })
    }
    // For SemiJoins estimation result is either zero, in cases when inputs
    // are non-overlapping according to statistics, or equal to number of rows

    Having avg_byte_size is a nice way of propagating the value without manual upscaling/downscaling based on operator selectivity

  5. jonathanc-n commented on Mar 7, 2026

    @jonathanc-n
    Contributor

    @gene-bordegaray Are you working on this? I would be interested in working on this if you are busy

  6. gene-bordegaray commented on Mar 7, 2026

    @gene-bordegaray
    Author

    @gene-bordegaray Are you working on this? I would be interested in working on this if you are busy

    Yes still working on it. Have a draft for this implemented, just waiting for other PRs to be merged in the epic for this

  7. gene-bordegaray commented on Mar 17, 2026

    @gene-bordegaray
    Author

    I know you guys were intersted in statistics work and with #19957 going in I am starting my work with this again
    cc: @asolimando @jonathanc-n @xudong963 @paleolimbot

  8. asolimando commented on Mar 17, 2026

    @asolimando
    Member

    Great write-up, the problem you identified with propagation being unreliable is real and worth fixing.

    I stumbled upon a message from @adriangb that got me thinking, quoting:

    I don't love the current representation Vec for several reasons: (1) for wide tables / scans having empty
    ColumnStatistics takes up a lot of memory. We had a situation where we essentially initialized every file in the table (say 10k) and initialized empty stats for each column (say 100) so we ended up with a couple hundred MB of empty stats;

    Given this, I wonder if we should avoid adding new fields to ColumnStatistics when the value is derivable. avg_byte_size is byte_size / num_rows, as @gabotechs also noted. For fixed-width types it's even simpler: just the type width from the schema.

    The real issue, as you point out, is that byte_size gets lost or corrupted during propagation (e.g. the join code @gabotechs found). But I think a stored avg_byte_size would face the same challenges, as it's still an estimation that degrade through operators, there is no way around that.

    What if instead we:

    1. Fix the propagation bugs for byte_size (which benefits all downstream consumers, not just avg size)
    2. Add a helper method like fn avg_byte_size(&self, num_rows: Precision<usize>) -> Precision<usize> so callers get the derived value conveniently

    This avoids growing ColumnStatistics while still making avg byte size easy to use. What do you think?

  9. gene-bordegaray commented on Mar 19, 2026

    @gene-bordegaray
    Author

    What if instead we:

    1. Fix the propagation bugs for byte_size (which benefits all downstream consumers, not just avg size)
    2. Add a helper method like fn avg_byte_size(&self, num_rows: Precision<usize>) -> Precision<usize> so callers get the derived value conveniently

    This avoids growing ColumnStatistics while still making avg byte size easy to use. What do you think?

    Had some time to read this and think about it a little more. I think this is true and the right direction to go if statistics continue to be an priority in the community since yes avg_byte_size can just be recomputed using byte_size / num_rows, but does rely on the fact that our stats propagation is good.

    My original motivation for keeping it stored as a column is this scenario:

    Helper-only:
        - Scan:
            - num_rows = 1000000
            - byte_size = 1000000000
        - Filter:
            - num_rows = 10
            - byte_size gets dropped or becomes stale or recompute
        - Result:
            - helper can no longer give a useful avg
    
    Stored avg_byte_size
        - Scan:
            - num_rows = 1000000
            - byte_size = 1000000000
            - avg_byte_size = 1000
        - Filter:
            - num_rows = 10
            - byte_size gets dropped or becomes stale
            - avg_byte_size = 1000 (no recompute)
        - Result:
            - downstream can still estimate byte_size using: 10 * 1000 = 10000
    

    As seen it is very easy to propagate the avg_byte_size compared to the byte_size since it is derived once at scan then naturally passes through filters, joins, limits, etc without recomputition.

    The stability of the avg make seems like a promising way to deal with losing stats, this could justify keeping it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions