Repository navigation
Description
Is your feature request related to a problem or challenge?
Right now Datafusion relies heavily on row count and total size statistics to estimate costs during planning. The statistics used come at two levels:
- Relational (modeled by
Statistics): containsnum_rows(amount of rows to be scanned) andtotal_byte_size(total output bytes) - Column (modeled by
ColumnStatistics): containsbyte_size(estimated size of column in bytes)
While these are useful metrics for stable operations, like reading files, they become unstable and inaccurate during query execution.
To show how, let's walk through an example.
A query initially reads 1 million rows from a Parquet file. On the initial read it finds that the `byte_size` for col_a is 1GB.
Now there is a filter that reduces the row count to 10 rows.
DataFusion now has to guess the new `byte_size` and because it lacks a stable metric it often loses this information or a linear scaling.
This issue can be solved by introducing an average row size statistic.
This way, on a file read we mark the average row size in the column, which no matter how the column is filtered, joined, etc will remain stable.
In our example, knowing that we have 10 rows after a filter, we can just calculate the column data size via `num_rows * avg_byte_size`
Describe the solution you'd like
I would like to introduce a new avg_byte_size: Precision<usize> column to the ColumnStatistics struct which tracks the average size of a row in a given column. This field would give DataFusion a new model of statistic: "data density" compared to the existing "data totals".
This field would allow ever operator in the plan to recalculate output size accurately via:
The first PR to tackle this issue is just adding the avg_byte_size field to ColumnStatistics, basic supporting methods and functionality: merging partitions' avg_byte_size fields and reconstructing total byte size for a relation through avg_byte_size, and updating parquet datasources to derive and calculate this field.
There will follow-up work to propagate and utilize these statistics.
Describe alternatives you've considered
I considered just using byte_size / num_rows because it's "the same thing" but this is not true. While this can be calculated on a file scan, it is a fragile value that can easily be corrupt. To show why here are some examples:
- When a filter reduces the row count, we have to "guess" how
byte_sizewill scale. Often, this cannot guarantee uniformity and will set it toAbsent. Oncebyte_sizeis lost, there is no recovery and can't derive these statistics for subsequent operators. - Re-calculating the average from two unstable totals at every operator is redundant and more importantly introduces cascading rounding errors.
Additional context
This statistics will be impactful for both single-node DataFusion and distributed DataFusion.
Single-Node: We will be able to more reliably and consistently calculate data size and get more accurate results. This will have rippling effects in how query are planned.
Distributed: This will reap all the benefits of better single-node statistics and planning. It will also open up the doors to make better network cost decisions.
There is ongoing effort to improve statistics:
datafusion-contrib/datafusion-distributed#296
#15265
Activity
gene-bordegaray commentedon Jan 24, 2026
cc: @gabotechs @asolimando @NGA-TRAN
gene-bordegaray commentedon Jan 24, 2026
take
gabotechs commentedon Feb 9, 2026
Aren't all the challenges you are describing here also applicable to
avg_byte_size?in other words: what would it mean if
avg_byte_sizeis different frombyte_size / num_rows?gabotechs commentedon Feb 9, 2026
Answering myself: there do is several places that forget to do the manual upscaling/downscaling of
byte_size. For example:datafusion/datafusion/physical-plan/src/joins/utils.rs
Lines 521 to 533 in b80bf2c
Having
avg_byte_sizeis a nice way of propagating the value without manual upscaling/downscaling based on operator selectivityjonathanc-n commentedon Mar 7, 2026
@gene-bordegaray Are you working on this? I would be interested in working on this if you are busy
gene-bordegaray commentedon Mar 7, 2026
Yes still working on it. Have a draft for this implemented, just waiting for other PRs to be merged in the epic for this
gene-bordegaray commentedon Mar 17, 2026
I know you guys were intersted in statistics work and with #19957 going in I am starting my work with this again
cc: @asolimando @jonathanc-n @xudong963 @paleolimbot
asolimando commentedon Mar 17, 2026
Great write-up, the problem you identified with propagation being unreliable is real and worth fixing.
I stumbled upon a message from @adriangb that got me thinking, quoting:
Given this, I wonder if we should avoid adding new fields to
ColumnStatisticswhen the value is derivable.avg_byte_sizeisbyte_size / num_rows, as @gabotechs also noted. For fixed-width types it's even simpler: just the type width from the schema.The real issue, as you point out, is that
byte_sizegets lost or corrupted during propagation (e.g. the join code @gabotechs found). But I think a storedavg_byte_sizewould face the same challenges, as it's still an estimation that degrade through operators, there is no way around that.What if instead we:
byte_size(which benefits all downstream consumers, not just avg size)fn avg_byte_size(&self, num_rows: Precision<usize>) -> Precision<usize>so callers get the derived value convenientlyThis avoids growing
ColumnStatisticswhile still making avg byte size easy to use. What do you think?gene-bordegaray commentedon Mar 19, 2026
Had some time to read this and think about it a little more. I think this is true and the right direction to go if statistics continue to be an priority in the community since yes
avg_byte_sizecan just be recomputed usingbyte_size / num_rows, but does rely on the fact that our stats propagation is good.My original motivation for keeping it stored as a column is this scenario:
As seen it is very easy to propagate the avg_byte_size compared to the byte_size since it is derived once at scan then naturally passes through filters, joins, limits, etc without recomputition.
The stability of the avg make seems like a promising way to deal with losing stats, this could justify keeping it.