You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Provide a way to enable source level statistics for tables registered in the CLI #3774
Describe the solution you'd like
We already have a per file extension listing option implementation for the read_ dataframe APIs (e.g. CsvReadOptions, ParquetReadOptions) and they have sane defaults (like collect_stats is false for CSV and true for Parquet). I wonder whether we can just use them here and obtain the ListingOptions directly from them.
Describe alternatives you've considered
Leaving as is, or enabling them globally (instead of refactoring that part to use ReadOptions) by just setting the flag to true.
I wonder if it ever can be enabled by default for parquet datasets.
The downside for parquet is that when using remote object storage, collecting of statistics takes quite a bit of IO, slowing down simple queries.
I guess at some point we have to switch testing with Delta Lake or Apache Iceberg I guess :)
I think this is now solved. There is a config option, that can be set in datafusion-cli, to collect stats or not during planning. @Dandandan@isidentical can we close the issue?
Activity
Dandandan commentedon Oct 16, 2022
I wonder if it ever can be enabled by default for parquet datasets.
The downside for parquet is that when using remote object storage, collecting of statistics takes quite a bit of IO, slowing down simple queries.
I guess at some point we have to switch testing with Delta Lake or Apache Iceberg I guess :)
adriangb commentedon May 18, 2025
I think this is now solved. There is a config option, that can be set in datafusion-cli, to collect stats or not during planning. @Dandandan @isidentical can we close the issue?