You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve optimizer performance by not using Errors in the happy path #7552
Is your feature request related to a problem or challenge?
We have had a few issues / reports where the creation of DataFusionError during query planning has contributed significantly to the overall planning time. Specifically, #5309 and more recently #7522
At the core of the issue is that creating a DataFusionError is a fairly slow operation (as it requires allocating a new String to hold the error message, and sometimes, as in #5309, the construction of the message itself is significant.
Describe the solution you'd like
I would like to update the DataFusion codebase so it does not use Errors on the "happy path"
Activity
alamb commentedon Sep 13, 2023
Here is an example of using Errors in the happy path (aka ignoring an error rather than fixing the underlying issue)
https://github.com/apache/arrow-datafusion/blob/c47ae0d13dbc9474897609f3dea4f80a93cf4de0/datafusion/physical-plan/src/filter.rs#L206-L209
comphead commentedon Sep 15, 2023
I wanna check that if no other volunteers
comphead commentedon Jul 9, 2024
I would also close this as a lot of job was done for planner to improve the performance
comphead commentedon Jul 9, 2024
@alamb I'm closing this, please feel free to reopen if needed
alamb commentedon Jul 9, 2024
I agree -- thanks for cleaning these up @comphead -- we can file new tickets if/when we have actionable changes to make