You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The use of Vec<Option<String>> to build a string array is non ideal as it requires additional allocations for each output row along with a second copy of the data
fn to_char_array(args:&[ColumnarValue]) -> Result<ColumnarValue>{let arrays = ColumnarValue::values_to_arrays(args)?;letmut results:Vec<Option<String>> = vec![];// <--- this is non ideal
Also, the code that falls back to date formatting calls cast for a single row every time the formatting fails -- since the failure is likely due to a date specifier that will get called for all rows, it would probably be faster to convert the entire input array to Date once the first time the specifier is retried
Describe the solution you'd like
Make to_char faster
You can evaluate your changes with the existing benchmark
Also, the code that falls back to date formatting calls cast for a single row every time the formatting fails -- since the failure is likely due to a date specifier that will get called for all rows, it would probably be faster to convert the entire input array to Date once the first time the specifier is retried
I was thinking of something even simpler -- convert the entire array (not just the elements that failed) -- the conversion will be super fast (it is some vectorized arithmetic)
I think this would be a good first issue for someone who was already an expert at arrow and rust. I tried for like 30 minutes and I found it pretty tricky
Activity
Omega359 commentedon Aug 12, 2025
Maybe. but that wasn't @friendlymatthew's experience as noted in this comment
alamb commentedon Aug 12, 2025
I was thinking of something even simpler -- convert the entire array (not just the elements that failed) -- the conversion will be super fast (it is some vectorized arithmetic)
Omega359 commentedon Aug 12, 2025
Good first issue perhaps?
alamb commentedon Aug 12, 2025
I think this would be a good first issue for someone who was already an expert at arrow and rust. I tried for like 30 minutes and I found it pretty tricky
huymq1710 commentedon Apr 30, 2026
Hi @alamb,
#20635 resolved
Vec<Option<String>>extra allocationsI created #21948 to resolve per-row cast on fallback
perf: Cast entire Date32 array to Date64 on 1st failure (apache#21948)