Skip to content

Optimize to_char for array conversions #17152

Closed

Description

@alamb

Is your feature request related to a problem or challenge?

Specifically the code that

The use of Vec<Option<String>> to build a string array is non ideal as it requires additional allocations for each output row along with a second copy of the data

fn to_char_array(args: &[ColumnarValue]) -> Result<ColumnarValue> {
    let arrays = ColumnarValue::values_to_arrays(args)?;
    let mut results: Vec<Option<String>> = vec![]; // <--- this is non ideal

Also, the code that falls back to date formatting calls cast for a single row every time the formatting fails -- since the failure is likely due to a date specifier that will get called for all rows, it would probably be faster to convert the entire input array to Date once the first time the specifier is retried

Describe the solution you'd like

Make to_char faster

You can evaluate your changes with the existing benchmark

cargo bench --bench to_char

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. Omega359 commented on Aug 12, 2025

    @Omega359
    Contributor

    Also, the code that falls back to date formatting calls cast for a single row every time the formatting fails -- since the failure is likely due to a date specifier that will get called for all rows, it would probably be faster to convert the entire input array to Date once the first time the specifier is retried

    Maybe. but that wasn't @friendlymatthew's experience as noted in this comment

  2. alamb commented on Aug 12, 2025

    @alamb
    ContributorAuthor

    I was thinking of something even simpler -- convert the entire array (not just the elements that failed) -- the conversion will be super fast (it is some vectorized arithmetic)

  3. Omega359 commented on Aug 12, 2025

    @Omega359
    Contributor

    Good first issue perhaps?

  4. alamb commented on Aug 12, 2025

    @alamb
    ContributorAuthor

    Good first issue perhaps?

    I think this would be a good first issue for someone who was already an expert at arrow and rust. I tried for like 30 minutes and I found it pretty tricky

  5. huymq1710 commented on Apr 30, 2026

    @huymq1710
    Contributor

    Hi @alamb,

    #20635 resolved Vec<Option<String>> extra allocations
    I created #21948 to resolve per-row cast on fallback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceMake DataFusion faster

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions