Skip to content

flatten should support Datatype::FixedSizeList #9138

Closed

Description

@universalmind303

Describe the bug

flatten only supports List not FixedSizeList

To Reproduce

select flatten(some_fixed_size_list);

Error: Internal error: Not reachable, data_type should be List.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior

flatten behaves the same on both List and FixedSizeList

Additional context

 BuiltinScalarFunction::Flatten => {
                fn get_base_type(data_type: &DataType) -> Result<DataType> {
                    match data_type {
                        DataType::List(field) => match field.data_type() {
                            DataType::List(_) => get_base_type(field.data_type()),
                            _ => Ok(data_type.to_owned()),
                        },
                        _ => internal_err!("Not reachable, data_type should be List"),
                    }
                }

                let data_type = get_base_type(&input_expr_types[0])?;
                Ok(data_type)
            }

probably just need to add a branch here for FixedSizeList

Activity

  1. Weijun-H commented on Feb 6, 2024

    @Weijun-H
    Member

    This issue can be fixed by #9108

  2. alamb commented on Feb 8, 2024

    @alamb
    Contributor

    This issue can be fixed by #9108

    Thanks @Weijun-H -- I will put #9108 on my review queue

  3. jonahgao commented on May 30, 2024

    @jonahgao
    Member

    Close it because it is already supported.

    DataFusion CLI v38.0.0
    
    > select flatten(arrow_cast(make_array([[1], [2]], [[3], [4]]), 'FixedSizeList(2, List(List(Int64)))')) as a;
    +--------------+
    | a            |
    +--------------+
    | [1, 2, 3, 4] |
    +--------------+
    1 row(s) fetched.
    Elapsed 0.011 seconds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions