Skip to content

Add support for GROUP BY Struct (Unsupported data type in hasher: Struct) #1952

Closed

Description

@silence-coding

panicked at 'index out of bounds: the len is 2 but the index is 2' at datafusion-6.0.0/src/datasource/file_format/parquet.rs:272:13

blogs.zip

toml

[dependencies]
datafusion = "6"
tokio = "1.0"

main:

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    // register the table
    let mut ctx = ExecutionContext::new();
    ctx.register_parquet("example", "tests/blogs.parquet").await?;
    let df = ctx.sql("SELECT reply FROM example GROUP BY reply LIMIT 100").await?;
    let record_batches = df.collect().await?;
    println!("sdasd {:?}", record_batches);


   
    Ok(())
}
```

Activity

  1. xudong963 commented on Mar 8, 2022

    @xudong963
    Member

    Hi @silence-coding, we released DataFusion 7.0.0 the other day. Could you please try again using datafusion = "7"?

  2. silence-coding commented on Mar 9, 2022

    @silence-coding
    ContributorAuthor

    The same applies to datafusion = "7".

  3. xudong963 commented on Mar 9, 2022

    @xudong963
    Member

    Thanks for your check again @silence-coding. I'll track the issue later in the week.

  4. silence-coding commented on Mar 28, 2022

    @silence-coding
    ContributorAuthor

    Do you have any ideas about this? @xudong963

  5. HuSen8891 commented on Jun 8, 2022

    @HuSen8891
    Contributor

    I run the test, and get the error
    ExternalError(Execution("Arrow error: External error: Internal error: Unsupported data type in hasher:.....

    The data type of column 'reply' is not supported in hash/hashAgg, I think we can use sortAgg instead for this case(unsupported data types in hash/hashagg).

  6. silence-coding commented on Oct 17, 2022

    @silence-coding
    ContributorAuthor

    Does anyone know how this is going?

  7. Dandandan commented on Oct 17, 2022

    @Dandandan
    Contributor

    What's the data type of the reply column? Can we add it to the hashing code?

  8. added this to the 14.0.0 milestone on Oct 30, 2022
  9. andygrove commented on Nov 1, 2022

    @andygrove
    Member

    Tested this just now with updated code:

    #[tokio::test]
    async fn test() -> Result<()> {
        let ctx = SessionContext::new();
        ctx.register_parquet("example", "/home/andy/Downloads/blogs.parquet", ParquetReadOptions::default()).await?;
        let df = ctx.sql("SELECT reply FROM example GROUP BY reply LIMIT 100").await?;
        let record_batches = df.collect().await?;
        println!("sdasd {:?}", record_batches);
        Ok(())
    }

    fails with:

    Error: ArrowError(ExternalError(Execution("Arrow error: External error: Internal error: 
    Unsupported data type in hasher: Struct([Field { name: \"reply_id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"next_id\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }]). This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker")))
    
    
  10. changed the title [-]panicked at 'index out of bounds: the len is 2 but the index is 2'[/-] [+]Unsupported data type in hasher: Struct[/+] on Nov 1, 2022
  11. andygrove commented on Nov 1, 2022

    @andygrove
    Member
  12. Dandandan commented on Nov 1, 2022

    @Dandandan
    Contributor

    Thanks @andygrove .

    So the current issue here is that reply is a struct, which isn't supported in group by yet.

  13. changed the title [-]Unsupported data type in hasher: Struct[/-] [+]Add support for GROUP BY Struct (Unsupported data type in hasher: Struct)[/+] on Nov 1, 2022
  14. removed this from the 14.0.0 milestone on Nov 1, 2022
  15. rspears74 commented on Dec 19, 2023

    @rspears74
    Contributor

    Bumping this issue. I'm running into this trying to union a couple DataFrames with Struct, and List(Struct) columns. I can try to tackle this eventually, but I won't have the time to dedicate to it for a while.

  16. alamb commented on Dec 22, 2023

    @alamb
    Contributor

    I believe we have a PR up to support hashing on structs: #8552 (thanks to @jayzhan211 )

  17. rspears74 commented on Dec 22, 2023

    @rspears74
    Contributor

    Awesome! Thanks @jayzhan211!

  18. alamb commented on Nov 2, 2024

    @alamb
    Contributor

    THanks to @drauschenbach for flagging this. I double checked and it works now

    DataFusion CLI v42.1.0
    > create table t as values ({a: 1, b:2}), ({a:2, b:3}), ({a:1, b:2});
    0 row(s) fetched.
    Elapsed 0.048 seconds.
    
    > select * from t;
    +--------------+
    | column1      |
    +--------------+
    | {a: 1, b: 2} |
    | {a: 2, b: 3} |
    | {a: 1, b: 2} |
    +--------------+
    3 row(s) fetched.
    Elapsed 0.009 seconds.
    
    > select column1, count(*) from t group by column1;
    +--------------+----------+
    | column1      | count(*) |
    +--------------+----------+
    | {a: 1, b: 2} | 2        |
    | {a: 2, b: 3} | 1        |
    +--------------+----------+
    2 row(s) fetched.
    Elapsed 0.020 seconds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions