You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add support for GROUP BY Struct (Unsupported data type in hasher: Struct) #1952
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
// register the table
let mut ctx = ExecutionContext::new();
ctx.register_parquet("example", "tests/blogs.parquet").await?;
let df = ctx.sql("SELECT reply FROM example GROUP BY reply LIMIT 100").await?;
let record_batches = df.collect().await?;
println!("sdasd {:?}", record_batches);
Ok(())
}
```
I run the test, and get the error
ExternalError(Execution("Arrow error: External error: Internal error: Unsupported data type in hasher:.....
The data type of column 'reply' is not supported in hash/hashAgg, I think we can use sortAgg instead for this case(unsupported data types in hash/hashagg).
#[tokio::test]asyncfntest() -> Result<()>{let ctx = SessionContext::new();
ctx.register_parquet("example","/home/andy/Downloads/blogs.parquet",ParquetReadOptions::default()).await?;let df = ctx.sql("SELECT reply FROM example GROUP BY reply LIMIT 100").await?;let record_batches = df.collect().await?;println!("sdasd {:?}", record_batches);Ok(())}
fails with:
Error: ArrowError(ExternalError(Execution("Arrow error: External error: Internal error:
Unsupported data type in hasher: Struct([Field { name: \"reply_id\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"next_id\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }]). This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker")))
changed the title [-]panicked at 'index out of bounds: the len is 2 but the index is 2'[/-][+]Unsupported data type in hasher: Struct[/+]on Nov 1, 2022
changed the title [-]Unsupported data type in hasher: Struct[/-][+]Add support for GROUP BY Struct (Unsupported data type in hasher: Struct)[/+]on Nov 1, 2022
Bumping this issue. I'm running into this trying to union a couple DataFrames with Struct, and List(Struct) columns. I can try to tackle this eventually, but I won't have the time to dedicate to it for a while.
Activity
xudong963 commentedon Mar 8, 2022
Hi @silence-coding, we released DataFusion 7.0.0 the other day. Could you please try again using
datafusion = "7"?silence-coding commentedon Mar 9, 2022
The same applies to datafusion = "7".
xudong963 commentedon Mar 9, 2022
Thanks for your check again @silence-coding. I'll track the issue later in the week.
silence-coding commentedon Mar 28, 2022
Do you have any ideas about this? @xudong963
HuSen8891 commentedon Jun 8, 2022
I run the test, and get the error
ExternalError(Execution("Arrow error: External error: Internal error: Unsupported data type in hasher:.....
The data type of column 'reply' is not supported in hash/hashAgg, I think we can use sortAgg instead for this case(unsupported data types in hash/hashagg).
silence-coding commentedon Oct 17, 2022
Does anyone know how this is going?
Dandandan commentedon Oct 17, 2022
What's the data type of the reply column? Can we add it to the hashing code?
andygrove commentedon Nov 1, 2022
Tested this just now with updated code:
fails with:
[-]panicked at 'index out of bounds: the len is 2 but the index is 2'[/-][+]Unsupported data type in hasher: Struct[/+]andygrove commentedon Nov 1, 2022
@Dandandan fyi
Dandandan commentedon Nov 1, 2022
Thanks @andygrove .
So the current issue here is that
replyis a struct, which isn't supported ingroup byyet.[-]Unsupported data type in hasher: Struct[/-][+]Add support for GROUP BY Struct (Unsupported data type in hasher: Struct)[/+]rspears74 commentedon Dec 19, 2023
Bumping this issue. I'm running into this trying to
uniona coupleDataFrames withStruct, andList(Struct)columns. I can try to tackle this eventually, but I won't have the time to dedicate to it for a while.alamb commentedon Dec 22, 2023
I believe we have a PR up to support hashing on structs: #8552 (thanks to @jayzhan211 )
rspears74 commentedon Dec 22, 2023
Awesome! Thanks @jayzhan211!
alamb commentedon Nov 2, 2024
THanks to @drauschenbach for flagging this. I double checked and it works now