You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
DataFrame::cache errors with Plan("Mismatch between schema and batches") but query works when not cached #8476
Dataframe::cache gives an error where an execution that doesn't first cache results succeeds.
I would have expected caching to have no effect on success/failure.
To Reproduce
use datafusion::prelude::SessionContext;#[tokio::main]asyncfnmain() -> Result<(),Box<dyn std::error::Error>>{let sql = "SELECT CASE WHEN true THEN NULL ELSE 1 END;";let ctx = SessionContext::new();let plan = ctx.state().create_logical_plan(sql).await?;let df = ctx.execute_logical_plan(plan).await?;// Comment out the next line to make the error go away.let df = df.cache().await?;let batches = df.collect().await?;let display = datafusion::arrow::util::pretty::pretty_format_batches(&batches).unwrap();println!("{}", display);Ok(())}
Expected behavior
Behavior with and without let df = df.cache().await? to be functionally same, only changing performance and memory use.
changed the title [-]DataFrame::cache errors with `Plan("Mismatch between schema and batches")` but work when not cached[/-][+]DataFrame::cache errors with `Plan("Mismatch between schema and batches")` but query works when not cached[/+]on Dec 9, 2023
Activity
[-]DataFrame::cache errors with `Plan("Mismatch between schema and batches")` but work when not cached[/-][+]DataFrame::cache errors with `Plan("Mismatch between schema and batches")` but query works when not cached[/+]Asura7969 commentedon Dec 11, 2023
The reason is that schema comparison:
https://github.com/apache/arrow-datafusion/blob/d091b55be6a4ce552023ef162b5d081136d3ff6d/datafusion/core/src/datasource/memory.rs#L68
schema:
batches_schema:
data_type:
Null&Int64Can we use the optimized schema to compare,like:
before:
after:
alamb commentedon Dec 11, 2023
In general, DataType::Null is normally resolved (via Coercion) to an actual type of the target schema
Maybe we could apply the same approach here