Skip to content

Fix DataFrame::cache errors with Plan("Mismatch between schema and batches") - #8510

Merged
alamb merged 4 commits into
apache:mainfrom
Asura7969:df_cache_error
Dec 13, 2023
Merged

alamb merged 4 commits into
apache:mainfrom
Asura7969:df_cache_error

Conversation

@Asura7969

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #8476.

Rationale for this change

#8476 (comment)

What changes are included in this PR?

Are these changes tested?

test_cache_mismatch

Are there any user-facing changes?

@github-actions github-actions Bot added the core Core DataFusion crate label Dec 12, 2023

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this contribution @Asura7969 -- I think we can do this slightly more efficiently / consistenly, but the overall idea is 👍

Comment thread datafusion/core/src/dataframe/mod.rs Outdated
/// ```
pub async fn cache(self) -> Result<DataFrame> {
let context = SessionContext::new_with_state(self.session_state.clone());
// type cast

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than explcilty running (only) coercion (which is also run as part of execution) I suggest you use the schema that comes directly from the output stream

Something like (copied from Self::collect): https://docs.rs/datafusion/latest/src/datafusion/dataframe/mod.rs.html#756-760

        let task_ctx = Arc::new(self.task_ctx());
        let plan = self.create_physical_plan().await?;
        let schema = plan.schema();
        collect(plan, task_ctx).await

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your advice

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @Asura7969 !

Comment thread datafusion/core/src/dataframe/mod.rs Outdated
// The schema is consistent with the output
let physical_plan = self.clone().create_physical_plan().await?;
let mem_table =
MemTable::try_new(physical_plan.schema(), self.collect_partitioned().await?)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW this now runs the planner twice -- we could make it more efficient by calling collect_partitioned() directly on the physical plan rather than Self::collect_partitioned that will replan (and recollect)

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Asura7969

@alamb
alamb merged commit 2bc67ef into apache:main Dec 13, 2023
@Asura7969
Asura7969 deleted the df_cache_error branch December 14, 2023 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFrame::cache errors with Plan("Mismatch between schema and batches") but query works when not cached

2 participants