Repository navigation
perf(physical_plan): avoid eagerly starting streams in coalesce_partitions - #27126
tschwarzinger wants to merge 2 commits into
Conversation
|
Possible Follow-Up: I think |
|
@adriangb Could you run the TPC-H SF10 (SF1 might be also fine) with filter pushdown turned on? |
|
cc @devanbenz if you're interested |
|
run benchmark tpch10 env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing lazy-coalesce (a2c3adb) to 70f2ed0 (merge-base) diff Run configurationrun benchmark tpch10
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing lazy-coalesce (a2c3adb) to 70f2ed0 (merge-base) diff Run configurationrun benchmark tpch10
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
Yeah that looks similar to my machine. Thanks! |
|
I am unable to run benchmarks in the PR, but, could we get them ran against tpcds with and without pushdown and tpch10 without pushdown? I ran through some more benchmarks locally and noticed some regressions, these runs all used the default config so no filter pushdown:
I think this change makes it so the build sides that used to run in parallel run sequentially now? Very much restricting parallelism. It helps when there is a dynamic filter to prune data but for anything else it limits parallelism. Something akin to this: (Thanks claude for taking my word vomit and turning it in to a nice little diagram 😆) |
|
Thanks @devanbenz ! I think you're observing what I meant with
I've only tested tpch sf10 with filter pushdown, so thanks for pointing out the regressions in the other benchmarks! It definitely makes sense as execution is now more sequentially (as you nice diagram shows :) ). Maybe also doing a run on TPC-DS with dynamic filters on would make sense. The questions I guess is whether this is an intended design or just a something that grew organically. Also if there is only a single input partition we don't poll the stream eagerly. 1 => {
// single-partition path: execute child directly, but ensure fetch is respected
// (wrap with ObservedStream only if fetch is present so we don't add overhead otherwise)
let child_stream = self.input.execute(0, context)?;
if self.fetch.is_some() {
let baseline_metrics = BaselineMetrics::new(&self.metrics, partition);
return Ok(Box::pin(ObservedStream::new(
child_stream,
baseline_metrics,
self.fetch,
)));
}
Ok(child_stream)
}It kind of seems to me that this is an unintended effect (I wasn't aware that Personally, I think it's kind of a surprise that Any takes on this? I will need to look at the queries though next week to see if there is a good solution. Marking this as draft for now so we don't merge it by accident. Thanks again! |
|
What's also interesting / related now is the HashJoinBuffering optimization which explicitly parallelizes build and probe side of a join. |
My assumption is that this is correct , there is nothing to merge when there is a single input partition so we just pass the child stream through and just starts whenever the child stream starts. That being said whether the eagerness of the streams is a side-effect or an actual design choice is up in the air. I'll need to look more in to the different Join operators this weekend more as I'm not as intimately familiar with them as I would like to be to offer much in the way of good suggestions. What you brought up from the HashJoinBuffering operator is interesting though. I'll take a peak when I have some time. Let me do some exploratory work and I'll get back to this PR :) |
|
Heh yeah it seems to have happened organically over time, there is no specific PR that outlines a reason for the implementation:
None of these commit messages calls the early start deliberate. The stated goal was running all partitions in parallel, and starting at Let's work on brainstorming some different approaches 🫡 |
|
I think I got some good results with controlling the parallelization explicitly from hash joins depending on whether we're still waiting for dynamic filters. However, I need to polish this a bit more. I guess the best way is to push all changes to this branch for now to see if we can make it work (i.e., run benchmarks) and then move the fixes to production code in smaller PRs. I hope I'll have some news on that tomorrow. |
|
IMO the issue with these sorts of optimizations is that they depend on the IO patterns, data and query. E.g. if there is no selective dynamic filter or the build side is very slow (e.g. wrong join order) then buffering the probe side will show great results. If the build side is fast and/or the dynamic filters it generates are very effective then buffering the probe side won't help. |
|
I've opened a discussion issue about being more explicit about concurrency in DataFusion plans. Would love your input there, if you're interested! |
Which issue does this PR close?
Rationale for this change
CoalescePartitionsExeceagerly starts and polls streams duringExecutionPlan::execute(a task does the polling). This can cause situations where dynamic filters of a parent join are not yet available and thus the first poll of scans is done without the dynamic filters.This should improve at least 2 TPC-H queries (q21, q7) with
pushdown_filters=true.What changes are included in this PR?
What is the testing strategy for this PR?
drops_input_plan_after_input_streams_startnow polls once before checking, as inputs no longer start inexecuteinputs_start_on_first_poll_not_on_executeasserts that the stream is not startedAre there any user-facing changes?