You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The main issue limiting scalability in Ballista today is that joins are implemented as hash joins where each partition of the probe side causes the entire left side to be loaded into memory.
Describe the solution you'd like
To make this scalable we need to hash partition left and right inputs so that we can join the left and right partitions in parallel.
There is already work underway in DataFusion to implement this that we can leverage.
Here is some additional information. When I run TPC-H query 5 in the benchmarks, against DataFusion, I see that the physical plan used partitioned joins.
For example, I see that both inputs to the join are partitioned on the join keys, and the join mode is Partitioned.
This means that the join can run in parallel because the inputs are partitioned. So partition 1 of the join reads partition 1 of the left and right inputs, and so on.
When I run the same query against Ballista, I see.
HashJoinExec: mode=CollectLeft, join_type=Inner, on=[("c_custkey", "o_custkey")]
ParquetExec: batch_size=8192, limit=None, partitions=[...]
FilterExec: o_orderdate >= CAST(1994-01-01 AS Date32) AND o_orderdate < CAST(1995-01-01 AS Date32)
ParquetExec: batch_size=8192, limit=None, partitions=[
Here, we see join mode CollectLeft, which means that each partition being executed will go and fetch the entire left-side of the join into memory. This is very inefficient both in terms of memory and compute and potentially gets exponentially slower the more partitions we have.
What we need to do is apply the same "partitioned hash join" pattern to Ballista.
Activity
boazberman commentedon Apr 30, 2021
I'd love to work on this if someone can provide further reading material and/or the area in the code
andygrove commentedon May 22, 2021
Here is some additional information. When I run TPC-H query 5 in the benchmarks, against DataFusion, I see that the physical plan used partitioned joins.
For example, I see that both inputs to the join are partitioned on the join keys, and the join mode is
Partitioned.This means that the join can run in parallel because the inputs are partitioned. So partition 1 of the join reads partition 1 of the left and right inputs, and so on.
When I run the same query against Ballista, I see.
Here, we see join mode
CollectLeft, which means that each partition being executed will go and fetch the entire left-side of the join into memory. This is very inefficient both in terms of memory and compute and potentially gets exponentially slower the more partitions we have.What we need to do is apply the same "partitioned hash join" pattern to Ballista.
andygrove commentedon Jun 9, 2021
I created a Google doc to discuss the design, and planned work, in more detail.
https://docs.google.com/document/d/1yUnGWsHKYOAxWijDJisEFYU4dIym_GSRSMpwfWjVZq8/edit?usp=sharing
Add baseline_metrics for FileStream to record metrics like elapsed ti…
Add baseline_metrics for FileStream to record metrics like elapsed ti…
Fix Binary & Binary View Unparsing (#13427)
feat: Add cause to native exception (apache#63)
[CI] - Add Release Audit Tool(RAT) in CI (apache#63)