Repository navigation
Description
Describe the bug
CometFairMemoryPool limits a consumer that can spill to (task_share - unspillable) / num_spill, where num_spill counts every registered consumer that can spill (fair_pool.rs). DataFusion registers consumers when execute() builds the stream tree, before any batch flows. So the consumers above a blocking operator count from the start, although they hold nothing until it emits. The operator doing the work is capped at a fraction of the task's share and spills while most of that share is unused.
This is the per-task form of apache/datafusion#22036, which describes the same policy in FairSpillPool. Since #7088 the per-consumer limit binds whenever tasks run concurrently, so it now shows up in ordinary plans.
Two stages of one Spark 4.0 job, on Comet main with DataFusion main. Each executor has 8 cores and spark.memory.offHeap.size=24g, so a task's share is 3 GiB with 8 tasks running. Spill compression is zstd.
Sort, Window, Sort, Window, then four HashAggregates (the plan of a count(DISTINCT) after two window functions). 216 tasks each sort about 4.95M rows, 1.08 GB of Arrow data:
| Comet | Spark | |
|---|---|---|
| Spills | 1,080 and 860 for the two sorts, 5 and 4 per task | only when the executor's pool was full |
| Spilled to disk | 56 GB | 19.7 GB |
| Peak off-heap execution memory per task | 653 MiB | 3 GiB |
| Free off-heap per executor during the stage, median / min | 21.8 / 19.0 GiB | 10.3 / 0.0 GiB |
The two sorts and the four aggregates all register as consumers that can spill, so each sort is held to about 3 GiB / 6 = 0.5 GiB. ExternalSorter also reserves about twice the bytes it buffers (get_reserved_bytes_for_record_batch). At their peaks the pools' reservations were 1.7x to 1.9x the live native allocations. So each sorted run holds about 0.25 GB.
Final HashAggregate with 17 hll_sketch_agg columns, feeding a RoundRobin native shuffle writer, 216 tasks. The aggregate and ShuffleRepartitioner get 1.5 GiB each. The shuffle writer receives nothing until the aggregate emits, so its half stays unused.
| Tasks | Peak off-heap execution memory | Spilled | Task time |
|---|---|---|---|
| 209 tasks on executors running only this stage | 1.5 GiB | yes (818 spills, 8.1 GB in total) | 48 to 56 s |
| 7 tasks that shared an executor with tasks of a non-native stage, so had a larger share | 5.1 GiB | no | 30 to 31 s |
Free off-heap per executor while those tasks spilled: median 19 to 20 GiB, minimum 12 GiB.
Steps to reproduce
A task plan where a blocking operator that can spill sits below other operators that can spill, run with enough concurrent tasks that off_heap_size / running_tasks binds. Examples are a sort below a window and an aggregate, or a final aggregate feeding a native shuffle writer. I do not have a minimal reproducer yet.
Expected behavior
A consumer doing the work can use the memory that registered but idle consumers are not using, as long as the task stays within its share.
Additional context
Possible directions:
- Count only the consumers that hold memory, or have reserved any, in
num_spill, and recompute the limit when another one starts. - Let a consumer exceed its fair limit while the task's total stays within
task_share, and take memory back by asking it to spill when another consumer starts. This needs a reclaim path, see Spark cannot reclaim memory from native operators (spill callback returns 0) #3873 and the closed Cooperative memory reclaim via asyncMemoryReclaimerdatafusion#22043.
Today spark.comet.exec.memoryPool=greedy_unified avoids the split, with the risks that #7088 and #7197 describe.
Related: apache/datafusion#22036, #7088, #7197, #3873, #7472.
Activity
fix: divide fair_unified's task share among active consumers