Skip to content

fix: divide fair_unified's task share among active consumers - #7630

Open
comphead wants to merge 2 commits into
apache:mainfrom
comphead:fair-pool-idle-consumers-7628
Open

comphead wants to merge 2 commits into
apache:mainfrom
comphead:fair-pool-idle-consumers-7628

Conversation

@comphead

@comphead comphead commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7628.

Rationale for this change

CometFairMemoryPool limits a consumer that can spill to (task_share - unspillable) / num_spill, and num_spill counted every registered consumer that can spill. DataFusion registers each operator's consumer when it builds the stream tree, before any batch flows (SortExec::execute and every aggregate stream's constructor in 55.2). So the operators above a blocking one count while they hold nothing, and the operator doing the work spills while most of the task's share is unused. The issue has the production numbers.

A new test reproduces it on main with real DataFusion operators. It sorts the same input in the same pool, sized so the sort fits the task's share:

The sort Spills on main With this PR
Alone 0 0
Below another SortExec 2 0
Below a Single aggregate 2 0

Spark and Polars avoid the problem with reclaim rather than shares. Spark divides its pool among tasks that hold memory and, within a task, spills other consumers on demand. Polars' out-of-core engine spills registered frames when a global budget is exceeded. Comet cannot ask a native operator to give memory back (#3873, apache/datafusion#21422), so it keeps the shares and counts only the consumers doing work.

What changes are included in this PR?

  • num_spill now counts the active consumers that can spill, plus the one asking. A consumer is active while it holds more than its low-water mark, the least it has held since its first reservation. That leaves out a registered consumer that holds nothing, and from DataFusion 56 also an aggregate that holds only the initial table it reserves when the plan starts (fix: reserve initial table memory in aggregate streams datafusion#25900). A consumer that emptied, such as an aggregate that emitted early, is active again once it holds memory, and one that only gives memory back, such as a sort emitting what it buffered, takes no share. The check of the task's total against the task's share is unchanged, so a consumer that starts later is refused until the others give memory back.
  • This reverses one decision of fix: make fair_unified's per-consumer limit bind when tasks run concurrently #7096. A sort below an OrderedFinalAggregateStream, which has no spill path over fully sorted input, was held to half of the share so the aggregate kept the other half. The sort can now fill the share while the aggregate is idle. It gives part of that back before the aggregate's first request: everything if it spilled, and otherwise what ExternalSorter reserves beyond its sorted runs. With the share set to the sort's own peak, the aggregate's first request found 47% of the share in use when the sort emits from one batch and 75% when it merges runs. A test pins this.
  • CometAggregateSuite's "PartialMerge over a sort-merge join on a prefix of its keys spills (issue Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880)" used a 16 MiB pool because the sorts under the join got a fraction of it. They now get the whole share while they read, so 16 MiB no longer makes the PartialMerge spill, which failed the fork CI run. It now uses 2 and 8 MiB, like its sibling test, and passes at 2, 4 and 8 MiB.
  • The migration note on memory pool limits moves from "Upgrading to Comet 1.2.0" to "Upgrading to Comet 1.3.0", since fix: make fair_unified's per-consumer limit bind when tasks run concurrently #7096's per-task shares are not in 1.2.0 or branch-1.2.
  • The contributor guide, the tuning guide, the migration note and the memory review skill describe the new divisor. Two code comments that gave "would halve the limit of a sort below it" as a reason are updated. Unit tests that relied on an idle consumer taking a share now make it active with a hold helper.

DataFusion 55's hash aggregates can fail with "Decreasing allocation after spilling should succeed" when their first request is refused and the task has less free than their emptied table reports: about 8 KiB with a string grouping key, 24 bytes for collect_list, nothing for count or hll_sketch_agg with an Int64 key. main behaves the same in that state. DataFusion 56 fixes it: apache/datafusion#25900 reserves the initial table up front, so the resize after a spill shrinks, and apache/datafusion#24857 releases the emptied string map. A DataFusion-only test on DataFusion main confirmed it. With this PR on DataFusion 55, an operator below that can spill and filled the share while the aggregate was idle can also produce that state, but only if it still holds all but those last bytes when the aggregate starts. In the shapes I measured it gave back far more first: sorts at least a quarter of their peak, and the lower aggregate of a count(DISTINCT) chain at least 3% of the share even when forced. A test pins the chain, so a DataFusion bump that changes how much the lower aggregate frees fails a test instead of a query.

#7096's per-task shares are not in 1.2.0 or branch-1.2, so this needs no backport.

How are these changes tested?

New tests in fair_pool.rs, each failing against the pool logic on main:

  • the_operators_above_a_sort_take_no_share_until_they_are_active runs real DataFusion plans on CometFairMemoryPool with the fake Spark. It is the table above.
  • a_sort_that_fills_the_share_leaves_room_for_the_final_aggregate_above_it sets the share to the sort's own peak and runs an OrderedFinalAggregateStream above it, through both the single-batch and the merge emission paths, without and with DataFusion's default 10 MiB merge reservation, and with 256 or 1024 groups in each batch. The sort does not spill and the aggregate completes. On main the sort spills twice.
  • a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spills runs Spark's count(DISTINCT) chain (Single keyed by (k, d), then Partial and Final keyed by k) with the share at the lower aggregate's own peak and 1 KiB values. The Final aggregate starts with 97% of the share in use, spills twice and completes, and the lower aggregate does not spill. On main the lower aggregate spills 6 times.
  • a_consumer_takes_a_share_while_it_holds_more_than_its_low_water_mark pins the rule: an initial table takes no share but the task's total keeps its bytes, growth past it takes a share, and an aggregate that emptied takes a share again as soon as it holds memory.

I ran the memory pool tests, the Iceberg fanout writer share test and the aggregate planner tests, plus cargo clippy --all-targets --workspace -- -D warnings, cargo fmt and prettier.

Locally on -Pspark-3.5, because the default 4.1 profile does not resolve offline here, these JVM suites pass: CometAggregateSuite, CometExecSuite, CometExecIteratorLifecycleSuite, CometWindowExecSuite, CometJoinSuite, CometTaskMemoryManagerSuite, CometNativeSuite, CometPluginsSuite, CometRuntimeShutdownSuite, CometInMemoryCacheSuite, CometGenerateExecSuite, CometTopKSuite, CometMergeRowsSuite and CometExec3_4PlusSuite. That is 769 tests, with 9 canceled by Spark version gates.

A local A/B of the two shapes from the issue, base f043d64367 against this PR. The JVM build is the same, the native libraries are optimized (--profile ci), and the pool is tight. There are three rounds with alternating order, each in a fresh JVM, and five runs per query with the first dropped as warm-up. Spills are identical in every round:

Query Pool Spills, base → PR Spilled bytes, base → PR Warm median per round, base → PR
Sort, window, sort, window, count(DISTINCT) 64 MiB sorts 7 → 4 21.3 → 19.3 MB 843, 558, 561 → 668, 603, 562 ms
Final aggregate feeding a native hash shuffle 8 MiB aggregate 35 → 16, shuffle writer 7 → 10 64.4 → 39.3 MB 475, 455, 462 → 425, 349, 379 ms

The first query shows no consistent time difference at this scale. In the second, the aggregate now takes the idle shuffle writer's half. The writer spills a little more, because it starts once the aggregate emits and is refused until the aggregate gives memory back. A final aggregate that spills and reads its spill files back at a tight pool is also covered by CometAggregateSuite's #6254 tests, which pass with this change.

Not run yet: the remaining exec suites, the Spark SQL tests, and the A/B on the job from the issue, which needs a cluster run.

@github-actions github-actions Bot added bug Something isn't working area:writer Native Parquet writer area:Iceberg area:memory Memory pools, reservations, OOM handling labels Nov 14, 2026
@comphead
comphead marked this pull request as ready for review November 15, 2026 17:33
CometFairMemoryPool limited each consumer that can spill to
(task_share - unspillable) / num_spill, where num_spill counted every
registered consumer that can spill. DataFusion registers each operator's
consumer when it builds the stream tree, before any batch flows, so the
operators above a blocking one counted while they held nothing. A sort
below a window and four aggregates was held to a sixth of the task's
share and spilled while most of it was unused.

Count only the active consumers that can spill, plus the one asking. A
consumer is active while it holds more than its low-water mark, the
least it has held since its first reservation, because from DataFusion
56 an aggregate reserves its initial table when the plan starts
(apache/datafusion#25900). The task's total is still checked against the
task's share, so a consumer that starts later is refused until the
others give memory back.

A sort below an OrderedFinalAggregateStream, which apache#7096 held to half of
the share, can now fill the share while the aggregate is idle. It gives
part of that back before the aggregate's first request, which a test
checks with the share set to the sort's own peak.

The apache#6880 test of a PartialMerge over a sort-merge join used a 16 MiB
pool because the sorts below the join got a fraction of it. They now get
the whole share while they read, so it uses 2 and 8 MiB, like its
sibling test.

Closes apache#7628.
@comphead
comphead force-pushed the fair-pool-idle-consumers-7628 branch from a467c2c to ff73d73 Compare November 15, 2026 18:29
@comphead comphead changed the title fix: divide fair_unified's task share among consumers that hold memory fix: divide fair_unified's task share among active consumers Nov 15, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @comphead. The tests that run real DataFusion operators on CometFairMemoryPool make the rule easy to trust, and counting only the consumers doing work is a good fix for #7628. I also ran two SortExecs under a SortMergeJoinExec on the pool, with the share between 100% and 200% of one sort's peak. Every run completed and the total spill count was the same or lower than before the PR, so I don't see the late-starting sort getting starved in that shape.

I have a few questions I'd like answered before this merges.

Benchmark. The description says the A/B on the job from #7628 is still to come. Since this changes when every query spills, could you post it before merge? A run at a tight pool where the operator spills and then reads its spill back would help too. The effect on whole queries (fewer spills where one operator works, later consumers refused until others give memory back) is the part the unit tests can't show.

Ordered final aggregate. #7096 left room for an OrderedFinalAggregateStream by construction, and a refused request there fails the query because it has no spill path. Now the room depends on ExternalSorter giving memory back before the aggregate's first request. a_sort_that_fills_the_share_leaves_room_for_the_final_aggregate_above_it uses with_sort_spill_reservation_bytes(0) and a share equal to the sort's own peak. What happens with the merge reservation Comet actually runs with, which cannot spill and takes its bytes out of the share? Could the test also use an aggregate with many more groups per batch?

DataFusion 55 failure state. The description says an operator below an aggregate can now reach the "Decreasing allocation after spilling should succeed" state on DataFusion 55, and that in the shapes you measured the lower operator gave back far more first. The sort case is pinned by a test. Could the lower aggregate of the count(DISTINCT) chain be pinned at a small share too? Then a DataFusion bump that changes how much it frees fails a test instead of a query.

Single-reservation consumers. low_water is set by the first reservation, so a spillable consumer that reserves once and holds that memory never becomes active and takes no share. Only the task-total check bounds it. Is there a consumer in Comet that behaves this way, such as the shuffle writer or the Iceberg fanout writer? If so, is the task-total check enough for it?

Migration guide. The paragraph you edit is under "Upgrading to Comet 1.2.0" and says "Comet 1.2.0 divides spark.memory.offHeap.size among the tasks running native plans". The description says the per-task shares are not in 1.2.0, and on branch-1.2 I see pool_size / registered consumers with no task division. Should that paragraph move under 1.3.0 together with this change?

… fair pool

Restore the count(DISTINCT) chain test, so a DataFusion bump that
changes how much the lower aggregate frees fails a test instead of a
query. Run the ordered final aggregate test with DataFusion's default
merge reservation and with 1024 groups per batch, and check that the
task's total keeps the bytes of a consumer that reserved only once.

Say that the sort frees its merge reservation before emitting, and move
the memory pool note to the 1.3.0 migration section, since apache#7096 is not
in 1.2.0.
@comphead

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review. Each point below is checked against the code or a run.

Benchmark. A local A/B of the two shapes from the issue, base against this PR. The native libraries are optimized, the pool is tight, and there are three rounds with alternating order in fresh JVMs:

Query Pool Spills, base → PR Spilled bytes Warm median per round
Sort, window, sort, window, count(DISTINCT) 64 MiB sorts 7 → 4 21.3 → 19.3 MB 843, 558, 561 → 668, 603, 562 ms
Final aggregate feeding a native hash shuffle 8 MiB aggregate 35 → 16, writer 7 → 10 64.4 → 39.3 MB 475, 455, 462 → 425, 349, 379 ms

The second row shows the cost you describe: the shuffle writer starts once the aggregate emits and spills a little more, while the total falls. For spills read back at a tight pool, CometAggregateSuite's #6254 tests, where a final aggregate spills at a 3 MiB and a 2.5 MiB pool and reads its spill files back, pass with this change. So do the #6857 and #6880 PartialMerge tests. That is 769 tests across 14 exec suites, run locally. The A/B on the job from #7628 needs a cluster run. I'll post it here before this merges.

Ordered final aggregate. I measured the share in use at the aggregate's first request, with the share set to the sort's own peak:

Groups per batch Merge reservation In use at the first request
256 0 46% (one sorted batch), 75% (merged runs)
256 10 MiB, DataFusion's default, which Comet runs with 2%, 3%
1024 0 46%, 75%
1024 10 MiB 2%, 3%

The merge reservation makes room rather than taking it. ExternalSorter::sort frees it before emitting in memory (the non-spill branch of sort() in DataFusion 55.2). When the sort spilled, its own reservation is already zero. With 1024 groups per batch, the aggregate's first request grows from 19.5 KB to 53 KB. The test now runs all eight combinations and asserts that the sort does not spill and the aggregate completes.

DataFusion 55 failure state. I restored a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spills. It runs the count(DISTINCT) chain with the share at the lower aggregate's own peak, and with 1 KiB values so the lower aggregate keeps 97% of the share while it emits. The final aggregate spills and completes, and the lower aggregate does not spill. On main it spills 6 times. For DataFusion 56: apache/datafusion#25900 reserves an aggregate's initial table up front, so the resize after a spill only shrinks. A DataFusion-only test on DataFusion main confirmed that.

Single-reservation consumers. Neither Comet consumer that can spill reserves only once. ShuffleRepartitioner reserves for every batch in reserve_and_may_spill, and the Iceberg fanout writer resizes as its files grow in InnerWriter::reserve. Both become active from their second request, and one that spilled is active again as soon as it holds memory. A consumer that reserves once and never asks again needs no share. Its bytes stay in the task's total, which every other request is checked against. a_consumer_takes_a_share_while_it_holds_more_than_its_low_water_mark now asserts that a sort can use what the aggregate's single reservation leaves and is refused past it.

Migration guide. Moved under "Upgrading to Comet 1.3.0". It now says "Comet 1.3.0 divides".

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through all of these, @comphead. The measurements on the ordered final aggregate with the real merge reservation answer what I was worried about, and pinning the count(DISTINCT) chain means a DataFusion bump will tell us if that changes. The single-reservation answer makes sense too, since both of Comet's own spillable consumers reserve on every batch.

The fork CI run on 44cda511 failed in the Spark 4.1 exec job, in CometTaskMetricsSuite's "native shuffle task metrics include existing child sort spill metrics once" (CometTaskMetricsSuite.scala:823, "Native child sort did not spill"). I think this is the same effect as the #6880 test you already adjusted. The test sizes the pool so the sort under the native shuffle spills, and the idle shuffle writer no longer takes half the share, so the sort now fits. Could you tighten that test's pool, or shrink its batches, so the sort still spills? Then could you run the whole exec job, since this suite wasn't in the local list?

After that, the cluster A/B on the job from #7628 is the last thing I'd like to see before this merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:memory Memory pools, reservations, OOM handling area:writer Native Parquet writer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fair_unified: consumers that have not started yet take an equal share, so the running sort or aggregate spills early

2 participants