Skip to content

fair_unified: consumers that have not started yet take an equal share, so the running sort or aggregate spills early #7628

Open

Description

@comphead

Describe the bug

CometFairMemoryPool limits a consumer that can spill to (task_share - unspillable) / num_spill, where num_spill counts every registered consumer that can spill (fair_pool.rs). DataFusion registers consumers when execute() builds the stream tree, before any batch flows. So the consumers above a blocking operator count from the start, although they hold nothing until it emits. The operator doing the work is capped at a fraction of the task's share and spills while most of that share is unused.

This is the per-task form of apache/datafusion#22036, which describes the same policy in FairSpillPool. Since #7088 the per-consumer limit binds whenever tasks run concurrently, so it now shows up in ordinary plans.

Two stages of one Spark 4.0 job, on Comet main with DataFusion main. Each executor has 8 cores and spark.memory.offHeap.size=24g, so a task's share is 3 GiB with 8 tasks running. Spill compression is zstd.

Sort, Window, Sort, Window, then four HashAggregates (the plan of a count(DISTINCT) after two window functions). 216 tasks each sort about 4.95M rows, 1.08 GB of Arrow data:

Comet Spark
Spills 1,080 and 860 for the two sorts, 5 and 4 per task only when the executor's pool was full
Spilled to disk 56 GB 19.7 GB
Peak off-heap execution memory per task 653 MiB 3 GiB
Free off-heap per executor during the stage, median / min 21.8 / 19.0 GiB 10.3 / 0.0 GiB

The two sorts and the four aggregates all register as consumers that can spill, so each sort is held to about 3 GiB / 6 = 0.5 GiB. ExternalSorter also reserves about twice the bytes it buffers (get_reserved_bytes_for_record_batch). At their peaks the pools' reservations were 1.7x to 1.9x the live native allocations. So each sorted run holds about 0.25 GB.

Final HashAggregate with 17 hll_sketch_agg columns, feeding a RoundRobin native shuffle writer, 216 tasks. The aggregate and ShuffleRepartitioner get 1.5 GiB each. The shuffle writer receives nothing until the aggregate emits, so its half stays unused.

Tasks Peak off-heap execution memory Spilled Task time
209 tasks on executors running only this stage 1.5 GiB yes (818 spills, 8.1 GB in total) 48 to 56 s
7 tasks that shared an executor with tasks of a non-native stage, so had a larger share 5.1 GiB no 30 to 31 s

Free off-heap per executor while those tasks spilled: median 19 to 20 GiB, minimum 12 GiB.

Steps to reproduce

A task plan where a blocking operator that can spill sits below other operators that can spill, run with enough concurrent tasks that off_heap_size / running_tasks binds. Examples are a sort below a window and an aggregate, or a final aggregate feeding a native shuffle writer. I do not have a minimal reproducer yet.

Expected behavior

A consumer doing the work can use the memory that registered but idle consumers are not using, as long as the task stays within its share.

Additional context

Possible directions:

Today spark.comet.exec.memoryPool=greedy_unified avoids the split, with the risks that #7088 and #7197 describe.

Related: apache/datafusion#22036, #7088, #7197, #3873, #7472.

Activity

  1. self-assigned this
    on Nov 14, 2026
  2. added a commit that references this issue on Nov 15, 2026
    ff73d73
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions