Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 24 additions & 18 deletions .ai/skills/review-comet-memory-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,24 +103,30 @@ configuration, from the inside out:
- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The
task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running
native plans, which `RunningTasks` counts while each task holds its task-shared pool, or
`pool_size` if that is smaller. A consumer that can spill is
refused if what it holds plus the request exceeds what the consumers that cannot spill leave
of the task's share, divided by the registered consumers that can spill, or if the task's
total plus the request exceeds the task's share. A consumer that cannot spill is refused only
if the task's total plus the request exceeds `pool_size`, which binds before Spark's own
limit only while the deprecated fraction is below `1 / N`. With eight running tasks, a 16 GiB
pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds
512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations
that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks
`reservation.size()` instead lets an operator with several reservations, such as a sort's
streaming merge, take other consumers' shares. It also depends on when DataFusion updates the
size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a
consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back
#7088: one operator takes the task's share from Spark and a merge or hash join build fails
the task. If the PR changes any of these comparisons it changes when every query spills, so
it needs benchmark evidence, not reasoning.
- [ ] **The consumers that divide the task's share are every consumer that can spill in the
task**, across every native plan, because the pool is task-shared.
`pool_size` if that is smaller. A consumer that can spill is refused if what it holds plus
the request exceeds what the consumers that cannot spill leave of the task's share, divided
by the active consumers that can spill, itself included, or if the task's total plus
the request exceeds the task's share. A consumer that cannot spill is refused only if the
task's total plus the request exceeds `pool_size`, which binds before Spark's own limit only
while the deprecated fraction is below `1 / N`. With eight running tasks, a 16 GiB pool and
two active consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge
holds 512 MiB. The pool keeps a running total for each consumer id, so the sibling
reservations that `new_empty()`, `split()` and `take()` create count against one share. A PR
that checks `reservation.size()` instead lets an operator with several reservations, such as
a sort's streaming merge, take other consumers' shares. It also depends on when DataFusion
updates the size, which is after it calls `try_grow` but before it calls `shrink`. A PR that
limits a consumer that cannot spill to a share, or sizes the shares from the whole pool,
brings back #7088: one operator takes the task's share from Spark and a merge or hash join
build fails the task. If the PR changes any of these comparisons it changes when every query
spills, so it needs benchmark evidence, not reasoning.
- [ ] **The consumers that divide the task's share are the active consumers in the task that can
spill**, across every native plan, because the pool is task-shared. A consumer is active
while it holds more than its low-water mark, the least it has held since its first
reservation. DataFusion registers every operator's consumer before any batch flows, and from
DataFusion 56 an aggregate also reserves its initial table then. A PR that divides by the
registered consumers, or counts a consumer for its first reservation, again holds a blocking
operator, such as a sort, to a fraction of the task's share while the operators above it have
not started (#7628).
- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide
`HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>`. The returned `Arc` is the only
lifetime handle, so there is no explicit release to forget and a `createPlan` that fails
Expand Down
57 changes: 35 additions & 22 deletions docs/source/contributor-guide/memory_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,17 @@ limit = (task_share - unspillable) / num_spill
```

Here `unspillable` is the bytes held by consumers that cannot spill, and `num_spill` is the number
of registered consumers that can.
of active consumers that can spill, counting the one asking. A consumer is active while it holds
more than its low-water mark, the least it has held since its first reservation. DataFusion
registers each operator's consumer when it builds the stream tree, before any batch flows, so the
operators above a blocking one, such as a sort, are registered while it reads its input. From
DataFusion 56 an aggregate also reserves its initial table then
([apache/datafusion#25900](https://github.com/apache/datafusion/pull/25900)). Neither shows that
the operator has started work, so it takes no share until it is active, which leaves its share to
the operator doing the work
([issue #7628](https://github.com/apache/datafusion-comet/issues/7628)). An operator that emptied,
such as an aggregate that emitted early, is active again once it holds memory, and one that only
gives memory back, such as a sort emitting what it buffered, takes no share.

Like Spark's own divisor, the number of running tasks follows tasks as they start and finish, so a
task that runs alone has the whole pool. It is still not Spark's number or Spark's pool, and the
Expand All @@ -382,17 +392,16 @@ Before it asks Spark for a consumer that can spill, the pool makes two checks, a
request without calling Spark if either fails:

- **The consumer against its share.** The share is what the consumers that cannot spill leave of
the task's share, divided by the number of registered consumers that can spill. What the
consumer already holds plus the request must not exceed it. That counts all of the consumer's
reservations, including the sibling reservations that `new_empty()`, `split()` and `take()`
create, so an operator that holds several reservations still gets one share. A sort's streaming
merge, for example, creates one for each batch it reads. The pool keeps a running total for each
consumer, because `reservation.size()` covers only one reservation. With eight running tasks, a
16 GiB pool and two registered consumers that can spill, the task's share is 2 GiB and each
consumer's is 1 GiB, or 768 MiB while a merge holds 512 MiB.
the task's share, divided by the number of active consumers that can spill, this one
included. What the consumer already holds plus the request must not exceed it. That counts all
of the consumer's reservations, including the sibling reservations that `new_empty()`, `split()`
and `take()` create, so an operator that holds several reservations still gets one share. A
sort's streaming merge, for example, creates one for each batch it reads. The pool keeps a
running total for each consumer, because `reservation.size()` covers only one reservation. With
eight running tasks, a 16 GiB pool and two active consumers that can spill, the task's share is
2 GiB and each consumer's is 1 GiB, or 768 MiB while a merge holds 512 MiB.
- **The task's total against the task's share.** The shares alone do not bound the total, because
a consumer keeps what it reserved while fewer consumers were registered or fewer tasks were
running.
a consumer keeps what it reserved while fewer consumers were active or fewer tasks were running.

A consumer that cannot spill takes what Spark grants. The pool refuses it only when the task's
total plus the request would exceed `pool_size`, which Spark's own limit on the task is at least as
Expand All @@ -404,21 +413,25 @@ as a workaround for [apache/datafusion#17334](https://github.com/apache/datafusi
builds a spill path only for partially sorted input. Fully sorted, as in a `CometSortAggregateExec`
over a native sort, a refusal fails the query, so the pool checks its requests only against the
pool, as it does those of a consumer that cannot spill (`limited_to_its_share` in `fair_pool.rs`).
It still counts as a consumer that can spill when the pool divides the task's share, so a sort
below it stays at half of the share and leaves the aggregate the other half. The partially sorted
stream still spills when Spark refuses it. The exemption goes by the stream's name, so it also
covers a `PartialMerge` that the planner runs as a final aggregate because its input is sorted on
some of its grouping keys. That one is not a `CometSortAggregateExec` and its input is only
partially sorted, but it is likewise checked only against the pool and Spark's grant. A
`PartialMerge` over input sorted on every grouping key stays on the hash aggregate, since the final
aggregate has no spill path there and would only halve the limit of a sort below it.
It still counts as a consumer that can spill once it is active. A sort below it reads all of its
input before the aggregate asks for any, so the sort can take the whole share, and it gives part of
that back before the aggregate's first request: all of it if it spilled, and otherwise its merge
reservation and what `ExternalSorter` reserves beyond its sorted runs and, if it merges them, their
sort keys, because it reserves about twice the bytes of each batch it buffers. The partially sorted stream still spills
when Spark refuses it. The exemption goes by the stream's name, so it also covers a `PartialMerge`
that the planner runs as a final aggregate because its input is sorted on some of its grouping
keys. That one is not a `CometSortAggregateExec` and its input is only partially sorted, but it is
likewise checked only against the pool and Spark's grant. A `PartialMerge` over input sorted on
every grouping key stays on the hash aggregate, since the final aggregate has no spill path there.

Three details matter for tuning:

- The pool is task-shared (see below), so the consumers that divide the task's share include those
of every native plan in the task, not just the plan making the request. Registering a consumer
that can spill lowers the share of every other one, but does not take back memory they already
hold. Registering one that cannot spill changes no share until it holds memory.
of every native plan in the task, not just the plan making the request. A consumer that can
spill lowers the share of every other one once it is active, not when it registers, but does not
take back memory they already hold, so a consumer that starts after another has filled the task's
share is refused until that one gives memory back. One that cannot spill changes the shares only
through the bytes it holds.
- A consumer at its share is refused, and spills, even when the rest of the task's share is free.
That keeps the memory for the task's other consumers, and is why `fair_unified` can spill earlier
than `greedy_unified`.
Expand Down
53 changes: 27 additions & 26 deletions docs/source/user-guide/latest/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,33 @@ longer has any effect. Comet always converts Arrow batches to Spark rows on the
already the default. The driver logs a warning when the setting is present in the Spark
configuration.

### Memory Pool Limits

The `fair_unified` memory pool, which is the default for `spark.comet.exec.memoryPool`, limited each
of a task's operators to a share of all of `spark.memory.offHeap.size`, while Spark limits each task
to an even share of it among the running tasks. Whenever more than one task was running, the share
was the looser limit, so one operator could take all of its task's memory, and another operator
that cannot spill, such as the final merge of a sort, then failed the task
([#7088](https://github.com/apache/datafusion-comet/issues/7088)). Spark retried such tasks, and the
retries usually succeeded. Comet 1.3.0 divides `spark.memory.offHeap.size` among the tasks running
native plans, as Spark does, and within a task limits only the operators that can spill, each to an
even share of what the operators that cannot spill leave, among those that are doing work. So sorts,
aggregations and the shuffle writer can spill sooner when several of them work at once in one task,
and the operators that cannot spill are limited only by Spark, unless the deprecated
`spark.comet.exec.memoryPool.fraction` is below `1.0`. The native Iceberg fanout writer is one of
the operators that can spill: it closes files rather than take memory from the others, so it can
write more files.

An operator's limit can now be well below what Spark would grant its task, and a sort can fail
where it used to run. A sort fails with "Not enough memory to continue external sort" instead of
spilling when a single batch is larger than its share, which wide rows, many cores per executor, a
small `spark.memory.offHeap.size` and several operators that can spill in one task make likelier.
It can also fail when operators that cannot spill, such as the build side of a shuffled hash join,
hold the whole of the task's share. To avoid either, give the executors more off-heap memory,
lower `spark.comet.batchSize`, or leave every limit to Spark by setting
`spark.comet.exec.memoryPool=greedy_unified`. See
[Configuring Comet Memory](tuning/memory.md#configuring-comet-memory).

## Upgrading to Comet 1.2.0

Comet `1.2.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below
Expand Down Expand Up @@ -128,32 +155,6 @@ that still sets it to `true` keeps the split plan, with iceberg-java writing the
on Spark 3.5+ it now also plans merge-on-read writes as the split plan. Unset it to plan Spark's
own operator.

### Memory Pool Limits

The `fair_unified` memory pool, which is the default for `spark.comet.exec.memoryPool`, limited each
of a task's operators to a share of all of `spark.memory.offHeap.size`, while Spark limits each task
to an even share of it among the running tasks. Whenever more than one task was running, the share
was the looser limit, so one operator could take all of its task's memory, and another operator
that cannot spill, such as the final merge of a sort, then failed the task
([#7088](https://github.com/apache/datafusion-comet/issues/7088)). Spark retried such tasks, and the
retries usually succeeded. Comet 1.2.0 divides `spark.memory.offHeap.size` among the tasks running
native plans, as Spark does, and within a task limits only the operators that can spill, each to an
even share of what the operators that cannot spill leave. So sorts, aggregations and the shuffle
writer can spill sooner when several of them run in one task, and the operators that cannot spill
are limited only by Spark, unless the deprecated `spark.comet.exec.memoryPool.fraction` is below
`1.0`. The native Iceberg fanout writer is one of the operators that can spill: it closes files
rather than take memory from the others, so it can write more files.

An operator's limit can now be well below what Spark would grant its task, and a sort can fail
where it used to run. A sort fails with "Not enough memory to continue external sort" instead of
spilling when a single batch is larger than its share, which wide rows, many cores per executor, a
small `spark.memory.offHeap.size` and several operators that can spill in one task make likelier.
It can also fail when operators that cannot spill, such as the build side of a shuffled hash join,
hold the whole of the task's share. To avoid either, give the executors more off-heap memory,
lower `spark.comet.batchSize`, or leave every limit to Spark by setting
`spark.comet.exec.memoryPool=greedy_unified`. See
[Configuring Comet Memory](tuning/memory.md#configuring-comet-memory).

### Native Iceberg Scan File Concurrency

`spark.comet.scan.icebergNative.dataFileConcurrencyLimit` now defaults to 2, where Comet `1.1.0`
Expand Down
13 changes: 7 additions & 6 deletions docs/source/user-guide/latest/tuning/memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,13 @@ The `fair_unified` pool keeps memory for the operators that cannot spill, such a
build side of a hash join. Spark limits each task to an even share of `spark.memory.offHeap.size` among the tasks
using it, and the pool divides `spark.memory.offHeap.size` the same way among the tasks running native plans. Within
a task, it limits each operator that can spill to an even share of what the operators that cannot spill leave of the
task's share, counting the operators of all of the task's native plans. The operators that cannot spill take what
Spark grants. So an operator that can spill sometimes spills even when there is memory to spare, in order to leave it
for the task's other operators. A limit that small can be less than one batch of wide rows, and a sort cannot spill a
batch it has not yet reserved, so it then fails with "Not enough memory to continue external sort". Give the
executors more off-heap memory, lower `spark.comet.batchSize`, or use `greedy_unified`, which leaves every limit to
Spark.
task's share, counting the operators of all of the task's native plans that are doing work. An operator that has
not started yet, such as one above a sort that is still reading its input, takes no share. The operators that cannot
spill take what Spark grants. So an operator that can spill sometimes spills even when there is memory to spare, in
order to leave it for the task's other operators. A limit that small can be less than one batch of wide rows, and a
sort cannot spill a batch it has not yet reserved, so it then fails with "Not enough memory to continue external
sort". Give the executors more off-heap memory, lower `spark.comet.batchSize`, or use `greedy_unified`, which leaves
every limit to Spark.

The `greedy_unified` pool type implements a greedy first-come first-serve limit. This pool works well for queries that do not
need to spill or have a single spillable operator.
Expand Down
Loading