Skip to content

fix: bound aggregate spill batches by the memory of the first spill - #27343

Draft
adriangb wants to merge 21 commits into
apache:mainfrom
adriangb:spill-agg-first-spill-bound
Draft

adriangb wants to merge 21 commits into
apache:mainfrom
adriangb:spill-agg-first-spill-bound

Conversation

@adriangb

@adriangb adriangb commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Note

Stacked on #25910. Review only the commits after #25910: from test: spilled aggregates over wide states in sqllogictest on. The other commits belong to #25910.

Which issue does this PR close?

Rationale for this change

A string_agg over many groups fails with ResourcesExhausted under a 2 MB limit:

SET datafusion.execution.target_partitions = 1;
SET datafusion.runtime.memory_limit = '2M';
SELECT count(*), sum(length(s)) FROM (
  SELECT k, string_agg(v, ',') AS s
  FROM (SELECT value % 40000 AS k, repeat('x', 200) AS v FROM generate_series(1, 160000))
  GROUP BY k
);
-- main: Resources exhausted: Additional allocation failed for SingleHashAggregateStream[0]

The aggregate spills runs of up to 1 MiB batches (#25853). The replay merge reserves twice the largest batch of each of at least two runs, which is more than the aggregate held when it first spilled.

What changes are included in this PR?

  1. test: the aggregate queries of fix: write sorted spill runs in byte-bounded batches and drop the halving re-spill #26987, which pass on main and guard against regressions (array_agg over a few large groups from Hash aggregate with a few large groups fails after spilling: spill batches are sized by rows, not bytes #25851, string_agg at 4 MB, the legacy stream with enable_migration_aggregate = false). Two new 2 MB queries record the error of main.
  2. fix: both aggregate streams set a bound on their SpillManager: min(1 MiB, sort_spill_reservation_bytes / 4). The 1 MiB target of fix: size aggregate spill batches by bytes as well as rows #25853 stays the default, and a small sort_spill_reservation_bytes lowers it, as in fix: write sorted spill runs in byte-bounded batches and drop the halving re-spill #26987. On the first spill, the bound drops so that the smallest merge, plus as much again for the replay, fits in half of what the aggregate held. What it held is capped by the memory limit of the pool, because the first spill can hold a whole input batch of state, more than the pool. The aggregate writes through append_batch_with_stats, so the merge sizes intermediate batches from the rows of each piece. The legacy stream passes the bound to its merge. The 2 MB queries return their results.
  3. fix: review fix. The bound is held / 16 (it was held / 8, which left no slack), and held is capped by the pool limit. A 1 MB query is added to the slt.
  4. test: review tests. spilled_shared_dictionary_is_written_once shows that a dictionary shared by many groups is written once (the fix is in fix: bound the spill batches of an external sort by bytes #25910). legacy_aggregate_spill_is_bounded_by_first_spill fails without the legacy-stream bound (1.1 M spilled rows instead of about 600 K).
  5. perf: the aggregate cuts its sorted state at the bound of the spill file, so the writer does not split each chunk again.
  6. docs: sort_spill_reservation_bytes also bounds spilled batches.

What is the testing strategy for this PR?

  • spill_batch_bound.slt (see above).
  • migrated_aggregate_spill_merge_leaves_memory_for_replay allows 16 bytes over the pool limit. The replay merge grows a second reservation of the aggregate's consumer while the first holds a few bytes. FairSpillPool checks each reservation against the fair share on its own, so the total can exceed the share by those bytes. That pool behavior exists on main. Smaller batches let the merge reach the share exactly, which exposes it.
  • The aggregate_memory_call_sequence.rs snapshots do not change, because the merge keeps its re-spill.
  • The extended suite passes.

Not fixed:

  • array_agg over 4 groups at 6 MB still fails, on main and here, because a group's state is too large for the pool.
  • With several partitions on a shared pool, each partition derives its bound from its own first spill, so the bounds together can exceed the pool. The same query can pass or fail from run to run (seen with 4 partitions at 2 MB to 8 MB). main fails at all of those limits.
  • A List group key at 7 MB fails, because the replay table needs more memory than the merge.

Are there any user-facing changes?

Spilling aggregates can write more, smaller spill batches, and finish under smaller memory limits. No API changes.

datafusion.execution.sort_spill_reservation_bytes now also sets the aggregate spill batch size: at most min(1 MiB, sort_spill_reservation_bytes / 4), and at least 64 KiB. The config doc says so.

🤖 Generated with Claude Code

adriangb and others added 6 commits November 11, 2026 12:46
An ORDER BY over 64 KiB rows fails with ResourcesExhausted under a 16 MB
limit, although each input batch fits in the pool and the sort spills.
The sort writes spill batches of batch_size rows, and the final merge
reserves twice the largest batch of each of at least two runs.

This commit adds spill_batch_bound.slt. It records the error of the wide
sort on main, so the fix shows the error becoming a result. A run-end
encoded sort at 32 MB passes on main and guards that the fix keeps it
passing. The queries come from apache#26987.

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a sort_spill benchmark with three cases: a dictionary column with
100,000 distinct values, short strings, and 1 KiB strings. Each case uses a
memory pool smaller than the input, so the sort spills and merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpillManager records the row count of the appended batch as the largest
batch of the file. A later commit writes one appended batch as several
smaller ones, so the writer must report what it wrote. A named struct
keeps the memory size and the row count apart, where a tuple of two
usize values does not.

This commit adds WrittenBatchStats and the *_with_stats append methods.
No change in behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spill batch has at most batch_size rows, whatever its size in bytes. A
merge of spill files reserves memory for the largest batch of each file,
so wide rows make that reservation larger than the pool. An operator
needs a way to bound the batches it spills by bytes.

This commit adds SpillManager::with_max_batch_bytes (crate-private) and
SpillPieces. A bounded writer cuts each appended batch into row ranges
by SpilledRowSizes, copies each range, and writes it as its own IPC
message. The recorded size of a bounded writer counts the IPC padding of
each buffer, so a batch read back never exceeds it.

A slice of a list, run-end encoded array, union or list view keeps whole
children or values, so its measured size is far above what the IPC reader
allocates for it. compact_batch copies each range into fresh buffers
that hold only its rows, so the measure is exact for every type and no
type has to be written whole. It uses arrow take, plus fixes for what
take keeps whole: view buffers, unused dictionary values and list view
children. Rows of a list view can share a child range, so it copies the
covered range once, and the halving of a piece stops when the halves do
not get smaller. arrow-rs minify (apache/arrow-rs#12184) replaces the
copy.

A dictionary that the rows of a split batch share is written once per
piece. A batch within the bound is written as before.

No operator sets a bound yet, and an unbounded writer records the same
size as before, so this commit changes no behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ORDER BY over wide rows fails with ResourcesExhausted under a memory
limit, even when each input batch fits and the sort spills. The sort
writes spill batches of batch_size rows, whatever their size in bytes.
The final merge reserves twice the largest batch of each run and reads
at least two runs, so with wide rows that reservation exceeds the pool.

ExternalSorter now sets a bound on its SpillManager: the largest batch
for which the smallest merge (MIN_SPILL_MERGE_FAN_IN runs, each charged
twice its batch) fits in sort_spill_reservation_bytes, at least 64 KiB.
It also passes the bound to StreamingMergeBuilder::with_max_batch_bytes,
so every merge pass ends its output batches at the same bound. The
multi-level merge keeps its halving re-spill, for single rows and rows
that share data larger than the bound.

The wide sort in spill_batch_bound.slt returns its result instead of the
error, and the run-end encoded sort now fits the same 16 MB limit. A
spilled sort can emit more, smaller batches, so the fuzz check
of the output batch count becomes a lower bound.

The bound formula follows a review suggestion on apache#25910,
and the merge output cap and the fuzz change follow
apache#26987.

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll column types

The oversized-batch fuzz test asserted that the merge re-spilled runs to
add output batches. The sort now bounds its spill batches, so the merge
seats two runs without a re-spill. The test now checks that the sort
spills and finishes. The re-spill keeps its unit tests in
multi_level_merge.rs.

A slice of a run-end encoded array, a dense union or a list view keeps
whole children or values. New integration tests sort wide rows of each
type under a memory limit and check the result against an unbounded
sort.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Nov 12, 2026
adriangb and others added 10 commits November 13, 2026 18:09
…nary

A sort that splits its spilled batches can return wrong values. A valid
dictionary key can reference a null value. compact_dictionary used
logical_nulls, which is also null for such rows, and remapped them to
key 0. The row then read the first used value, or the piece failed to
build when no row used a non-null value.

The compaction now skips only the rows whose key is null. A null value
that a key references is kept like any other used value. The new unit
test covers top-level, List, Struct and ListView dictionaries. The sort
test compares a spilled sort with an unlimited one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bounded spill could write many times the data of a batch. Two cases
were found in review:

- Views that share a few large values. SpilledRowSizes counted each view
  as a copy of its value, so the batch was cut into pieces of one or two
  rows, and gc copied the shared value into each piece. 512 rows over
  three 1 MiB values wrote 537 MB instead of 3 MB.
- A dictionary that many keys share. Each piece kept a copy of the
  values its keys used, which multiplied the file by up to 15. When the
  keys alone fit, halving stopped after one level and left pieces at 3.4
  times the bound.

The pieces now keep shared data whole wherever splitting it would only
copy it:

- A dictionary that SpilledRowSizes does not compact keeps all its
  values in every piece. The pieces share one values array, so the IPC
  writer writes it once and the reader decodes it once. Its bytes do not
  count against the bound, so a piece can exceed the bound by them.
- A view array copies each referenced byte range once per piece, and
  rows are cut by counting only the first view of each value. When more
  than half of the view bytes of a batch are repeats, the batch is
  written whole, because any split could copy them per row. The pieces
  of other batches hold at most twice the bytes of the batch.
- When the first half of a range is not clearly smaller than the range,
  the range is written whole instead of its halves.

Writing shared data whole means the merge needs about the memory it
needs without a bound for such batches. That is the behavior of main,
where the alternative was a spill file hundreds of times larger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The byte bound of the sort's spill files also applied to the final merge
pass, which produces the output and writes no spill file. A low
sort_spill_reservation_bytes, as memory-limited setups use, then shrank
every output batch. With a reservation of 0, narrow rows came out in
batches of 580 rows instead of 8192.

StreamingMergeBuilder gets with_spill_max_batch_bytes, which bounds only
the merge passes whose output is spilled again. The sort uses it in
place of with_max_batch_bytes. The aggregate replay keeps using
with_max_batch_bytes, which bounds every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l piece

The recorded size of a piece is the upper bound that the merge reserves
for each batch it reads back. With compression, the IPC writer stores a
buffer that does not get smaller as it is, and the reader keeps it as a
slice of the whole message. The decoded batch then also holds the
message. For a List<Utf8> of compressible strings the list offsets stay
uncompressed, and the batch read back was 1.3% above the recorded size,
more than the reader's 4 KiB margin above 512 KiB.

A bounded manager now adds the bytes written for a compressed piece. It
is too large by the uncompressed buffers, at most twice the decoded
size. An unbounded manager records the same size as before. The
re-estimate of the multi-level merge also counts the memory size of the
decoded batch, so that it does not lower the recorded size again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A string_agg over 40000 groups of 200 or 600 byte values fails with
ResourcesExhausted under a 2 MB limit. The aggregate spills runs of
1 MiB batches, and the smallest merge of two runs does not fit in what
the aggregate held.

This commit adds the aggregate queries of apache#26987 to
spill_batch_bound.slt. They pass on main and guard against regressions:
array_agg over a few large groups (apache#25851), string_agg at 4 MB, and the
legacy aggregate stream. It also adds the two 2 MB queries, recorded
with the error of main.

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A string_agg over many groups fails with ResourcesExhausted under a 2 MB
limit. The aggregate spills runs of up to 1 MiB batches, and the replay
merge reserves twice the largest batch of each of at least two runs.
That is more than the aggregate held when it first spilled.

This commit sets a bound on the SpillManager of both aggregate streams:
the 1 MiB replay target, or less when sort_spill_reservation_bytes fits
only smaller batches (the config bound of apache#26987). On
the first spill, the aggregate lowers the bound so that the smallest
merge, plus as much again for the replay, fits in half of what it held.
Later spills, such as the remainder at the end of the input, do not lower
it. The aggregate writes through append_batch_with_stats, so the merge
sizes its intermediate batches from the exact rows of each piece. The
legacy stream also passes the bound to its merge.

The two 2 MB queries in spill_batch_bound.slt return their results.

The replay test allows 1 KiB over the pool limit. The merge grows a
second reservation of the aggregate's consumer while the first one
holds a few bytes, and FairSpillPool checks each reservation against
the fair share on its own. Smaller batches let the merge reach that
share exactly. The pool behavior exists on main.

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first-spill bound of an aggregate failed in two ways.

The aggregate checks its memory after it aggregates a whole input batch,
so its first spill can hold more than the pool. Under a 1 MB limit the
600-byte string_agg of spill_batch_bound.slt held 5 MB, got the same
bound as at 2 MB, and failed. What it held is now capped by the limit of
the memory pool.

The bound was held / 8, so the smallest merge (4 * bound) plus as much
again for the replay took all of what the aggregate held. Some limits
that passed before this PR failed, for example a ROLLUP at 7 MB, and
every merge read only two runs. The bound is now held / 16, which leaves
half of it as slack. In the review sweep this variant fixed the ROLLUP at
7 MB and rewrote fewer rows (820 K to 660 K rows for the 2 MB
string_agg). A List group key at 7 MB still fails, because the replay
table needs more than the merge. That cause is not in this PR.

The partitions of a query still derive their bounds on their own, so on
a shared pool their bounds together can exceed it. The doc comment says
so.

The replay test allows the 16 bytes that FairSpillPool lets two
reservations of one consumer exceed the share, not 1 KiB, so that it
stays sensitive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review asked for tests that show two behaviors of the aggregate
spill bound.

- spilled_shared_dictionary_is_written_once groups 400 K rows by a key
  and a dictionary of 3000 values of 600 bytes, larger than the bound.
  The review measured about 240 MB of spill for this shape, before
  apache#25910 kept shared
  dictionaries whole. Both streams now write less than 30 MB.
- legacy_aggregate_spill_is_bounded_by_first_spill runs array_agg in the
  legacy stream at 8 MB. Without the bound of the legacy stream, its
  merge needs the halving re-spill and writes 1.1 M rows for 300 K input
  rows. With it, it writes about 600 K.

Both assert on spill metrics, not on timing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate cut its sorted state at the 1 MiB replay target. With a
lower bound on the spill manager, the writer then split each chunk
again. The sort iterator now cuts at the bound of the manager, so the
writer does not copy each chunk a second time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The setting bounds the batches that a sort and an aggregate write to
spill files, but its doc described only the sort's merge reservation.
Users who lower it to fit a small pool, often to 0, also get smaller
spill batches. The doc now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adriangb
adriangb force-pushed the spill-agg-first-spill-bound branch from c5d2a16 to 300938d Compare November 14, 2026 00:37
# Conflicts:
#	datafusion/physical-plan/src/sorts/sort.rs
@github-actions github-actions Bot added documentation Improvements or additions to documentation common Related to common crate labels Nov 14, 2026
adriangb and others added 4 commits November 13, 2026 19:07
Main has apache#26264, which leaves
fan-in headroom when a spilled sort merges its runs. The conflicts in
sort.rs and spill_tests.rs were only imports and neighbouring code.

The new test test_sort_exec_keeps_headroom_only_while_siblings_read_input
sizes its pool so that every run fits the final merge without headroom
but not with it. That size assumes runs of whole 1024-row batches of
about 135 KiB. This branch bounds spill batches by
sort_spill_reservation_bytes / 4, at least 64 KiB, so each batch is
spilled in two pieces and each run reserves half as much for the merge.
With the 3500 KiB pool, all 6 runs and the headroom then fit, and the
idle-sibling case did not merge in an extra pass. With the bound turned
off, the test passes at 3500 KiB, so the bound is the cause, not the
headroom logic. The fixture pool is now 2700 KiB, inside the range
(2350 to 3100 KiB) where every case of the test holds with bounded runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the merge of apache#26264
into apache#25910, and its fixture
change: the headroom test sizes its pool for runs of byte-bounded spill
batches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tes doc

The config doc now says that the setting bounds spilled batches, and
information_schema.slt prints that doc, so CI failed on the stale
description. This commit regenerates the expected output with
--complete. configs.md was already up to date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant