Repository navigation
Conversation
An ORDER BY over 64 KiB rows fails with ResourcesExhausted under a 16 MB limit, although each input batch fits in the pool and the sort spills. The sort writes spill batches of batch_size rows, and the final merge reserves twice the largest batch of each of at least two runs. This commit adds spill_batch_bound.slt. It records the error of the wide sort on main, so the fix shows the error becoming a result. A run-end encoded sort at 32 MB passes on main and guards that the fix keeps it passing. The queries come from apache#26987. Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a sort_spill benchmark with three cases: a dictionary column with 100,000 distinct values, short strings, and 1 KiB strings. Each case uses a memory pool smaller than the input, so the sort spills and merges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpillManager records the row count of the appended batch as the largest batch of the file. A later commit writes one appended batch as several smaller ones, so the writer must report what it wrote. A named struct keeps the memory size and the row count apart, where a tuple of two usize values does not. This commit adds WrittenBatchStats and the *_with_stats append methods. No change in behavior. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spill batch has at most batch_size rows, whatever its size in bytes. A merge of spill files reserves memory for the largest batch of each file, so wide rows make that reservation larger than the pool. An operator needs a way to bound the batches it spills by bytes. This commit adds SpillManager::with_max_batch_bytes (crate-private) and SpillPieces. A bounded writer cuts each appended batch into row ranges by SpilledRowSizes, copies each range, and writes it as its own IPC message. The recorded size of a bounded writer counts the IPC padding of each buffer, so a batch read back never exceeds it. A slice of a list, run-end encoded array, union or list view keeps whole children or values, so its measured size is far above what the IPC reader allocates for it. compact_batch copies each range into fresh buffers that hold only its rows, so the measure is exact for every type and no type has to be written whole. It uses arrow take, plus fixes for what take keeps whole: view buffers, unused dictionary values and list view children. Rows of a list view can share a child range, so it copies the covered range once, and the halving of a piece stops when the halves do not get smaller. arrow-rs minify (apache/arrow-rs#12184) replaces the copy. A dictionary that the rows of a split batch share is written once per piece. A batch within the bound is written as before. No operator sets a bound yet, and an unbounded writer records the same size as before, so this commit changes no behavior. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An ORDER BY over wide rows fails with ResourcesExhausted under a memory limit, even when each input batch fits and the sort spills. The sort writes spill batches of batch_size rows, whatever their size in bytes. The final merge reserves twice the largest batch of each run and reads at least two runs, so with wide rows that reservation exceeds the pool. ExternalSorter now sets a bound on its SpillManager: the largest batch for which the smallest merge (MIN_SPILL_MERGE_FAN_IN runs, each charged twice its batch) fits in sort_spill_reservation_bytes, at least 64 KiB. It also passes the bound to StreamingMergeBuilder::with_max_batch_bytes, so every merge pass ends its output batches at the same bound. The multi-level merge keeps its halving re-spill, for single rows and rows that share data larger than the bound. The wide sort in spill_batch_bound.slt returns its result instead of the error, and the run-end encoded sort now fits the same 16 MB limit. A spilled sort can emit more, smaller batches, so the fuzz check of the output batch count becomes a lower bound. The bound formula follows a review suggestion on apache#25910, and the merge output cap and the fuzz change follow apache#26987. Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll column types The oversized-batch fuzz test asserted that the merge re-spilled runs to add output batches. The sort now bounds its spill batches, so the merge seats two runs without a re-spill. The test now checks that the sort spills and finishes. The re-spill keeps its unit tests in multi_level_merge.rs. A slice of a run-end encoded array, a dense union or a list view keeps whole children or values. New integration tests sort wide rows of each type under a memory limit and check the result against an unbounded sort. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nary A sort that splits its spilled batches can return wrong values. A valid dictionary key can reference a null value. compact_dictionary used logical_nulls, which is also null for such rows, and remapped them to key 0. The row then read the first used value, or the piece failed to build when no row used a non-null value. The compaction now skips only the rows whose key is null. A null value that a key references is kept like any other used value. The new unit test covers top-level, List, Struct and ListView dictionaries. The sort test compares a spilled sort with an unlimited one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bounded spill could write many times the data of a batch. Two cases were found in review: - Views that share a few large values. SpilledRowSizes counted each view as a copy of its value, so the batch was cut into pieces of one or two rows, and gc copied the shared value into each piece. 512 rows over three 1 MiB values wrote 537 MB instead of 3 MB. - A dictionary that many keys share. Each piece kept a copy of the values its keys used, which multiplied the file by up to 15. When the keys alone fit, halving stopped after one level and left pieces at 3.4 times the bound. The pieces now keep shared data whole wherever splitting it would only copy it: - A dictionary that SpilledRowSizes does not compact keeps all its values in every piece. The pieces share one values array, so the IPC writer writes it once and the reader decodes it once. Its bytes do not count against the bound, so a piece can exceed the bound by them. - A view array copies each referenced byte range once per piece, and rows are cut by counting only the first view of each value. When more than half of the view bytes of a batch are repeats, the batch is written whole, because any split could copy them per row. The pieces of other batches hold at most twice the bytes of the batch. - When the first half of a range is not clearly smaller than the range, the range is written whole instead of its halves. Writing shared data whole means the merge needs about the memory it needs without a bound for such batches. That is the behavior of main, where the alternative was a spill file hundreds of times larger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The byte bound of the sort's spill files also applied to the final merge pass, which produces the output and writes no spill file. A low sort_spill_reservation_bytes, as memory-limited setups use, then shrank every output batch. With a reservation of 0, narrow rows came out in batches of 580 rows instead of 8192. StreamingMergeBuilder gets with_spill_max_batch_bytes, which bounds only the merge passes whose output is spilled again. The sort uses it in place of with_max_batch_bytes. The aggregate replay keeps using with_max_batch_bytes, which bounds every pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l piece The recorded size of a piece is the upper bound that the merge reserves for each batch it reads back. With compression, the IPC writer stores a buffer that does not get smaller as it is, and the reader keeps it as a slice of the whole message. The decoded batch then also holds the message. For a List<Utf8> of compressible strings the list offsets stay uncompressed, and the batch read back was 1.3% above the recorded size, more than the reader's 4 KiB margin above 512 KiB. A bounded manager now adds the bytes written for a compressed piece. It is too large by the uncompressed buffers, at most twice the decoded size. An unbounded manager records the same size as before. The re-estimate of the multi-level merge also counts the memory size of the decoded batch, so that it does not lower the recorded size again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A string_agg over 40000 groups of 200 or 600 byte values fails with ResourcesExhausted under a 2 MB limit. The aggregate spills runs of 1 MiB batches, and the smallest merge of two runs does not fit in what the aggregate held. This commit adds the aggregate queries of apache#26987 to spill_batch_bound.slt. They pass on main and guard against regressions: array_agg over a few large groups (apache#25851), string_agg at 4 MB, and the legacy aggregate stream. It also adds the two 2 MB queries, recorded with the error of main. Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A string_agg over many groups fails with ResourcesExhausted under a 2 MB limit. The aggregate spills runs of up to 1 MiB batches, and the replay merge reserves twice the largest batch of each of at least two runs. That is more than the aggregate held when it first spilled. This commit sets a bound on the SpillManager of both aggregate streams: the 1 MiB replay target, or less when sort_spill_reservation_bytes fits only smaller batches (the config bound of apache#26987). On the first spill, the aggregate lowers the bound so that the smallest merge, plus as much again for the replay, fits in half of what it held. Later spills, such as the remainder at the end of the input, do not lower it. The aggregate writes through append_batch_with_stats, so the merge sizes its intermediate batches from the exact rows of each piece. The legacy stream also passes the bound to its merge. The two 2 MB queries in spill_batch_bound.slt return their results. The replay test allows 1 KiB over the pool limit. The merge grows a second reservation of the aggregate's consumer while the first one holds a few bytes, and FairSpillPool checks each reservation against the fair share on its own. Smaller batches let the merge reach that share exactly. The pool behavior exists on main. Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first-spill bound of an aggregate failed in two ways. The aggregate checks its memory after it aggregates a whole input batch, so its first spill can hold more than the pool. Under a 1 MB limit the 600-byte string_agg of spill_batch_bound.slt held 5 MB, got the same bound as at 2 MB, and failed. What it held is now capped by the limit of the memory pool. The bound was held / 8, so the smallest merge (4 * bound) plus as much again for the replay took all of what the aggregate held. Some limits that passed before this PR failed, for example a ROLLUP at 7 MB, and every merge read only two runs. The bound is now held / 16, which leaves half of it as slack. In the review sweep this variant fixed the ROLLUP at 7 MB and rewrote fewer rows (820 K to 660 K rows for the 2 MB string_agg). A List group key at 7 MB still fails, because the replay table needs more than the merge. That cause is not in this PR. The partitions of a query still derive their bounds on their own, so on a shared pool their bounds together can exceed it. The doc comment says so. The replay test allows the 16 bytes that FairSpillPool lets two reservations of one consumer exceed the share, not 1 KiB, so that it stays sensitive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review asked for tests that show two behaviors of the aggregate spill bound. - spilled_shared_dictionary_is_written_once groups 400 K rows by a key and a dictionary of 3000 values of 600 bytes, larger than the bound. The review measured about 240 MB of spill for this shape, before apache#25910 kept shared dictionaries whole. Both streams now write less than 30 MB. - legacy_aggregate_spill_is_bounded_by_first_spill runs array_agg in the legacy stream at 8 MB. Without the bound of the legacy stream, its merge needs the halving re-spill and writes 1.1 M rows for 300 K input rows. With it, it writes about 600 K. Both assert on spill metrics, not on timing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate cut its sorted state at the 1 MiB replay target. With a lower bound on the spill manager, the writer then split each chunk again. The sort iterator now cuts at the bound of the manager, so the writer does not copy each chunk a second time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The setting bounds the batches that a sort and an aggregate write to spill files, but its doc described only the sort's merge reservation. Users who lower it to fit a small pool, often to 0, also get smaller spill batches. The doc now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
spill-agg-first-spill-bound
branch
from
November 14, 2026 00:37
c5d2a16 to
300938d
Compare
# Conflicts: # datafusion/physical-plan/src/sorts/sort.rs
Main has apache#26264, which leaves fan-in headroom when a spilled sort merges its runs. The conflicts in sort.rs and spill_tests.rs were only imports and neighbouring code. The new test test_sort_exec_keeps_headroom_only_while_siblings_read_input sizes its pool so that every run fits the final merge without headroom but not with it. That size assumes runs of whole 1024-row batches of about 135 KiB. This branch bounds spill batches by sort_spill_reservation_bytes / 4, at least 64 KiB, so each batch is spilled in two pieces and each run reserves half as much for the merge. With the 3500 KiB pool, all 6 runs and the headroom then fit, and the idle-sibling case did not merge in an extra pass. With the bound turned off, the test passes at 3500 KiB, so the bound is the cause, not the headroom logic. The fixture pool is now 2700 KiB, inside the range (2350 to 3100 KiB) where every case of the test holds with bounded runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the merge of apache#26264 into apache#25910, and its fixture change: the headroom test sizes its pool for runs of byte-bounded spill batches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tes doc The config doc now says that the setting bounds spilled batches, and information_schema.slt prints that doc, so CI failed on the stale description. This commit regenerates the expected output with --complete. configs.md was already up to date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # docs/source/user-guide/configs.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #25910. Review only the commits after #25910: from
test: spilled aggregates over wide states in sqllogicteston. The other commits belong to #25910.Which issue does this PR close?
spill_batch_bound.slt.Rationale for this change
A
string_aggover many groups fails withResourcesExhaustedunder a 2 MB limit:The aggregate spills runs of up to 1 MiB batches (#25853). The replay merge reserves twice the largest batch of each of at least two runs, which is more than the aggregate held when it first spilled.
What changes are included in this PR?
test:the aggregate queries of fix: write sorted spill runs in byte-bounded batches and drop the halving re-spill #26987, which pass onmainand guard against regressions (array_agg over a few large groups from Hash aggregate with a few large groups fails after spilling: spill batches are sized by rows, not bytes #25851, string_agg at 4 MB, the legacy stream withenable_migration_aggregate = false). Two new 2 MB queries record the error ofmain.fix:both aggregate streams set a bound on theirSpillManager:min(1 MiB, sort_spill_reservation_bytes / 4). The 1 MiB target of fix: size aggregate spill batches by bytes as well as rows #25853 stays the default, and a smallsort_spill_reservation_byteslowers it, as in fix: write sorted spill runs in byte-bounded batches and drop the halving re-spill #26987. On the first spill, the bound drops so that the smallest merge, plus as much again for the replay, fits in half of what the aggregate held. What it held is capped by the memory limit of the pool, because the first spill can hold a whole input batch of state, more than the pool. The aggregate writes throughappend_batch_with_stats, so the merge sizes intermediate batches from the rows of each piece. The legacy stream passes the bound to its merge. The 2 MB queries return their results.fix:review fix. The bound isheld / 16(it washeld / 8, which left no slack), andheldis capped by the pool limit. A 1 MB query is added to the slt.test:review tests.spilled_shared_dictionary_is_written_onceshows that a dictionary shared by many groups is written once (the fix is in fix: bound the spill batches of an external sort by bytes #25910).legacy_aggregate_spill_is_bounded_by_first_spillfails without the legacy-stream bound (1.1 M spilled rows instead of about 600 K).perf:the aggregate cuts its sorted state at the bound of the spill file, so the writer does not split each chunk again.docs:sort_spill_reservation_bytesalso bounds spilled batches.What is the testing strategy for this PR?
spill_batch_bound.slt(see above).migrated_aggregate_spill_merge_leaves_memory_for_replayallows 16 bytes over the pool limit. The replay merge grows a second reservation of the aggregate's consumer while the first holds a few bytes.FairSpillPoolchecks each reservation against the fair share on its own, so the total can exceed the share by those bytes. That pool behavior exists onmain. Smaller batches let the merge reach the share exactly, which exposes it.aggregate_memory_call_sequence.rssnapshots do not change, because the merge keeps its re-spill.Not fixed:
array_aggover 4 groups at 6 MB still fails, onmainand here, because a group's state is too large for the pool.mainfails at all of those limits.Listgroup key at 7 MB fails, because the replay table needs more memory than the merge.Are there any user-facing changes?
Spilling aggregates can write more, smaller spill batches, and finish under smaller memory limits. No API changes.
datafusion.execution.sort_spill_reservation_bytesnow also sets the aggregate spill batch size: at mostmin(1 MiB, sort_spill_reservation_bytes / 4), and at least 64 KiB. The config doc says so.🤖 Generated with Claude Code