Repository navigation
perf: skip fragments a compaction could not merge because of dictionary capacity - #11087
shu-chen-dot-ca wants to merge 3 commits into
Conversation
|
The planner now holds back a non-reducing group's outputs only when every neighboring pair overflows a narrow key type. To check, it reads just the dictionary columns of those outputs. A rewrite that kept its count for another reason, such as a smaller row target, stays plannable. |
|
Rebased onto main (d829354). The only conflict was in the imports at the top of The doc comment on |
57a1920 to
32e1d77
Compare
There was a problem hiding this comment.
Projected dictionary reads are a reasonable trade-off for avoiding repeated writes without persisted state. The row-target case remains fixed.
The author’s update accepts and documents that smaller batch_size values can leave useful consolidation skipped until a later commit, such as a configuration update, clears the memo. Incremental compaction’s repeated rewrites from remembering only the latest rewrite remain an accepted performance risk.
Closes #10995.
Since #10368 the writer finishes a data file before a narrow-key dictionary column (
Int8,UInt8,Int16,UInt16keys) would hold more distinct values than its keys address. Compaction plans by row count, so small fragments whose values cannot share one dictionary come back as many fragments as went in, and every later run rewrote them again.The default planner now reads the transaction of the version it plans against. When that transaction is a
Rewrite, it looks at every group that had more than one input and produced at least as many outputs, and whose outputs are all still in the manifest unchanged (same data file, no deletion file, no overlays). For such a group it reads only the narrow-key dictionary columns of the outputs. It feeds each pair of neighbouring outputs through the writer's own dictionary budget (DictionaryValueBudget). Only when every pair overflows a key type do the outputs stop beingCompactWithNeighborscandidates, because a rewrite that reads each of them whole would split them apart again. A group that kept its count for another reason, such as a smaller row target in the previous run, stays plannable. Outputs are matched by data file path, because the transaction can record them before the commit assigns their ids. Deletions, overlay limits and dropped columns still compact a fragment on its own (CompactItself).The memo only lasts while that rewrite is the latest commit. An append, delete, config update or index commit makes another transaction the latest one, so the fragments are planned again. Nothing new is persisted. Datasets without a dictionary column narrower than 32-bit keys skip the check without reading the transaction. If the check fails, a warning is logged and planning goes ahead as before. Grouped planning (
plan_groups) is unchanged, since there the caller chooses the groups and their output properties.Known limits:
max_source_fragmentsonly sees the latest rewrite, so groups from earlier runs are not remembered.batch_sizecan pack values across the file boundaries of held-back fragments. Such a run merges them only after another commit (for example a config update) clears the memo.How was this patch tested
compaction_skips_fragments_dictionary_keeps_apart(V2_1 and V2_2) writes threeDictionary(Int8, Utf8)fragments of 1,000 rows with 100 values each and no values in common, then runscompact_filesrepeatedly and asserts(fragments_removed, fragments_added):(3, 3)(0, 0), with the same data files and dataset versiond = 'v0'),(3, 3)with the deletions materialized, then(0, 0)(4, 3)with fragments of 990, 1,000 and 2,000 rowsIt then checks every value against the written data and runs
validate(). On main the second run returns(3, 3)instead of(0, 0).compaction_merges_rewritten_fragments_whose_dictionaries_fit(V2_1 and V2_2) covers the changed-options case. Two fragments over the same 100 values, with 20 values deleted, give(2, 2)with a 1,000-row target, then(2, 1)with default options, then(0, 0).Run with
--profile ciand toolchain 1.98.1, on the branch rebased onto main:python3 ci/check_module_size.py origin/main HEADcargo fmt --all --checkcargo test --profile ci --locked -p lance --lib -- dataset::optimize:: dataset::tests::dataset_io::: 893 passedcargo clippy --all --tests --benches -- -D warnings: clean