Skip to content

perf: skip fragments a compaction could not merge because of dictionary capacity - #11087

Open
shu-chen-dot-ca wants to merge 3 commits into
lance-format:mainfrom
shu-chen-dot-ca:perf-10995-skip-ineffective-compaction
Open

shu-chen-dot-ca wants to merge 3 commits into
lance-format:mainfrom
shu-chen-dot-ca:perf-10995-skip-ineffective-compaction

Conversation

@shu-chen-dot-ca

@shu-chen-dot-ca shu-chen-dot-ca commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #10995.

Since #10368 the writer finishes a data file before a narrow-key dictionary column (Int8, UInt8, Int16, UInt16 keys) would hold more distinct values than its keys address. Compaction plans by row count, so small fragments whose values cannot share one dictionary come back as many fragments as went in, and every later run rewrote them again.

The default planner now reads the transaction of the version it plans against. When that transaction is a Rewrite, it looks at every group that had more than one input and produced at least as many outputs, and whose outputs are all still in the manifest unchanged (same data file, no deletion file, no overlays). For such a group it reads only the narrow-key dictionary columns of the outputs. It feeds each pair of neighbouring outputs through the writer's own dictionary budget (DictionaryValueBudget). Only when every pair overflows a key type do the outputs stop being CompactWithNeighbors candidates, because a rewrite that reads each of them whole would split them apart again. A group that kept its count for another reason, such as a smaller row target in the previous run, stays plannable. Outputs are matched by data file path, because the transaction can record them before the commit assigns their ids. Deletions, overlay limits and dropped columns still compact a fragment on its own (CompactItself).

The memo only lasts while that rewrite is the latest commit. An append, delete, config update or index commit makes another transaction the latest one, so the fragments are planned again. Nothing new is persisted. Datasets without a dictionary column narrower than 32-bit keys skip the check without reading the transaction. If the check fails, a warning is logged and planning goes ahead as before. Grouped planning (plan_groups) is unchanged, since there the caller chooses the groups and their output properties.

Known limits:

  • A rewrite that reduced the fragment count, but not all the way, has no memo, so one more ineffective run can follow it.
  • Incremental compaction with max_source_fragments only sees the latest rewrite, so groups from earlier runs are not remembered.
  • A run with a smaller batch_size can pack values across the file boundaries of held-back fragments. Such a run merges them only after another commit (for example a config update) clears the memo.

How was this patch tested

compaction_skips_fragments_dictionary_keeps_apart (V2_1 and V2_2) writes three Dictionary(Int8, Utf8) fragments of 1,000 rows with 100 values each and no values in common, then runs compact_files repeatedly and asserts (fragments_removed, fragments_added):

  • first run (3, 3)
  • next two runs (0, 0), with the same data files and dataset version
  • after deleting 10 rows (d = 'v0'), (3, 3) with the deletions materialized, then (0, 0)
  • after appending rows that share the last fragment's values, (4, 3) with fragments of 990, 1,000 and 2,000 rows

It then checks every value against the written data and runs validate(). On main the second run returns (3, 3) instead of (0, 0).

compaction_merges_rewritten_fragments_whose_dictionaries_fit (V2_1 and V2_2) covers the changed-options case. Two fragments over the same 100 values, with 20 values deleted, give (2, 2) with a 1,000-row target, then (2, 1) with default options, then (0, 0).

Run with --profile ci and toolchain 1.98.1, on the branch rebased onto main:

  • python3 ci/check_module_size.py origin/main HEAD
  • cargo fmt --all --check
  • cargo test --profile ci --locked -p lance --lib -- dataset::optimize:: dataset::tests::dataset_io::: 893 passed
  • cargo clippy --all --tests --benches -- -D warnings: clean
Before After
before after

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 15, 2026
@shu-chen-dot-ca

Copy link
Copy Markdown
Contributor Author

The planner now holds back a non-reducing group's outputs only when every neighboring pair overflows a narrow key type. To check, it reads just the dictionary columns of those outputs. A rewrite that kept its count for another reason, such as a smaller row target, stays plannable. compaction_merges_rewritten_fragments_whose_dictionaries_fit covers that case: (2, 2) with a 1,000-row target, then (2, 1) with default options, then (0, 0). Incremental runs with max_source_fragments still only see the latest rewrite, as stated in the description.

@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 15, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 15, 2026
@shu-chen-dot-ca

Copy link
Copy Markdown
Contributor Author

Rebased onto main (d829354). The only conflict was in the imports at the top of planning.rs; I kept both sides (main's new Cow import and this PR's VecDeque).

The doc comment on are_kept_apart_by_dictionaries no longer claims the outputs would be split apart again whatever the compaction options. It now says plainly that a smaller batch_size can still pack values across the file boundaries, and that such a run merges the held-back fragments only once a later commit clears the check. I'll add this to the description as a third known limit.

@shu-chen-dot-ca
shu-chen-dot-ca force-pushed the perf-10995-skip-ineffective-compaction branch from 57a1920 to 32e1d77 Compare November 16, 2026 18:44
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 16, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

Projected dictionary reads are a reasonable trade-off for avoiding repeated writes without persisted state. The row-target case remains fixed.

The author’s update accepts and documents that smaller batch_size values can leave useful consolidation skipped until a later commit, such as a configuration update, clears the memo. Incremental compaction’s repeated rewrites from remembering only the latest rewrite remain an accepted performance risk.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: avoid repeated ineffective compaction of dictionary-limited fragments

1 participant