Repository navigation
perf(compressor): Improve string dictionary selection - #9762
Conversation
Merging this PR will improve performance by 10.84%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (14)
Powered by Polar Signals Cloud |
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
vortex / vortex-file-compressed / bytes (0.997x ➖, 0↑ 0↓)
vortex / parquet / ns (1.001x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.996x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.072x ➖, 3↑ 4↓)
duckdb / parquet / ns (1.022x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
File Size Changes (5 files changed, -0.0% overall, 4↑ 1↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.957x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
File Size Changes (100 files changed, +0.1% overall, 67↑ 33↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.914x ➖, 2↑ 2↓)
datafusion / parquet / ns (0.949x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.770x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.936x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.014x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
File Size Changes (99 files changed, +0.1% overall, 85↑ 14↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 6↑ 3↓)
duckdb / parquet / ns (0.998x ➖, 3↑ 4↓)
File Size Changes (12 files changed, -0.1% overall, 8↑ 4↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
File Size Changes (5 files changed, -1.9% overall, 3↑ 2↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.819x ➖, 4↑ 0↓)
datafusion / parquet / ns (0.945x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed / ns (0.974x ➖, 3↑ 1↓)
duckdb / parquet / ns (0.899x ➖, 0↑ 0↓)
|
|
Do we know why some files got larger? |
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
ab167f7 to
c87ff2d
Compare
26ea413 to
90f08fa
Compare
90f08fa to
85fbc04
Compare
3ada629 to
72faa09
Compare
…eset (#10917) ## Summary - Tracking Issue: #10914 - Depends on #10915 and #10916; this branch contains both. The diff of its own is one line plus a test. `CompressionMode::Cuda` excludes `StringDictScheme`, but `execute_dict_varbinview` already decodes string dictionaries on the GPU. Keeping the scheme makes GPU-mode files 16.0% smaller across compress-bench's 17-dataset GPU suite (3,351.3 → 2,813.9 MB). Per-dataset gains: CMSprovider 45.7%, Food 52.9%, HashTags 33.2%, both TPC-H l_comment variants 12.2%, Euro2016 10.7%. The issue has the full table. ## Changes - `vortex-btrblocks/src/builder.rs`: drops `StringDictScheme` from the Cuda exclusions and updates the comment. A new test checks that the preset keeps string dictionaries but not binary ones. This is the only change to CPU-compressor code. Evidence with this branch (A40, CUDA 13.0): - `--gpu-verify` passes on all 17 datasets. develop plus only this line fails 5: taxi, CMSprovider, Euro2016, Food, HashTags. - The CPU-fallback census is identical to develop's: 59 constant-array fallbacks, none from dictionaries. - The dictionary gather takes at most 6.1% of decode time and reaches 62-85% of the measured 563 GB/s. - `vortex-cuda` lib suite on this branch: 723 passed, 0 failed. - `compress_gpu` fuzz (CI flags, ASAN): 100,000 runs in 256 s, 0 crashes. ## Benchmarks Setup: - `compress-bench --gpu-decompress --iterations 5`, a fresh process per run, `taskset -c 2-7`, 3 ABBA rounds. - Speedup = develop time / branch time. - The same-design A/A floor is 0.14 in |log|, so a "faster" claim needs a 95% CI above 1 and a speedup above 1.32×. | Dataset | File MB, develop → branch | Speedup [95% CI] | | |---|---|---:|---| | Food | 84.7 → 39.9 | 1.55× [1.41, 1.71] | faster | | CMSprovider | 771.0 → 418.5 | 1.37× [0.66, 2.86] | within noise | | Euro2016 | 163.3 → 145.9 | 1.46× [0.92, 2.32] | within noise | | TPC-H l_comment chunked | 169.6 → 148.8 | 1.36× [0.55, 3.34] | within noise | | taxi | 52.0 → 51.5 | 1.28× [0.32, 5.11] | within noise | | HashTags | 235.2 → 157.0 | 1.22× [0.89, 1.69] | within noise | | TPC-H l_comment canonical | 169.6 → 148.8 | 1.18× [0.71, 1.96] | within noise | | Arade | 144.2 → 141.7 | 1.05× [0.42, 2.68] | within noise | The nine datasets whose files don't change got one AB/BA round each. Eight came out at 1.01-1.18×, and one (wide table cols=1000 chunks=50) at 0.91×, inside the A/A floor. The measurements are from an A40 (GA102); the repo's GPU benchmarks run on A10G. ## API changes `only_cuda_compatible()`, and therefore `cuda_write_strategy`, can now emit string dictionaries. - An `ArrowDeviceArrayStream` with the default `DictionaryExport::Preserve` rejects a batch whose Arrow schema differs from the first. - With this change, 6 of CMSprovider's 14 string columns mix dictionary and constant chunks. - Today's Cuda files already do this in numeric columns: 7 of taxi's 18 numeric columns and 5 of CMSprovider's 12. - Streams with `DictionaryExport::Decode`, which the FFI scan uses, are unaffected. Overlap: #10425 adds a line next to the removed one, so whichever merges second needs a one-line rebase. #9762 changes dictionary selection, so the sizes are as of a59294e. ## AI assistance Developed with Claude Code (Claude Opus 5.5), which wrote most of the code, tests and this description under my direction. I reviewed the diff and the measurements. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tanish Gudise <tgudise@andrew.cmu.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Alexander Droste <alexander.droste@protonmail.com>
Replace the head-based distinct-count heuristic for UTF-8 dictionary admission with a distributed full-value probe followed by a bounded trial dictionary built from the complete input. A winning trial is cached and reused by the compression phase. Inconclusive probes fall back to the sample estimator, and string samples are compacted before estimation so unreferenced payload does not inflate their size. Signed-off-by: Will Manning <will@willmanning.io> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHwz6Q4P43fu8kBhnNFJKf
Replace the fixed 75% probe-distinct cutoff with a cost model. The probe estimates the input's distinct count with Chao1, and an optimistic dictionary ratio built from that estimate, the input row count, and the current best estimate decides whether a full trial is worth building. Columns with more distinct values than an 8192-row probe can see, but that still compress well as a dictionary, are no longer rejected. Fall back to sample estimation, rather than skipping, when the trial would exceed its entry, storage, or work bounds. A one-pass profile checks the input size before building, so the bytes dictionary builder no longer needs an input-byte limit. Score the trial against the compacted input size, matching the compacted samples other schemes are estimated on. Evaluate sampled estimates before deferred callbacks so callbacks see the tightest threshold regardless of registration order. Apply the same selection to BinaryDictScheme. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHwz6Q4P43fu8kBhnNFJKf
DictStrategy decided whether to dictionary-encode a column by fully compressing a probe sample and checking for a Dict result. Add `CompressorPlugin::chooses_dictionary`, which defaults to that, and override it for BtrBlocks to run only scheme selection through the new `CascadingCompressor::choose_scheme`. The probe and chunk compression now share one decision path, and the probe no longer compresses the winning scheme's output just to read its encoding. Share the code-width rule. `codes_ptype_for_max_len` picks the code type for the bytes builder and the trial, and the layout documents it. Measure trial and sample inputs with the compressor's `exact_uncompressed_size` instead of separate compaction and byte accounting. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHwz6Q4P43fu8kBhnNFJKf
A trait method on the public `CompressorPlugin` cannot be private, and the selection-only probe it enabled saved no measurable time on the compression benchmarks. Restore DictStrategy's probe to compressing the sample and checking for a Dict result, and remove the `CascadingCompressor::choose_scheme` entry point that only it used. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHwz6Q4P43fu8kBhnNFJKf
Profiling HashTags showed two avoidable costs in dictionary selection. Trials that lost spent nearly all their time compressing dictionary values. The cost model assumed the values compress as well as the best scheme compresses the whole column, but the best scheme often wins by exploiting repetition the dictionary removes, such as sparse nulls. Dictionary values are distinct, and FSST and OnPair can compress a value at most eight-fold, so bound the assumed values ratio by that. Near-unique columns paid to hash all 8192 probe rows before rejection. Probe a 2048-row head first, and reject without the full probe when none of its 1024 or more values repeat and the cost model rejects. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHwz6Q4P43fu8kBhnNFJKf
A dictionary trial reports the ratio it measured, but it competed against estimates that could be far better than what their schemes deliver. Comparing every real string column against a forced dictionary showed two such estimates costing 1% of all compressed string bytes. The string NullDominatedSparseScheme estimated the fraction of rows that are null. It keeps every value's bytes, so on long values it promised ~25x and delivered 1.3x. Estimate from the bytes it removes and adds instead. A 1% sample of a column that is 99.9% null often holds only nulls, which compress to nothing under any scheme. Give such a sample no ratio. Keep a trial's output even when its ratio loses. The compressor now prefers it to the winner's output when it is smaller, since the trial was measured and the winner was only estimated. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHwz6Q4P43fu8kBhnNFJKf
0640540 to
524ac05
Compare
## Summary QC `develop` is missing 13 upstream commits. Merge `vortex-data/vortex:develop` at `cb613ac1c0adc56cbe63dfc79e4bbc431ad0bc52` into QC `develop` at `b2c429ac2830b883cb37c1ff9262dc9e3cb7e483`. Merge this PR with a merge commit to preserve upstream ancestry for future syncs. ## Changes - Import typed Arrow extension plugins and round-trip metadata preservation, run-end export fixes, and DuckDB dictionary reuse across chunks ([vortex-data#11360](vortex-data#11360), [vortex-data#11375](vortex-data#11375), [vortex-data#11378](vortex-data#11378)). - Bring in improved string dictionary selection, spatial contains/within pruning, and safe PCO decoding ([vortex-data#9762](vortex-data#9762), [vortex-data#11372](vortex-data#11372), [vortex-data#11363](vortex-data#11363)). - Include CUDA Widen execution, bounded dictionary gathers, encoded-child host copies, and string dictionaries in the CUDA compression preset. - Reconcile the upstream Java I/O ports with their existing QC implementations, preserving read metrics, Kryo-safe Hadoop broadcasts, lifecycle handling, session defaults, and regression coverage. Correct the coalescing default's Javadoc to 1 MiB. ## Testing - Pinned `nightly-2026-09-10` Rust formatting for all affected crates, root/subproject `./gradlew spotlessApply`, and clang-format for the changed CUDA kernels passed. Reviewed and included the formatter diff. - `git diff --check` passed; no unresolved merge entries or conflict markers remain. The worktree is clean and both remote tips are parents of the signed-off merge commit. - All 13 imported commits have sign-off trailers and retain their original upstream metadata. - Tests, builds, lint, and benchmarks were not run locally, following repository guidance. --------- Signed-off-by: Daniel King <dan@spiraldb.com> Signed-off-by: Tanish Gudise <tgudise@andrew.cmu.edu> Signed-off-by: Alexander Droste <alexander.droste@protonmail.com> Signed-off-by: Christopher Wirt <cwirt@quantcast.com> Signed-off-by: Nemo Yu <zyu379@wisc.edu> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Signed-off-by: Will Manning <will@willmanning.io> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-authored-by: Dan King <dan@spiraldb.com> Co-authored-by: Tanish Gudise <tgudise@andrew.cmu.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Alexander Droste <alexander.droste@protonmail.com> Co-authored-by: Connor Tsui <connor.tsui20@gmail.com> Co-authored-by: Nemo Yu <83347615+HarukiMoriarty@users.noreply.github.com> Co-authored-by: Will Manning <will@spiraldb.com>
String Dictionary selection now probes a distributed sample before building an exact candidate. The previous estimator missed profitable dictionaries when columns contained thousands of repeated strings. Samples above 75% distinct values reject Dictionary; an incomplete probe uses the existing estimator. Admitted candidates have explicit limits for dictionary entries, value bytes, code bytes, and input bytes. The compressor compares the candidate's actual size and reuses it if it wins.
The probe reads at most 8,192 rows and 1 MiB of values. A row that exceeds the byte budget is excluded from the completed-row count, including the final sampled row. String samples compact their payload before size estimates so unrelated retained bytes do not distort the score.
Merged
develop, preserving explicit allocator use and updating the integration tests and benchmark for the current compressor APIs. Adversarial review covered probe boundaries, null handling, candidate work/storage limits, sampling fallback, and candidate reuse. The final-row byte-budget regression fails with the previous code and passes with the fix.Validation: 165 targeted and golden tests passed; formatting passed with nightly-2026-09-10; clippy passed for vortex-array, vortex-compressor, and vortex-btrblocks with all targets, all features, and warnings denied. No golden snapshots changed.
Fresh CI compression results compare
ab167f77withdevelopat05d0cba3. HashTags used 1.2% fewer bytes, with write time +0.8% and full materialization time +3.3%. Euro2016 used 0.1% fewer bytes, with write time +2.0% and full materialization time -4.1%. These percentages come from one CI comparison; the report flagged no runtime regressions. Dictionary expansion still carries a materialization tradeoff, while engines preserving Dictionary can retain the encoded form.All current build, lint, test, compatibility, coverage, sanitizer, compression benchmark, SQL benchmark, and CodSpeed Performance Analysis checks passed. The PR is ready for human review; the approval policy awaits a human reviewer.