Repository navigation
Evaluate sparse string dictionary predicates only for referenced entries - #11147
joseph-isaacs wants to merge 2 commits into
Conversation
A positive row selection, such as the row ids of a late-materialization lookup, was scanned like any other: every 65,536-row chunk of the range was planned, and segments were decoded and sliced before the few selected rows were filtered out. Choose `AccessMode::RandomAccess` when a positive selection keeps fewer than one row in sixteen of the scanned range. Plan only the 65,536-row regions holding selected rows, gather selected rows from segments in their stored encoding before decoding, and keep encoded take kernels visible for sparse numeric dictionary lookups as well as string ones. Native arrays keep their zero-copy slices, and dense selections keep the scan path. `ScanBuilder::with_access_mode` overrides the choice. Rows and their order are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A predicate pushed onto dictionary values, such as a `LIKE` over a dictionary-encoded string column, was evaluated over every dictionary entry before the codes picked the few they reference. With a large dictionary and a selective chunk, most of that work was wasted, and each chunk repeated it. When a strict, infallible, static predicate over string or binary dictionary values sees fewer referenced codes than a quarter of the entries, gather the referenced entries, evaluate the predicate on those alone, and memoize each entry's result in a per-file cache that concurrent chunks share. Once the missing entries are dense, evaluate the whole dictionary once and reuse the packed result through the normal dictionary kernels. Numeric predicates and dynamic comparisons keep their vectorized evaluation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.010x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.980x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.001x ➖, 1↑ 1↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Summary
Split out of #10964. Stacked on #11146, whose commit it includes. Review only the top commit.
A predicate pushed onto dictionary values, such as a
LIKEover a dictionary-encoded string column, was evaluated over every dictionary entry before the codes picked the few they reference. With a large dictionary and a selective chunk, most of that work was wasted, and every chunk repeated it.Changes
PredicateCachethat concurrent chunks share.Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations. Base-vs-base noise: +1.3% geomean, up to ±8% per query.
develop(median / min)TPC-H string predicates run over dense chunks. ClickBench runs (string
LIKEover dictionary columns) are still in progress locally.cargo nextest run -p vortex-layout: 789 passed.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code