Repository navigation
Use encoded random access for sparse Scan V2 row selections - #11146
joseph-isaacs wants to merge 1 commit into
Conversation
A positive row selection, such as the row ids of a late-materialization lookup, was scanned like any other: every 65,536-row chunk of the range was planned, and segments were decoded and sliced before the few selected rows were filtered out. Choose `AccessMode::RandomAccess` when a positive selection keeps fewer than one row in sixteen of the scanned range. Plan only the 65,536-row regions holding selected rows, gather selected rows from segments in their stored encoding before decoding, and keep encoded take kernels visible for sparse numeric dictionary lookups as well as string ones. Native arrays keep their zero-copy slices, and dense selections keep the scan path. `ScanBuilder::with_access_mode` overrides the choice. Rows and their order are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.013x ➖, 1↑ 1↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.082x ➖, 1↑ 5↓)
datafusion / parquet / ns (1.049x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.049x ➖, 0↑ 2↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.073x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 1↑ 3↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Summary
Split out of #10964 as an independently measurable piece.
A positive row selection, such as the row ids of a late-materialization lookup, was scanned like any other. Every 65,536-row chunk of the range was planned, and segments were decoded and sliced before the few selected rows were filtered out.
Changes
AccessMode::RandomAccessis chosen when a positive selection keeps fewer than one row in sixteen of the scanned range. Dense selections keepAccessMode::Scan.ScanBuilder::with_access_modeoverrides the choice. Rows and their order are unchanged.Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations against
develop(246a397). Base-vs-base noise: +1.3% geomean, up to ±8% per query.TPC-H issues no sparse selections, so this shows the dense path is unchanged. The ClickBench run, which exercises sparse lookups, is still in progress locally.
cargo nextest run -p vortex-layout: 768 passed.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code