Repository navigation
Order Scan V2 conjuncts by measured cost per rejected row - #11144
joseph-isaacs wants to merge 2 commits into
Conversation
Every conjunct plan was wrapped in a Filter, which compiled the predicate over all rows of the chunk and only then applied the rows earlier conjuncts kept. A selective first conjunct therefore saved no decoding or evaluation work for later ones. Plan conjuncts unwrapped so the selected rows reach their leaves, and wrap only when at least half of the rows remain, where evaluating the whole batch vectorized is cheaper than gathering it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Conjuncts ran in an order picked from selectivity alone, so an expensive but selective predicate, such as a string match, could run before a cheap numeric filter that rejects nearly as many rows. Share per-file estimates of each conjunct's CPU cost per row and pass rate, and order chunks by expected cost per rejected row once every conjunct has a few observations, switching only for a clear estimated gain. Recent observations are weighted more, a new dynamic bound resets its conjunct's estimates, and an occasional chunk swaps two neighbours so a conjunct hidden by short-circuiting is still measured. Chunks keep the order they started with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.979x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.015x ➖, 0↑ 1↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 1↑ 2↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.102x ➖, 0↑ 5↓)
datafusion / parquet / ns (1.118x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.062x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.043x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Summary
Split out of #10964. Stacked on #11143, whose commit it includes. Review only the top commit.
Conjuncts ran in an order chosen from selectivity alone. An expensive but selective predicate, such as a string match, could therefore run before a cheap numeric filter that rejects nearly as many rows.
Changes
filter/ordering.rs).Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations. Base-vs-base noise: +1.3% geomean, up to ±8% per query.
develop(median / min)TPC-H's conjuncts are already well ordered by selectivity, so this shows no gain there. The DataFusion run is still in progress locally.
cargo nextest run -p vortex-layout: 747 passed, including 10 new ordering tests.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code