Repository navigation
Re-prune and re-filter Scan V2 chunks when dynamic bounds tighten - #11145
joseph-isaacs wants to merge 3 commits into
Conversation
Every conjunct plan was wrapped in a Filter, which compiled the predicate over all rows of the chunk and only then applied the rows earlier conjuncts kept. A selective first conjunct therefore saved no decoding or evaluation work for later ones. Plan conjuncts unwrapped so the selected rows reach their leaves, and wrap only when at least half of the rows remain, where evaluating the whole batch vectorized is cheaper than gathering it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Conjuncts ran in an order picked from selectivity alone, so an expensive but selective predicate, such as a string match, could run before a cheap numeric filter that rejects nearly as many rows. Share per-file estimates of each conjunct's CPU cost per row and pass rate, and order chunks by expected cost per rejected row once every conjunct has a few observations, switching only for a clear estimated gain. Recent observations are weighted more, a new dynamic bound resets its conjunct's estimates, and an occasional chunk swaps two neighbours so a conjunct hidden by short-circuiting is still measured. Chunks keep the order they started with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A chunk evaluated a dynamic conjunct, such as a TopK or join bound, once, against whatever bound existed when the chunk started. Chunks queued behind a scan's first results therefore read and filtered data the tightened bound already excluded, and passed rows to projection that a later bound rejects. Before prefetching a chunk's data, prune it again with each dynamic conjunct whose bound changed since the chunk's rows were pruned, and trim the newly excluded edge zones. Before projecting, filter again with each dynamic conjunct whose bound changed since it was evaluated. Versions are sampled before evaluation, so an update during IO or compute is checked again. Rechecks do not train the conjunct order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.002x ➖, 1↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.065x ➖, 1↑ 3↓)
datafusion / parquet / ns (1.072x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.044x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.033x ➖, 0↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Summary
Split out of #10964. Stacked on #11144 and #11143, whose commits it includes. Review only the top commit.
A chunk evaluated a dynamic conjunct, such as a TopK or join bound, once, against the bound that existed when the chunk started. Chunks queued behind a scan's first results therefore read and filtered data that a tightened bound already excluded, and passed rows to projection that a later bound rejects.
Changes
Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations. Base-vs-base noise: +1.3% geomean, up to ±8% per query.
develop(median / min)TPC-H has no TopK over a base table, so dynamic filters barely occur there. The target workloads are ClickBench-style
ORDER BY … LIMITqueries, which the CI ClickBench run covers. The DataFusion run is still in progress locally.cargo nextest run -p vortex-layout: 750 passed.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code