Repository navigation
Push DataFusion dynamic filters into V2 scans - #11073
joseph-isaacs wants to merge 5 commits into
Conversation
Polar Signals Profiling ResultsLatest Run
Previous Runs (21)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.006x ➖, 1↑ 1↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.034x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.981x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.990x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.980x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.081x ➖, 2↑ 38↓)
datafusion / parquet / ns (0.986x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.974x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.973x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.954x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 3↓)
duckdb / parquet / ns (0.928x ➖, 10↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.772x ➖, 5↑ 0↓)
datafusion / parquet / ns (0.936x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.928x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.926x ➖, 0↑ 0↓)
|
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
Inline roots shared one session, and the run clears a root's session when the root retires. Once the first root finished, the remaining roots waited for reads that were never issued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Evaluate TopK and hash join dynamic filters in the V2 file scan, so zones and rows are dropped before the engine sees them. The producing operator keeps enforcing them. V1 scans keep them out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's hash table lookup has no Vortex equivalent, so one unconvertible conjunct turned the whole filter into a constant true and its bounds were lost. Convert each conjunct alone and keep the ones that convert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
7a59376 to
58a4d05
Compare
A join's dynamic filter bounds its probe keys with key >= min AND key <= max. Each comparison decoded the key column again, and the scan's between rewrite only ran on static filters, so the bounds now merge into one between when the filter is read. TPC-DS SF10, 10 join queries against the previous commit: geomean 0.972, Q69 and Q10 0.93, none slower than 1.007. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A scan of at most one filtered chunk evaluates its conjuncts on that chunk next, so pruning its zones first saves little and delays the chunk, and pruning a dynamic conjunct proves its current predicate over the zones again. DataFusion splits small join probes into many such scans. Larger scans keep pruning their dynamic conjuncts: skipping it for every scan made TPC-DS SF10 Q82 and Q37 3.4-3.7x slower. TPC-DS, all 99 queries against the previous commit: SF1 geomean 0.97, SF10 0.99 on best runs and 1.01 on medians; the SF10 outliers were flat when rerun. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Ported from #10789 into the combined V2 scan perf branch. Kept the branch's ScanCache OnceCell API, chunk sizing (#10964) and filter planner (#11073/#11087); added the cores-per-reader concurrency and inline gate, per-root sessions in chunk tasks, batch-before-poll IO registration, MaskScan (wrapped inside the branch's ActiveConjunct and timed prune planner), one-scan windowed ProjectionMorsel for filtered chunks, the optimizer rules and V2_MIN_ROWS_PER_PARTITION; concurrency only shrinks filtered chunks (floor 64Ki), dropped projected_rows_per_chunk since unfiltered projections keep streaming stored batches in 1Mi chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBQHsaCfMVb8tMFxK5 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Five independent commits:
scan_v2, so V1 keeps develop's plans.hash_lookupused to turn the whole filter intotrue.key >= min AND key <= maxdecoded the key column once per comparison; the scan's between rewrite only ran on static filters. TPC-DS SF10, 10 join queries against commit 3: geomean 0.972, Q69 and Q10 0.93.true, Q69 was still 5.4ms slower than develop. Skipping that pass everywhere made Q82 and Q37 at SF10 3.4–3.7× slower, so it is skipped only for scans of at most one filtered chunk (128K rows). All 99 TPC-DS queries against commit 4: SF1 geomean 0.97; SF10 0.99 on best runs and 1.01 on medians, and the SF10 outliers were flat when rerun with 54 runs each.Each commit was checked alone with fmt, clippy and
vortex-datafusionandvortex-layouttests. Other candidates were measured and dropped: skipping dynamic zone pruning for every scan (Q82 and Q37 3.4–3.7× slower), reusing zone results that another partition of the file already proved (no change), and, with no benefit, a per-generation conversion cache, dropping IN lists (it slowed TPC-DS SF10 by 1.058×) and skipping advisory filters that keep at least 90% of rows.Local results
These tables cover commits 1–3; commits 4 and 5 were measured against the commit before each, as listed above.
Develop vs this branch, DataFusion,
--scan-v2 true,-C target-cpu=native. Each query ran in A-B-B-A order (fastest of the runs), with warm using 4 iterations and dropping the first.Best warm results: TPC-DS Q72 at SF10 18.9s → 5.3s, Q25 at SF20 364ms → 98ms, Q7 at SF20 621ms → 178ms.
The host was shared with another benchmark job, so per-query numbers are noisy. TPC-H SF20 Q16, Q9 and Q12 show about 2× in this run, but develop and this branch both have fast and slow rounds at that scale, and interleaved repeats of Q16 and Q9 were flat.
Known regression at SF1
CI runs TPC-H, TPC-DS, ClickBench and FineWeb on V2 through
bench-orchestrator. On commit 58a4d05 it reports DataFusion on Vortex as:On commit bb6955d, with commits 4 and 5, it reports TPC-DS SF1 +9.6%, TPC-H SF1 +5.4%, TPC-H SF10 +2.0%, ClickBench -4.0% and FineWeb +0.8%, so the local SF1 recovery from commits 4 and 5 does not show in CI.
CI's verdict for each is "No clear signal". The TPC-DS SF1 slowdown reproduces locally (Q10 and Q69 about 1.3×, most others about 1.1×, 1–8ms each). TPC-DS Q72 and Q18 still improve in CI. It comes from the pushdown commit. Most of it is the zone pruning pass a dynamic filter adds before each scan: Q69 runs about 114 scans, each waiting around 0.5ms for that pass before its first chunk. Commits 4 and 5 recover part of it: locally, SF1 Q69 went from 30.4ms to about 28.9ms, against 23.4ms on develop. V1 is flat against develop on 8 TPC-DS SF10 queries.
The trade-off is a remaining cost on very small TPC-DS scans for a gain of 20–25% at SF10 and SF20.
Checks
-D warnings, andcargo nextest run -p vortex-datafusion -p vortex-layout(commit 4:vortex-datafusiononly). The new tests in commits 4 and 5 were checked to fail without their change.-p vortex-sqllogictest. Commit 5: 1529 tests passed invortex-layoutandvortex-datafusion; SQL logic tests were not rerun locally. The skipped ones run in CI, which is green.vortex-arraytests, doctests. No check compares query results between binaries locally; the bench runs here only checked exit status.🤖 Generated with Claude Code
https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM