Repository navigation
Fix and speed up the V2 scan's planner chain - #10789
joseph-isaacs wants to merge 1 commit into
Conversation
Polar Signals Profiling ResultsLatest Run
Previous Runs (64)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.017x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.428x ❌, 1↑ 7↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 3↑ 3↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.985x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 9↑ 5↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.039x ➖, 0↑ 9↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.025x ➖, 1↑ 4↓)
datafusion / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.012x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.119x ➖, 1↑ 7↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.919x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.038x ➖, 0↑ 1↓)
|
Merging this PR will improve performance by 18.62%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing |
143c86e to
0f43247
Compare
0f43247 to
b1e639a
Compare
commented
Nov 15, 2026
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
Rebased onto develop as one commit. Conflicts with #10799 resolved: - The scan cache keeps develop's structural ExprKey and retained segments, with this branch's build-once OnceCell entries. - A filtered chunk is sized by concurrency up to develop's per-shape cap (128Ki rows, or 1Mi for one fixed-width field), never below 64Ki. Rebased again onto develop. Conflicts resolved: - The filter planner keeps develop's dynamic-conjunct skipping, with the conjunct's version held beside its MaskScan. - Filter plans built once per filter still honour develop's skip_unselective_dynamic: a scan whose setting differs from the cached plans plans its own. - Roots lower through Layout::new_plan, and is_identity_expression keeps develop's bool return. - v2_inline_chunks_read_through_their_own_sessions now sets scan_concurrency 1: 0 means unset here, and 1 is what runs inline. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEEc3okQ7JZh6AZDur Claude-Session: https://claude.ai/code/session_01UeniB9i8ce2ie4BaCQkHSc
b1e639a to
a05e277
Compare
commented
Nov 16, 2026
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 2↑ 6↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Summary
Fixes and perf work on the planner-driven V2 scan, found by dumping and timing real V2 scan plans for every TPC-H SF1 and SF10 query. The diff against
developis only this PR's changes.What the plans showed:
WHEREpredicate. A proof costs about 1 ms per partition. Join-time dynamic filters are not pushed into Vortex at all (Support datafusion dynamic filter expressions #4034), which is a separate pruning opportunity (for example Q18 over thel_orderkey-sortedlineitem).Changes
concurrency * cores <= 1, so each partition still ran a core's worth of chunks, about cores² chunks in flight overall.develop; this PR extends it to spawned chunks' own runs.)dense_chunk_yields_a_batch_per_window.OnceCells. The worst case drops from 14 ms to about 1 ms.MaskScan. A mask that is a single piece is no longer copied.Evals left after pushing a projection into aPack.is_zone_proof's boolean dtype before walking the expression.V2_MIN_ROWS_PER_PARTITION). An unfiltered scan keeps the engine's partitions, because every row it reads flows into the operators above it. Cappingcustomerandpartat SF10 made Q10 and Q14 13–17% slower; forcing them to one partition reproduces this locally. Test:v2_scan_keeps_partitions_unless_filtered.Benchmarks (CI, 96 cores,
6a4c019vsdevelop)DataFusion and DuckDB both run TPC-H on V2. The other suites run V1 and are neutral as expected.
DuckDB on Vortex is flat on TPC-H at both scales.
Checks run locally:
cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warningscargo fmt --checkforvortex-layoutandvortex-datafusioncargo test -p vortex-layout --lib: 675 passed, also with the default concurrency forced to 96cargo test -p vortex-datafusion --lib: 774 passedRebase onto
develop(2026-11-16)Rebased again: the branch was 295 commits behind and conflicted. Resolved:
FilterPlannerkeeps develop's dynamic-conjunct skipping; the conjunct's version sits beside itsMaskScan.OnceCell), and still honour develop'sskip_unselective_dynamic: a scan whose setting differs from the cached plans plans its own.Layout::new_plan;is_identity_expressionkeeps develop'sboolreturn (nowpub(super)forEvalIdentityRule).v2_inline_chunks_read_through_their_own_sessions(from develop) now setsscan_concurrency: Some(1). On this branch 0 means unset, and 1 is what runs chunks inline.Overlaps with #11448: both edit the
in_flight/inlinelines of the DataFusionScanDriver::run. Whichever merges second should keepin_flight = concurrencyfrom here and&& !orderedfrom there.Checks after the rebase:
cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warnings: cleancargo +nightly-2026-11-13 fmt -p vortex-layout -p vortex-datafusion -p vortex-file: no changescargo nextest run -p vortex-layout -p vortex-datafusion -p vortex-file: 2008 passedNot rerun: the benchmarks above predate the rebase.
🤖 Generated with Claude Code
https://claude.ai/code/session_01GwhHqFVpQqPgQQkkK2y9kX