Repository navigation
Share scan filtering and planning work; reuse file handles and rebalance tails - #10844
joseph-isaacs wants to merge 1 commit into
Conversation
Merging this PR will improve performance by 17.72%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing |
Polar Signals Profiling ResultsLatest Run
Previous Runs (47)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.962x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.887x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.967x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 2↑ 2↓)
duckdb / parquet / ns (0.934x ➖, 1↑ 0↓)
|
Reconcile with merged #10799 and leave overlapping scheduling and read ownership policies to #10789, #10857, and #10969. Preserve current develop's structural cache, sparse dictionary threshold, chunk sizing, short-edge merging, retained segments, and benchmark defaults. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
6cea8d0 to
2ad2f11
Compare
…nce tails Ported from #10844 into the combined V2 scan perf branch. Kept the branch's lazy struct-root expansion and field-path fast path in ExpressionPackRule (adding project_fields, the single-field fast path and PartitionCache after them), the branch's ActiveConjunct filter planner with its dense FilterPlan threshold (dropping 10844's dense_whole/0.2-density path; only infallible cross-column ORs run over the whole chunk range in Scan mode), the 64 KiB local coalescing config next to the cached file handle, and the startup initial_chunks alongside tail rebalancing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBQHsaCfMVb8tMFxK5 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
V2 can repeat filtering when packed fields split lazy arrays at different boundaries, repeat expression-partition planning, reopen the same immutable local file, and emit small tails that make downstream joins copy batches. This PR shares that work and rebalances unfiltered DataFusion output tails.
Reconciled onto
develop1c2f6eb353after #10799 merged. The diff is now 17 files (previously 25).Remaining changes
Reconciliation with other PRs
These open PRs are not dependencies and were not merged into this branch. Some touch the same files; that alone does not mean their changes are duplicates. They still require integration review when they land.
The completed-dynamic-filter/footer-pruning experiments, including the latest nearly-dense IN-list guard, are not included. They are preserved locally on
ji/codex/scan-v2-local-experiments-20261029atb688a72c66. The original published implementation is preserved locally onji/codex/scan-v2-published-before-reconcile-20261029at6cea8d0792.Validation and performance
cargo +nightly-2026-09-10 fmt -p vortex-datafusion -p vortex-layout -p vortex-io.git diff --check origin/develop...HEAD: passed.1c2f6eb353: conflict-free; the new commit directly parents that develop commit.2ad2f11d58: all 30 jobs passed, including Rust tests, SQL logic tests, Clippy, formatting, and supported-platform builds.Previous Q28 improvements and the earlier CI comparison describe the old combined implementation. They do not validate this reduced diff. No current all-query parity or 20%-per-query improvement claim is made.