Repository navigation
Default DuckDB to Scan V2 and optimize sparse and wide scans - #10964
joseph-isaacs wants to merge 28 commits into
Conversation
e503266 to
f0752c6
Compare
Merging this PR will degrade performance by 5.77%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
47bbb02 to
e1d38d8
Compare
e1d38d8 to
d3f067e
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (63)
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.886x ➖, 3↑ 1↓)
datafusion / parquet / ns (1.025x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.959x ➖, 1↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 2↑ 4↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.942x ➖, 6↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.017x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.009x ➖, 1↑ 2↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.986x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.948x ➖, 4↑ 1↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.063x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.011x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.077x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.983x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.977x ➖, 3↑ 1↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
|
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Benchmark commit_id records the runtime checkout. Verify the source revision and executable SHA-256 from each build artifact independently, and preserve the runtime checkout in the comparison manifest. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Give each admitted planning root its own IO session so retiring one root preserves pending reads of other roots. Bound default per-partition chunk streams by the workers left per engine partition, while retaining explicit per-thread overrides. Exercise delayed payload reads, startup ranges and sparse selections with inline execution. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Retain failures and incomplete coverage explicitly while comparing the remaining complete query pairs. Bound individual query processes, skip failed queries on later rounds, and use dedicated NVMe paths for input and spill files. Exercise process failures and timeouts through real subprocesses. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Drive the partition through one owned task when engine partitions already fill the workers. Defer that task until startup work has drained, and retain independent IO sessions for admitted roots. Avoid the repeated CPU-task handoffs of the previous inline path. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Redirect the normal data and target directories to local NVMe instead of changing the workspace root and supplying a local URL override. ClickBench generation uses its default cache path; both generators and benchmark binaries now share that path while downloads and spill files remain on NVMe. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Keep filter and IO work in large chunks while returning stored scan pieces directly for conversion. Avoid gathering filtered scan results into a chunked array and copying across stored boundaries. Keep bounded gathering for filtered random-access work. Verify filtered and unfiltered stored boundaries and array equality across three layout strategies. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Retain the previous per-worker chunk overlap after measurements showed that limiting each partition to one stream regressed Q6, Q15 and ClickBench. Keep independent IO sessions and the single-task path for explicit inline execution. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Avoid constructing a hash map of every file segment before a narrow scan can read. Ordered locations share their existing storage and use binary search; unordered tables retain indexed lookup and duplicate ranges keep their last segment ID. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
955e389 to
a114a2a
Compare
Slice oversized encoded output pieces before Arrow conversion, retaining the scan work and read window while bounding decoded values and downstream casts. Cover filtered and unfiltered inline and overlapping scans with exact output verification. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Allow 900 seconds for a seven-iteration SF50 process, retaining the 300-second limit for other cases and the 180-minute job budget. Continue to report failed or timed-out queries as incomplete. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Reuse source-verified native artifacts on Graviton4 for serial V1/V2 TPC-H Q13 and ClickBench Q19 metrics and Samply profiles. Resolve names on the runner after checking GNU build IDs, retaining every sample and stack table. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Retain stored boundaries for wide and variable-width conversions while bounding projected rows up to 16 bytes to 65536 rows. This keeps the narrow aggregate working-set benefit without the measured wide TPC-H Q1 regression. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
For dense scans with one static predicate, project each completed piece within the existing I/O session instead of waiting for the full filter window. Keep the combined-mask path for multiple predicates, dynamic bounds and sparse demand. A gated-read regression fails before the change and verifies early output plus the complete result after it. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
The guarded seven-query comparison was inconclusive overall and Q13 was 1.7% slower than the preserved V2 baseline. Keep the existing combined-mask handoff; early-output correctness alone does not justify this performance experiment. The ClickBench attempt never started because the shared host remained busy. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Align narrow unfiltered byte partitions to 65536-row boundaries before preparing scan plans. This reduces repeated setup on small dimensions while preserving exact row coverage. Keep filtered and wide partitions on the existing 8192-row grid. Extend cold/warm repartition coverage to 30000 and 150000 rows. Use four reversed rounds and eleven iterations outside SF50 to resolve the sub-percent differences seen in the previous Actions comparison. SF50 retains two rounds of seven iterations and its 900-second process limit. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
V2 benefits from amortized planning and reads, but oversized materialization and duplicated setup can delay consumers. This PR sizes narrow and wide work separately, preserves stored projection pieces, and overlaps DuckDB reads across two active work items per thread.
DuckDB extension scans default to V2;
VORTEX_SCAN_V2=0restores V1. Benchmark binaries retain--scan-v1 falsefor V2 and--scan-v1 truefor V1.Sparse payload retrieval preserves exact file/row pairs and gathers referenced dictionary entries before conversion. Dynamic bounds refresh before projection, dictionaries retain chunk boundaries, and scans preserve requested column order. The optional startup budget remains disabled by default. These paths use existing filters and compute operations.
Actions evidence
Run 41673564950 is complete for head
ac26753966cfcc66c758ed32d2f0e336b1386f67and pinned develop7146a1b31783b6609c5de7f08652fb5416ac3fbd. Four cases completed successfully; TPC-DS SF50 is incomplete because baseline DuckDB Q64 exhausted 150.6 GiB.Ratios are fixed-workload geometric means; below one means faster V2. The within-head comparison holds compute code fixed. The develop baseline includes newer decimal-compare and zip kernels, so the cross-revision comparison also includes compute changes. DataFusion TPC-H SF10 and TPC-DS SF50 remain inconclusive against V1; ClickBench regresses. The full speed objective is not established.
Each case ran sequential base V2, head V2 and head V1 on identical head-prepared inline-metadata files, on a dedicated Graviton 4 host with 94 pinned workers. Non-SF50 cases used four reversed-order rounds, eleven iterations per process and two excluded warmups. SF50 used two rounds of seven iterations and the same warmup exclusion. SF50 processes allow 900 seconds; other cases allow 300 seconds. Source revisions and executable hashes are verified independently of checkout metadata. Artifacts include raw samples, manifests and failed-query logs.
Mental model and remaining limits
Larger scan windows amortize conjunct setup, metadata planning and reads. Smaller Arrow materializations improve narrow decode/cast working sets without reducing the read window. Applying the row bound to wide output had regressed local TPC-H Q1 by 7.5%; restricting it to narrow values retained the approximately 5% ClickBench Q29 benefit and reduced that Q1 difference.
Earlier dedicated profiles, from the verified 6c3 binaries, show less CPU in V2 for TPC-H Q13 and ClickBench Q19 despite worse elapsed time. Q13’s small unfiltered customer scan produced 100 batches versus V1’s 23. This motivates studying setup, scheduling and handoff latency; it does not establish a missing compute kernel.
Develop’s DuckDB cardinality correction resolved the earlier SF10 Q64 OOM; all 99 SF10 queries now complete. SF50 Q64 fails in the current pinned develop baseline before either head variant is measured. Its 98-query DuckDB results cannot describe the full suite.
Bounded local SF50 Q64 checks reach 12 GiB with V1, V2 and Parquet; V2 also reaches that bound with one thread. These controlled stops are not engine OOMs and do not prove what a larger-memory run would do. A single-thread V2 profile, stopped at 8 GiB and symbolicated against exact Rust and DuckDB ELF build IDs, attributes 93.5% of sampled CPU to DuckDB’s hash-aggregate sink and about 0.08% to Vortex’s scan adaptor during the memory-growth phase. Scan read-buffer tuning alone cannot release the engine’s retained group state.
Validation
The measured checkpoint passes all 836 DataFusion library tests, workspace all-target/all-feature Clippy with warnings denied, pinned formatting, strict workflow YAML lint and shell parsing. Sixteen cold/warm-cache repartition cases cover 30K/150K rows and three/eight requested partitions with exact full-file output. Earlier unchanged layout, scan and DuckDB code passed their affected library tests, doctests, and changed C++ formatting checks. The comparison runner’s 13 tests and Python lint/type checks passed.
Additional early-filter handoff, partial-partition work-budget and byte-aware Arrow trials were rejected when paired comparisons did not establish the targeted gains. Temporary benchmark switches are removed from the final source; their samples and signed commits are retained locally for audit.
Keep this PR as a draft. Ordinary CI skips draft PRs, and skipped checks are not passes. The full performance objective remains unmet.