Skip to content

Share scan filtering and planning work; reuse file handles and rebalance tails - #10844

Draft
joseph-isaacs wants to merge 1 commit into
developfrom
ji/codex/optimize-scan-v2-10799
Draft

joseph-isaacs wants to merge 1 commit into
developfrom
ji/codex/optimize-scan-v2-10799

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

V2 can repeat filtering when packed fields split lazy arrays at different boundaries, repeat expression-partition planning, reopen the same immutable local file, and emit small tails that make downstream joins copy batches. This PR shares that work and rebalances unfiltered DataFusion output tails.

Reconciled onto develop 1c2f6eb353 after #10799 merged. The diff is now 17 files (previously 25).

Remaining changes

  • Share lazy filtered arrays, dictionary codes, and extension components before packed fields split them.
  • Lower direct field projections without general expression partitioning, fast-path single-field expressions, and cache pure partition rewrites with a 64-entry/256-node bound.
  • Evaluate eligible infallible dense/OR predicates in their original row domain, avoiding boolean gather/scatter.
  • Share scan state across concurrent DataFusion sources using object-store identity and file-version metadata. Registries retain weak references; prepared scans own file caches. Dictionary values remain file-scoped.
  • Skip empty V2 partitions before adapting expressions and exclude cached whole values when discovering prefetch reads.
  • Reuse immutable local file handles for a reader and its in-flight reads; object-store streams retain their GET behavior. No mmap.
  • Rebalance unfiltered, unlimited DataFusion output tails while preserving row order, nulls and full slices. Pending rows between inputs remain below three target batches; this is not a backing-buffer byte bound.

Reconciliation with other PRs

PR Disposition
Merged #10799 Keep develop's V2 defaults, structural expression keys, retained-segment implementation, sparse dictionary threshold, chunk sizing, and short-edge merging. No competing changes to those implementations remain here.
Open #10789 Drop this branch's chunk/output sizing and projection-piece/prefetch scheduling changes. Leave inline execution, root I/O sessions, per-key plan construction, and window streaming to that PR. Our bounded expression-partition cache and packed-field sharing are different layers.
Open #10857 Do not import or duplicate row-claim-based read ownership, DuckDB partitioning/output policy, or dictionary-value lowering changes. This PR only changes DataFusion file-version registry keys and generic immutable-reader handle reuse; its file-lifetime regression coverage remains.
Open #10969 Keep develop's task sizing unchanged here; leave sparse-filter subdivision and unfiltered 256K tasks to #10969.

These open PRs are not dependencies and were not merged into this branch. Some touch the same files; that alone does not mean their changes are duplicates. They still require integration review when they land.

The completed-dynamic-filter/footer-pruning experiments, including the latest nearly-dense IN-list guard, are not included. They are preserved locally on ji/codex/scan-v2-local-experiments-20261029 at b688a72c66. The original published implementation is preserved locally on ji/codex/scan-v2-published-before-reconcile-20261029 at 6cea8d0792.

Validation and performance

  • Scoped formatting: cargo +nightly-2026-09-10 fmt -p vortex-datafusion -p vortex-layout -p vortex-io.
  • git diff --check origin/develop...HEAD: passed.
  • Local merge-tree against 1c2f6eb353: conflict-free; the new commit directly parents that develop commit.
  • Existing tests for packed filtering, cross-column OR/dense/sparse selection, file-cache lifetime, immutable file handles, and eight batch-rebalance cases remain in the patch.
  • Linters and Tests on reconciled head 2ad2f11d58: all 30 jobs passed, including Rust tests, SQL logic tests, Clippy, formatting, and supported-platform builds.
  • SQL benchmark run: binary build passed. ClickBench and TPC-DS are queued awaiting runners; performance validation remains incomplete.

Previous Q28 improvements and the earlier CI comparison describe the old combined implementation. They do not validate this reduced diff. No current all-query parity or 20%-per-query improvement claim is made.

@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 26, 2026 — with ChatGPT Codex Connector
@joseph-isaacs joseph-isaacs changed the title Reduce V2 scan predicate planning and sparse OR filtering work Reduce V2 scan filtering, planning, and repeated DataFusion file reads Oct 26, 2026
@joseph-isaacs joseph-isaacs removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026
@codspeed

codspeed Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 17.72%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 2 improved benchmarks
✅ 2302 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation missing_stats[16] 84.6 µs 67.2 µs +25.95%
⚡ Walltime threshold_avx2[80] 3 µs 2.7 µs +10.03%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/codex/optimize-scan-v2-10799 (2ad2f11) with develop (1c2f6eb)

Open in CodSpeed

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 2ad2f11 tpch-nvme-10 1 Explore Profiling Data
🟢 Done 2ad2f11 tpch-s3 1 Explore Profiling Data
🟢 Done 2ad2f11 fineweb 1 Explore Profiling Data
🟢 Done 2ad2f11 polarsignals 1 Explore Profiling Data
Previous Runs (47)
Status Commit Job Attempt Link
🟢 Done 6cea8d0 polarsignals 1 Explore Profiling Data
🟢 Done 6cea8d0 fineweb 1 Explore Profiling Data
🟢 Done 6cea8d0 tpch-nvme-10 1 Explore Profiling Data
🟢 Done 6cea8d0 tpcds-nvme 1 Explore Profiling Data
🟢 Done 6cea8d0 tpch-s3 1 Explore Profiling Data
🟢 Done 6cea8d0 tpch-nvme 1 Explore Profiling Data
🟢 Done c035c3f fineweb 1 Explore Profiling Data
🟢 Done c035c3f tpch-nvme-10 1 Explore Profiling Data
🟢 Done c035c3f tpcds-nvme 1 Explore Profiling Data
🟢 Done c035c3f tpch-s3 1 Explore Profiling Data
🟢 Done c035c3f polarsignals 1 Explore Profiling Data
🟢 Done c035c3f tpch-nvme 1 Explore Profiling Data
🟢 Done d88dd98 clickbench-nvme 1 Explore Profiling Data
🟢 Done d88dd98 tpch-nvme-10 1 Explore Profiling Data
🟢 Done d88dd98 polarsignals 1 Explore Profiling Data
🟢 Done d88dd98 fineweb 1 Explore Profiling Data
🟢 Done d88dd98 tpch-s3 1 Explore Profiling Data
🟢 Done d88dd98 tpch-nvme 1 Explore Profiling Data
🟢 Done d88dd98 tpcds-nvme 1 Explore Profiling Data
🟢 Done aa534ec clickbench-nvme 1 Explore Profiling Data
🟢 Done aa534ec polarsignals 1 Explore Profiling Data
🟢 Done aa534ec tpch-nvme-10 1 Explore Profiling Data
🟢 Done aa534ec fineweb 1 Explore Profiling Data
🟢 Done aa534ec tpcds-nvme 1 Explore Profiling Data
🟢 Done aa534ec tpch-nvme 1 Explore Profiling Data
🟢 Done aa534ec tpch-s3 1 Explore Profiling Data
🟢 Done b0caf5a clickbench-nvme 1 Explore Profiling Data
🟢 Done b0caf5a tpch-nvme-10 1 Explore Profiling Data
🟢 Done b0caf5a polarsignals 1 Explore Profiling Data
🟢 Done b0caf5a fineweb 1 Explore Profiling Data
🟢 Done b0caf5a tpch-s3 1 Explore Profiling Data
🟢 Done b0caf5a tpcds-nvme 1 Explore Profiling Data
🟢 Done b0caf5a tpch-nvme 1 Explore Profiling Data
🟢 Done 83eadcb clickbench-nvme 1 Explore Profiling Data
🟢 Done 83eadcb polarsignals 1 Explore Profiling Data
🟢 Done 83eadcb tpch-nvme-10 1 Explore Profiling Data
🟢 Done 83eadcb tpch-nvme 1 Explore Profiling Data
🟢 Done 83eadcb fineweb 1 Explore Profiling Data
🟢 Done 83eadcb tpcds-nvme 1 Explore Profiling Data
🟢 Done 83eadcb tpch-s3 1 Explore Profiling Data
🟢 Done 388c0cd clickbench-nvme 1 Explore Profiling Data
🟢 Done 388c0cd polarsignals 1 Explore Profiling Data
🟢 Done 388c0cd tpch-nvme-10 1 Explore Profiling Data
🟢 Done 388c0cd fineweb 1 Explore Profiling Data
🟢 Done 388c0cd tpch-s3 1 Explore Profiling Data
🟢 Done 388c0cd tpch-nvme 1 Explore Profiling Data
🟢 Done 388c0cd tpcds-nvme 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: PolarSignals Profiling 📖

Commits: PR 2ad2f11d vs base e4dce8b7
Vortex (geomean): hot 0.983x ➖ · cold 1.035x ➖


datafusion / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
polarsignals_q00/datafusion:vortex-file-compressed 44485342 / 44542838 / -0.1% 91550933 / 89892344 / +1.8% 0.49 / 0.50 / -1.9%
polarsignals_q01/datafusion:vortex-file-compressed 49688434 / 50214840 / -1.0% 113953736 / 113889124 / +0.1% 0.44 / 0.44 / -1.1%
polarsignals_q02/datafusion:vortex-file-compressed 43233239 / 43719912 / -1.1% 88642500 / 86090170 / +3.0% 0.49 / 0.51 / -4.0%
polarsignals_q03/datafusion:vortex-file-compressed 53338800 / 53425202 / -0.2% 116007475 / 115285323 / +0.6% 0.46 / 0.46 / -0.8%
polarsignals_q04/datafusion:vortex-file-compressed 8772760 / 9302224 / -5.7% 46617909 / 44753886 / +4.2% 0.19 / 0.21 / -9.5%
polarsignals_q05/datafusion:vortex-file-compressed 11538836 / 11759190 / -1.9% 51063712 / 49264118 / +3.7% 0.23 / 0.24 / -5.3%
polarsignals_q06/datafusion:vortex-file-compressed 23316620 / 23731105 / -1.7% 60291767 / 59138928 / +1.9% 0.39 / 0.40 / -3.6%
polarsignals_q07/datafusion:vortex-file-compressed 10548650 / 10927024 / -3.5% 50137233 / 45867334 / +9.3% 0.21 / 0.24 / -11.7%
polarsignals_q08/datafusion:vortex-file-compressed 97900288 / 97959968 / -0.1% 163472439 / 157519393 / +3.8% 0.60 / 0.62 / -3.7%
polarsignals_q09/datafusion:vortex-file-compressed 9961946 / 10124616 / -1.6% 45843435 / 42928250 / +6.8% 0.22 / 0.24 / -7.9%

No file size changes detected.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: FineWeb NVMe 📖

Commits: PR 2ad2f11d vs base 2a9c7628
Verdict: No clear signal (low confidence)
Attributed Vortex impact: -2.3%
Engines: DataFusion No clear signal (-3.6%, low confidence) · DuckDB No clear signal (-1.0%, low confidence)
Vortex (geomean): hot 0.978x ➖ · cold 1.064x ➖
Parquet (geomean): hot 1.001x ➖ · cold 1.007x ➖
Shifts: Parquet (control) +0.1% · Median polish -0.3%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
  • Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
  • Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

datafusion / vortex-file-compressed / ns (0.962x ➖, 1↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
fineweb_q00/datafusion:vortex-file-compressed 6915861 / 6903610 / +0.2% 18047649 / 17910056 / +0.8% 0.38 / 0.39 / -0.6%
fineweb_q01/datafusion:vortex-file-compressed 8313357 / 8689116 / -4.3% 44086779 / 44097276 / -0.0% 0.19 / 0.20 / -4.3%
fineweb_q02/datafusion:vortex-file-compressed 9682328 / 10214480 / -5.2% 48388356 / 51600306 / -6.2% 0.20 / 0.20 / +1.1%
fineweb_q03/datafusion:vortex-file-compressed 20500835 / 19591272 / +4.6% 109392207 / 107891043 / +1.4% 0.19 / 0.18 / +3.2%
fineweb_q04/datafusion:vortex-file-compressed 67585988 / 66685910 / +1.3% 163031930 / 160241075 / +1.7% 0.41 / 0.42 / -0.4%
fineweb_q05/datafusion:vortex-file-compressed 66046842 / 67421159 / -2.0% 153515991 / 157717787 / -2.7% 0.43 / 0.43 / +0.6%
fineweb_q06/datafusion:vortex-file-compressed 18719087 / 18932958 / -1.1% 118036305 / 114653888 / +3.0% 0.16 / 0.17 / -4.0%
fineweb_q07/datafusion:vortex-file-compressed 18209688 / 18335188 / -0.7% 107213893 / 105325486 / +1.8% 0.17 / 0.17 / -2.4%
fineweb_q08/datafusion:vortex-file-compressed 7741012 / 10204749 / -24.1% 🟢 64024978 / 50634223 / +26.4% 🔴 0.12 / 0.20 / -40.0%
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
fineweb_q00/datafusion:parquet 9226032 / 9482157 / -2.7% 33863757 / 32111399 / +5.5% 0.27 / 0.30 / -7.7%
fineweb_q01/datafusion:parquet 74249062 / 75129645 / -1.2% 155687411 / 156360166 / -0.4% 0.48 / 0.48 / -0.7%
fineweb_q02/datafusion:parquet 74553621 / 73747780 / +1.1% 156261270 / 159846521 / -2.2% 0.48 / 0.46 / +3.4%
fineweb_q03/datafusion:parquet 73041936 / 74768342 / -2.3% 159524820 / 157975411 / +1.0% 0.46 / 0.47 / -3.3%
fineweb_q04/datafusion:parquet 84403838 / 82819587 / +1.9% 153799695 / 157992861 / -2.7% 0.55 / 0.52 / +4.7%
fineweb_q05/datafusion:parquet 82203567 / 81080730 / +1.4% 153946425 / 157515065 / -2.3% 0.53 / 0.51 / +3.7%
fineweb_q06/datafusion:parquet 75613166 / 76820271 / -1.6% 157816254 / 167173020 / -5.6% 0.48 / 0.46 / +4.3%
fineweb_q07/datafusion:parquet 75535474 / 73624110 / +2.6% 152978085 / 156859062 / -2.5% 0.49 / 0.47 / +5.2%
fineweb_q08/datafusion:parquet 72837134 / 73579522 / -1.0% 155538123 / 161416701 / -3.6% 0.47 / 0.46 / +2.7%
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
fineweb_q00/duckdb:vortex-file-compressed 2064522 / 2060458 / +0.2% 7399532 / 7235617 / +2.3% 0.28 / 0.28 / -2.0%
fineweb_q01/duckdb:vortex-file-compressed 9304948 / 9533635 / -2.4% 34552908 / 33428263 / +3.4% 0.27 / 0.29 / -5.6%
fineweb_q02/duckdb:vortex-file-compressed 10826404 / 10827112 / -0.0% 38650797 / 33795730 / +14.4% 🔴 0.28 / 0.32 / -12.6%
fineweb_q03/duckdb:vortex-file-compressed 20174514 / 19616776 / +2.8% 104224296 / 88574457 / +17.7% 🔴 0.19 / 0.22 / -12.6%
fineweb_q04/duckdb:vortex-file-compressed 66295825 / 66064766 / +0.3% 119973865 / 108051698 / +11.0% 🔴 0.55 / 0.61 / -9.6%
fineweb_q05/duckdb:vortex-file-compressed 67243462 / 66540793 / +1.1% 118816594 / 105711829 / +12.4% 🔴 0.57 / 0.63 / -10.1%
fineweb_q06/duckdb:vortex-file-compressed 18858904 / 19522226 / -3.4% 107708521 / 94334392 / +14.2% 🔴 0.18 / 0.21 / -15.4%
fineweb_q07/duckdb:vortex-file-compressed 18754803 / 18991482 / -1.2% 106043161 / 88019344 / +20.5% 🔴 0.18 / 0.22 / -18.0%
fineweb_q08/duckdb:vortex-file-compressed 5384718 / 5492126 / -2.0% 14558735 / 14753686 / -1.3% 0.37 / 0.37 / -0.6%
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
fineweb_q00/duckdb:parquet 23155232 / 23448581 / -1.3% 26960576 / 27593808 / -2.3% 0.86 / 0.85 / +1.1%
fineweb_q01/duckdb:parquet 54825947 / 54933198 / -0.2% 81648982 / 76123605 / +7.3% 0.67 / 0.72 / -6.9%
fineweb_q02/duckdb:parquet 60577002 / 58051781 / +4.3% 82030208 / 79202689 / +3.6% 0.74 / 0.73 / +0.8%
fineweb_q03/duckdb:parquet 118783701 / 118155770 / +0.5% 189447619 / 179057039 / +5.8% 0.63 / 0.66 / -5.0%
fineweb_q04/duckdb:parquet 163020408 / 161386539 / +1.0% 229096564 / 216551920 / +5.8% 0.71 / 0.75 / -4.5%
fineweb_q05/duckdb:parquet 159342578 / 157119474 / +1.4% 210826797 / 208784938 / +1.0% 0.76 / 0.75 / +0.4%
fineweb_q06/duckdb:parquet 88726828 / 89287572 / -0.6% 153138904 / 154201019 / -0.7% 0.58 / 0.58 / +0.1%
fineweb_q07/duckdb:parquet 91056010 / 91067884 / -0.0% 152336231 / 145204462 / +4.9% 0.60 / 0.63 / -4.7%
fineweb_q08/duckdb:parquet 28905059 / 29225694 / -1.1% 30703944 / 30385447 / +1.0% 0.94 / 0.96 / -2.1%

No file size changes detected.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: TPC-H SF=10 on NVME 📖

Commits: PR 2ad2f11d vs base e4dce8b7
Verdict: No clear signal (low confidence)
Attributed Vortex impact: -1.4%
Engines: DataFusion No clear signal (-1.4%, low confidence) · DuckDB No clear signal (-1.4%, low confidence)
Vortex (geomean): hot 0.987x ➖ · cold 0.954x ➖
Parquet (geomean): hot 1.001x ➖ · cold 0.999x ➖
Shifts: Parquet (control) +0.1% · Median polish -0.3%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
  • Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
  • Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

datafusion / vortex-file-compressed / ns (0.987x ➖, 1↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/datafusion:vortex-file-compressed 88874427 / 89830011 / -1.1% 147724472 / 151818144 / -2.7% 0.60 / 0.59 / +1.7%
tpch_q02/datafusion:vortex-file-compressed 63950073 / 64854619 / -1.4% 96429640 / 96510938 / -0.1% 0.66 / 0.67 / -1.3%
tpch_q03/datafusion:vortex-file-compressed 62227408 / 63550931 / -2.1% 117493646 / 119844418 / -2.0% 0.53 / 0.53 / -0.1%
tpch_q04/datafusion:vortex-file-compressed 33715651 / 34118013 / -1.2% 67480141 / 74172009 / -9.0% 0.50 / 0.46 / +8.6%
tpch_q05/datafusion:vortex-file-compressed 117901412 / 119303737 / -1.2% 211340950 / 176595992 / +19.7% 🔴 0.56 / 0.68 / -17.4%
tpch_q06/datafusion:vortex-file-compressed 10284804 / 10214165 / +0.7% 56524194 / 58267176 / -3.0% 0.18 / 0.18 / +3.8%
tpch_q07/datafusion:vortex-file-compressed 133865093 / 134451358 / -0.4% 188642393 / 204294147 / -7.7% 0.71 / 0.66 / +7.8%
tpch_q08/datafusion:vortex-file-compressed 142783415 / 141418962 / +1.0% 200988357 / 218182653 / -7.9% 0.71 / 0.65 / +9.6%
tpch_q09/datafusion:vortex-file-compressed 184348405 / 186296815 / -1.0% 262137363 / 344793657 / -24.0% 🟢 0.70 / 0.54 / +30.2%
tpch_q10/datafusion:vortex-file-compressed 69490452 / 70616992 / -1.6% 131763190 / 139484560 / -5.5% 0.53 / 0.51 / +4.2%
tpch_q11/datafusion:vortex-file-compressed 43760266 / 43151620 / +1.4% 64307216 / 68265687 / -5.8% 0.68 / 0.63 / +7.7%
tpch_q12/datafusion:vortex-file-compressed 34208686 / 34764636 / -1.6% 76542198 / 79411471 / -3.6% 0.45 / 0.44 / +2.1%
tpch_q13/datafusion:vortex-file-compressed 45750857 / 47706128 / -4.1% 80986789 / 81386369 / -0.5% 0.56 / 0.59 / -3.6%
tpch_q14/datafusion:vortex-file-compressed 21189406 / 20531467 / +3.2% 84415720 / 85266103 / -1.0% 0.25 / 0.24 / +4.2%
tpch_q15/datafusion:vortex-file-compressed 32333670 / 33441075 / -3.3% 87845685 / 92057550 / -4.6% 0.37 / 0.36 / +1.3%
tpch_q16/datafusion:vortex-file-compressed 36409441 / 37244823 / -2.2% 63976273 / 69393168 / -7.8% 0.57 / 0.54 / +6.0%
tpch_q17/datafusion:vortex-file-compressed 174625795 / 175378873 / -0.4% 223830677 / 239083332 / -6.4% 0.78 / 0.73 / +6.4%
tpch_q18/datafusion:vortex-file-compressed 205085561 / 206077691 / -0.5% 271477856 / 271336435 / +0.1% 0.76 / 0.76 / -0.5%
tpch_q19/datafusion:vortex-file-compressed 25396939 / 29236220 / -13.1% 🟢 83113969 / 84425029 / -1.6% 0.31 / 0.35 / -11.8%
tpch_q20/datafusion:vortex-file-compressed 59437442 / 59415400 / +0.0% 121357613 / 120037556 / +1.1% 0.49 / 0.49 / -1.1%
tpch_q21/datafusion:vortex-file-compressed 184463455 / 179428225 / +2.8% 240766920 / 245075636 / -1.8% 0.77 / 0.73 / +4.6%
tpch_q22/datafusion:vortex-file-compressed 25128401 / 25427504 / -1.2% 50886614 / 53010554 / -4.0% 0.49 / 0.48 / +2.9%
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/datafusion:parquet 87761806 / 87800478 / -0.0% 129599295 / 124663848 / +4.0% 0.68 / 0.70 / -3.9%
tpch_q02/datafusion:parquet 175867974 / 175634980 / +0.1% 199745613 / 199100564 / +0.3% 0.88 / 0.88 / -0.2%
tpch_q03/datafusion:parquet 138303893 / 137912852 / +0.3% 184156193 / 186509083 / -1.3% 0.75 / 0.74 / +1.6%
tpch_q04/datafusion:parquet 59031596 / 58892301 / +0.2% 94294691 / 90000676 / +4.8% 0.63 / 0.65 / -4.3%
tpch_q05/datafusion:parquet 170049212 / 169261575 / +0.5% 227713621 / 225899960 / +0.8% 0.75 / 0.75 / -0.3%
tpch_q06/datafusion:parquet 37232980 / 37055532 / +0.5% 78138503 / 76020246 / +2.8% 0.48 / 0.49 / -2.2%
tpch_q07/datafusion:parquet 217540473 / 218334890 / -0.4% 271772221 / 274775085 / -1.1% 0.80 / 0.79 / +0.7%
tpch_q08/datafusion:parquet 221407169 / 223397220 / -0.9% 296783755 / 293135311 / +1.2% 0.75 / 0.76 / -2.1%
tpch_q09/datafusion:parquet 315150683 / 313323388 / +0.6% 382803669 / 373153344 / +2.6% 0.82 / 0.84 / -2.0%
tpch_q10/datafusion:parquet 320996508 / 321738718 / -0.2% 381300225 / 384951266 / -0.9% 0.84 / 0.84 / +0.7%
tpch_q11/datafusion:parquet 126513479 / 126230808 / +0.2% 150263350 / 151825601 / -1.0% 0.84 / 0.83 / +1.3%
tpch_q12/datafusion:parquet 84417115 / 84520245 / -0.1% 124154923 / 119787021 / +3.6% 0.68 / 0.71 / -3.6%
tpch_q13/datafusion:parquet 217054258 / 216706113 / +0.2% 244959806 / 248737828 / -1.5% 0.89 / 0.87 / +1.7%
tpch_q14/datafusion:parquet 87633324 / 87320756 / +0.4% 194223523 / 189870970 / +2.3% 0.45 / 0.46 / -1.9%
tpch_q15/datafusion:parquet 105743238 / 105600133 / +0.1% 155163307 / 154090694 / +0.7% 0.68 / 0.69 / -0.6%
tpch_q16/datafusion:parquet 97927552 / 97578567 / +0.4% 121752621 / 123943111 / -1.8% 0.80 / 0.79 / +2.2%
tpch_q17/datafusion:parquet 214519968 / 213561188 / +0.4% 258229695 / 262174166 / -1.5% 0.83 / 0.81 / +2.0%
tpch_q18/datafusion:parquet 269129123 / 271257080 / -0.8% 313872624 / 309546314 / +1.4% 0.86 / 0.88 / -2.2%
tpch_q19/datafusion:parquet 109417902 / 108789195 / +0.6% 166470403 / 163981763 / +1.5% 0.66 / 0.66 / -0.9%
tpch_q20/datafusion:parquet 189168546 / 189538186 / -0.2% 241278229 / 255326074 / -5.5% 0.78 / 0.74 / +5.6%
tpch_q21/datafusion:parquet 247269536 / 247379946 / -0.0% 307726335 / 306670089 / +0.3% 0.80 / 0.81 / -0.4%
tpch_q22/datafusion:parquet 153493997 / 153645112 / -0.1% 173284672 / 178130561 / -2.7% 0.89 / 0.86 / +2.7%
duckdb / vortex-file-compressed / ns (0.987x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/duckdb:vortex-file-compressed 31673967 / 32077948 / -1.3% 58360375 / 70515169 / -17.2% 🟢 0.54 / 0.45 / +19.3%
tpch_q02/duckdb:vortex-file-compressed 31754450 / 31689639 / +0.2% 45706450 / 45999869 / -0.6% 0.69 / 0.69 / +0.8%
tpch_q03/duckdb:vortex-file-compressed 51508186 / 52252090 / -1.4% 92354912 / 104288678 / -11.4% 🟢 0.56 / 0.50 / +11.3%
tpch_q04/duckdb:vortex-file-compressed 50835252 / 51363015 / -1.0% 82114844 / 88809134 / -7.5% 0.62 / 0.58 / +7.0%
tpch_q05/duckdb:vortex-file-compressed 67911826 / 68452969 / -0.8% 102095662 / 109042430 / -6.4% 0.67 / 0.63 / +6.0%
tpch_q06/duckdb:vortex-file-compressed 14200283 / 14744661 / -3.7% 47909950 / 53147272 / -9.9% 0.30 / 0.28 / +6.8%
tpch_q07/duckdb:vortex-file-compressed 53647104 / 54042271 / -0.7% 102781467 / 107217288 / -4.1% 0.52 / 0.50 / +3.6%
tpch_q08/duckdb:vortex-file-compressed 87211348 / 87867832 / -0.7% 128768146 / 135645945 / -5.1% 0.68 / 0.65 / +4.6%
tpch_q09/duckdb:vortex-file-compressed 96802918 / 98164827 / -1.4% 150185502 / 155130197 / -3.2% 0.64 / 0.63 / +1.9%
tpch_q10/duckdb:vortex-file-compressed 87455693 / 87134754 / +0.4% 133454850 / 139118698 / -4.1% 0.66 / 0.63 / +4.6%
tpch_q11/duckdb:vortex-file-compressed 12809932 / 13390970 / -4.3% 26923062 / 26765247 / +0.6% 0.48 / 0.50 / -4.9%
tpch_q12/duckdb:vortex-file-compressed 42351228 / 43546451 / -2.7% 76273061 / 83700266 / -8.9% 0.56 / 0.52 / +6.7%
tpch_q13/duckdb:vortex-file-compressed 67185977 / 67540566 / -0.5% 86862191 / 89707214 / -3.2% 0.77 / 0.75 / +2.7%
tpch_q14/duckdb:vortex-file-compressed 27455181 / 28563596 / -3.9% 67555821 / 72628672 / -7.0% 0.41 / 0.39 / +3.3%
tpch_q15/duckdb:vortex-file-compressed 24429174 / 24418965 / +0.0% 58514345 / 63395202 / -7.7% 0.42 / 0.39 / +8.4%
tpch_q16/duckdb:vortex-file-compressed 49739683 / 49970484 / -0.5% 58536944 / 58954296 / -0.7% 0.85 / 0.85 / +0.2%
tpch_q17/duckdb:vortex-file-compressed 33776398 / 34691816 / -2.6% 61476202 / 64570526 / -4.8% 0.55 / 0.54 / +2.3%
tpch_q18/duckdb:vortex-file-compressed 100476647 / 102146185 / -1.6% 119034072 / 119685578 / -0.5% 0.84 / 0.85 / -1.1%
tpch_q19/duckdb:vortex-file-compressed 24256668 / 25282184 / -4.1% 69523101 / 74371680 / -6.5% 0.35 / 0.34 / +2.6%
tpch_q20/duckdb:vortex-file-compressed 38974320 / 38634831 / +0.9% 82719932 / 87016480 / -4.9% 0.47 / 0.44 / +6.1%
tpch_q21/duckdb:vortex-file-compressed 94407046 / 93192133 / +1.3% 140553462 / 146063804 / -3.8% 0.67 / 0.64 / +5.3%
tpch_q22/duckdb:vortex-file-compressed 34919550 / 35119478 / -0.6% 51968647 / 51565567 / +0.8% 0.67 / 0.68 / -1.3%
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/duckdb:parquet 68832358 / 68262903 / +0.8% 94144940 / 88536814 / +6.3% 0.73 / 0.77 / -5.2%
tpch_q02/duckdb:parquet 87486450 / 87656653 / -0.2% 95848858 / 99659690 / -3.8% 0.91 / 0.88 / +3.8%
tpch_q03/duckdb:parquet 110148549 / 111506958 / -1.2% 142479444 / 141440734 / +0.7% 0.77 / 0.79 / -1.9%
tpch_q04/duckdb:parquet 76827256 / 77574264 / -1.0% 98528087 / 99420174 / -0.9% 0.78 / 0.78 / -0.1%
tpch_q05/duckdb:parquet 123211023 / 124869472 / -1.3% 159229119 / 158164322 / +0.7% 0.77 / 0.79 / -2.0%
tpch_q06/duckdb:parquet 36091607 / 35794079 / +0.8% 60470112 / 61327927 / -1.4% 0.60 / 0.58 / +2.3%
tpch_q07/duckdb:parquet 118755725 / 117352182 / +1.2% 157192448 / 157089827 / +0.1% 0.76 / 0.75 / +1.1%
tpch_q08/duckdb:parquet 150667193 / 150500878 / +0.1% 194123715 / 195037624 / -0.5% 0.78 / 0.77 / +0.6%
tpch_q09/duckdb:parquet 240526874 / 238356231 / +0.9% 280332669 / 278458130 / +0.7% 0.86 / 0.86 / +0.2%
tpch_q10/duckdb:parquet 318260891 / 315937428 / +0.7% 339975349 / 344280953 / -1.3% 0.94 / 0.92 / +2.0%
tpch_q11/duckdb:parquet 43952149 / 43938893 / +0.0% 51954411 / 51316202 / +1.2% 0.85 / 0.86 / -1.2%
tpch_q12/duckdb:parquet 73014805 / 72804373 / +0.3% 92451521 / 92638335 / -0.2% 0.79 / 0.79 / +0.5%
tpch_q13/duckdb:parquet 293578923 / 293227096 / +0.1% 309936608 / 312586726 / -0.8% 0.95 / 0.94 / +1.0%
tpch_q14/duckdb:parquet 102434996 / 102160925 / +0.3% 139000003 / 138700487 / +0.2% 0.74 / 0.74 / +0.1%
tpch_q15/duckdb:parquet 59543526 / 58545984 / +1.7% 87704546 / 88765349 / -1.2% 0.68 / 0.66 / +2.9%
tpch_q16/duckdb:parquet 101345392 / 101830675 / -0.5% 108709308 / 109668032 / -0.9% 0.93 / 0.93 / +0.4%
tpch_q17/duckdb:parquet 82314676 / 81951009 / +0.4% 111758829 / 114997347 / -2.8% 0.74 / 0.71 / +3.4%
tpch_q18/duckdb:parquet 154644342 / 158384707 / -2.4% 167157055 / 172904089 / -3.3% 0.93 / 0.92 / +1.0%
tpch_q19/duckdb:parquet 100331468 / 99875581 / +0.5% 132038465 / 135151679 / -2.3% 0.76 / 0.74 / +2.8%
tpch_q20/duckdb:parquet 147813736 / 147572331 / +0.2% 184129115 / 186394590 / -1.2% 0.80 / 0.79 / +1.4%
tpch_q21/duckdb:parquet 257128990 / 258661149 / -0.6% 283372552 / 284540040 / -0.4% 0.91 / 0.91 / -0.2%
tpch_q22/duckdb:parquet 168909128 / 168980292 / -0.0% 184042805 / 184513322 / -0.3% 0.92 / 0.92 / +0.2%

No file size changes detected.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=1 on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Clickbench on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: TPC-H SF=1 on S3 📖

Commits: PR 2ad2f11d vs base e4dce8b7
Verdict: No clear signal (environment too noisy confidence)
Attributed Vortex impact: -1.7%
Engines: DataFusion No clear signal (-8.3%, environment too noisy confidence) · DuckDB No clear signal (+5.4%, environment too noisy confidence)
Vortex (geomean): hot 0.934x ➖ · cold 0.963x ➖
Parquet (geomean): hot 0.950x ➖ · cold 0.947x ➖
Shifts: Parquet (control) -5.0% · Median polish -3.1%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
  • Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
  • Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

datafusion / vortex-file-compressed / ns (0.887x ➖, 1↑ 2↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/datafusion:vortex-file-compressed 224584855 / 252318706 / -11.0% 793601369 / 721967905 / +9.9% 0.28 / 0.35 / -19.0%
tpch_q02/datafusion:vortex-file-compressed 320333690 / 362397240 / -11.6% 652306386 / 772685510 / -15.6% 0.49 / 0.47 / +4.7%
tpch_q03/datafusion:vortex-file-compressed 350319525 / 439054430 / -20.2% 978406299 / 1125793637 / -13.1% 0.36 / 0.39 / -8.2%
tpch_q04/datafusion:vortex-file-compressed 192568416 / 226516957 / -15.0% 396101435 / 670485382 / -40.9% 🟢 0.49 / 0.34 / +43.9%
tpch_q05/datafusion:vortex-file-compressed 440760243 / 462256955 / -4.7% 588699798 / 655532810 / -10.2% 0.75 / 0.71 / +6.2%
tpch_q06/datafusion:vortex-file-compressed 147008146 / 184092348 / -20.1% 280115579 / 351970105 / -20.4% 0.52 / 0.52 / +0.3%
tpch_q07/datafusion:vortex-file-compressed 482599811 / 520218033 / -7.2% 684524252 / 851750664 / -19.6% 0.71 / 0.61 / +15.4%
tpch_q08/datafusion:vortex-file-compressed 440389409 / 619189991 / -28.9% 718804076 / 725632307 / -0.9% 0.61 / 0.85 / -28.2%
tpch_q09/datafusion:vortex-file-compressed 299788236 / 349275672 / -14.2% 763285718 / 552481034 / +38.2% 🔴 0.39 / 0.63 / -37.9%
tpch_q10/datafusion:vortex-file-compressed 334756999 / 392457340 / -14.7% 527080786 / 569940788 / -7.5% 0.64 / 0.69 / -7.8%
tpch_q11/datafusion:vortex-file-compressed 385917537 / 349129311 / +10.5% 423390768 / 445172707 / -4.9% 0.91 / 0.78 / +16.2%
tpch_q12/datafusion:vortex-file-compressed 358139295 / 246516056 / +45.3% 🔴 467467207 / 600302877 / -22.1% 0.77 / 0.41 / +86.6%
tpch_q13/datafusion:vortex-file-compressed 137473061 / 137625019 / -0.1% 397635104 / 422300586 / -5.8% 0.35 / 0.33 / +6.1%
tpch_q14/datafusion:vortex-file-compressed 175900149 / 189432359 / -7.1% 305241418 / 359913304 / -15.2% 0.58 / 0.53 / +9.5%
tpch_q15/datafusion:vortex-file-compressed 312629358 / 394658290 / -20.8% 541242897 / 533557281 / +1.4% 0.58 / 0.74 / -21.9%
tpch_q16/datafusion:vortex-file-compressed 139037503 / 196123916 / -29.1% 269087419 / 269928546 / -0.3% 0.52 / 0.73 / -28.9%
tpch_q17/datafusion:vortex-file-compressed 439392151 / 333869135 / +31.6% 🔴 933365341 / 824514056 / +13.2% 0.47 / 0.40 / +16.3%
tpch_q18/datafusion:vortex-file-compressed 247137312 / 687407189 / -64.0% 🟢 474891217 / 1138273564 / -58.3% 🟢 0.52 / 0.60 / -13.8%
tpch_q19/datafusion:vortex-file-compressed 264589859 / 287675105 / -8.0% 431452911 / 417014963 / +3.5% 0.61 / 0.69 / -11.1%
tpch_q20/datafusion:vortex-file-compressed 301506577 / 298085290 / +1.1% 464635797 / 461692875 / +0.6% 0.65 / 0.65 / +0.5%
tpch_q21/datafusion:vortex-file-compressed 609619547 / 518039523 / +17.7% 728235207 / 734539368 / -0.9% 0.84 / 0.71 / +18.7%
tpch_q22/datafusion:vortex-file-compressed 155956186 / 186397294 / -16.3% 414363507 / 515337092 / -19.6% 0.38 / 0.36 / +4.1%
datafusion / parquet / ns (0.967x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/datafusion:parquet 158644164 / 173350016 / -8.5% 552041127 / 533896368 / +3.4% 0.29 / 0.32 / -11.5%
tpch_q02/datafusion:parquet 329471861 / 291983032 / +12.8% 540115295 / 591713098 / -8.7% 0.61 / 0.49 / +23.6%
tpch_q03/datafusion:parquet 237424005 / 271089279 / -12.4% 697733495 / 777116006 / -10.2% 0.34 / 0.35 / -2.5%
tpch_q04/datafusion:parquet 150378089 / 155149936 / -3.1% 298522782 / 365764170 / -18.4% 0.50 / 0.42 / +18.8%
tpch_q05/datafusion:parquet 372013751 / 344613900 / +8.0% 555001029 / 474051756 / +17.1% 0.67 / 0.73 / -7.8%
tpch_q06/datafusion:parquet 105689195 / 137995794 / -23.4% 208146477 / 273088746 / -23.8% 0.51 / 0.51 / +0.5%
tpch_q07/datafusion:parquet 352872077 / 353847750 / -0.3% 448031581 / 530553096 / -15.6% 0.79 / 0.67 / +18.1%
tpch_q08/datafusion:parquet 433210992 / 454413998 / -4.7% 523120649 / 555999225 / -5.9% 0.83 / 0.82 / +1.3%
tpch_q09/datafusion:parquet 396953548 / 417123291 / -4.8% 545940014 / 839369051 / -35.0% 🟢 0.73 / 0.50 / +46.3%
tpch_q10/datafusion:parquet 314766133 / 312760623 / +0.6% 455158728 / 474341308 / -4.0% 0.69 / 0.66 / +4.9%
tpch_q11/datafusion:parquet 281998773 / 288766902 / -2.3% 385895001 / 413074940 / -6.6% 0.73 / 0.70 / +4.5%
tpch_q12/datafusion:parquet 178004622 / 180075465 / -1.1% 295944014 / 374511749 / -21.0% 0.60 / 0.48 / +25.1%
tpch_q13/datafusion:parquet 283103622 / 287582781 / -1.6% 481505066 / 495117703 / -2.7% 0.59 / 0.58 / +1.2%
tpch_q14/datafusion:parquet 162833414 / 181192519 / -10.1% 281469111 / 340765749 / -17.4% 0.58 / 0.53 / +8.8%
tpch_q15/datafusion:parquet 333549546 / 397193321 / -16.0% 551751930 / 597758706 / -7.7% 0.60 / 0.66 / -9.0%
tpch_q16/datafusion:parquet 135610997 / 115202914 / +17.7% 338691872 / 249185877 / +35.9% 🔴 0.40 / 0.46 / -13.4%
tpch_q17/datafusion:parquet 295184612 / 305214779 / -3.3% 481096051 / 460600666 / +4.4% 0.61 / 0.66 / -7.4%
tpch_q18/datafusion:parquet 318002740 / 347808877 / -8.6% 563243523 / 478905958 / +17.6% 0.56 / 0.73 / -22.3%
tpch_q19/datafusion:parquet 156259352 / 183646441 / -14.9% 269525905 / 292384593 / -7.8% 0.58 / 0.63 / -7.7%
tpch_q20/datafusion:parquet 263466770 / 276822025 / -4.8% 466078077 / 444029927 / +5.0% 0.57 / 0.62 / -9.3%
tpch_q21/datafusion:parquet 329051255 / 306044024 / +7.5% 445982212 / 517594616 / -13.8% 0.74 / 0.59 / +24.8%
tpch_q22/datafusion:parquet 173485354 / 157174669 / +10.4% 319114637 / 347411279 / -8.1% 0.54 / 0.45 / +20.2%
duckdb / vortex-file-compressed / ns (0.984x ➖, 2↑ 2↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/duckdb:vortex-file-compressed 245366999 / 255410313 / -3.9% 261242959 / 280824589 / -7.0% 0.94 / 0.91 / +3.3%
tpch_q02/duckdb:vortex-file-compressed 438991780 / 419851957 / +4.6% 561061705 / 526997579 / +6.5% 0.78 / 0.80 / -1.8%
tpch_q03/duckdb:vortex-file-compressed 470087335 / 401898807 / +17.0% 579200815 / 487846114 / +18.7% 0.81 / 0.82 / -1.5%
tpch_q04/duckdb:vortex-file-compressed 246323701 / 390430961 / -36.9% 🟢 302408168 / 374005706 / -19.1% 0.81 / 1.04 / -22.0%
tpch_q05/duckdb:vortex-file-compressed 560591161 / 650321656 / -13.8% 805185933 / 621268846 / +29.6% 0.70 / 1.05 / -33.5%
tpch_q06/duckdb:vortex-file-compressed 200102032 / 171784473 / +16.5% 216981716 / 245292798 / -11.5% 0.92 / 0.70 / +31.7%
tpch_q07/duckdb:vortex-file-compressed 464140709 / 525270510 / -11.6% 551570875 / 449630250 / +22.7% 0.84 / 1.17 / -28.0%
tpch_q08/duckdb:vortex-file-compressed 901348990 / 655308590 / +37.5% 🔴 709553860 / 643447135 / +10.3% 1.27 / 1.02 / +24.7%
tpch_q09/duckdb:vortex-file-compressed 530186243 / 451743689 / +17.4% 659087041 / 479277749 / +37.5% 🔴 0.80 / 0.94 / -14.7%
tpch_q10/duckdb:vortex-file-compressed 572353522 / 527475319 / +8.5% 580026808 / 563771575 / +2.9% 0.99 / 0.94 / +5.5%
tpch_q11/duckdb:vortex-file-compressed 312633064 / 338452979 / -7.6% 387638129 / 324730945 / +19.4% 0.81 / 1.04 / -22.6%
tpch_q12/duckdb:vortex-file-compressed 308254159 / 250522601 / +23.0% 293833659 / 267013854 / +10.0% 1.05 / 0.94 / +11.8%
tpch_q13/duckdb:vortex-file-compressed 391481935 / 291505096 / +34.3% 🔴 480161820 / 296014987 / +62.2% 🔴 0.82 / 0.98 / -17.2%
tpch_q14/duckdb:vortex-file-compressed 249020022 / 242005067 / +2.9% 334184778 / 301305453 / +10.9% 0.75 / 0.80 / -7.2%
tpch_q15/duckdb:vortex-file-compressed 272300383 / 237461722 / +14.7% 346197524 / 299362710 / +15.6% 0.79 / 0.79 / -0.8%
tpch_q16/duckdb:vortex-file-compressed 206385765 / 205006085 / +0.7% 241842869 / 243932522 / -0.9% 0.85 / 0.84 / +1.5%
tpch_q17/duckdb:vortex-file-compressed 311833534 / 356947592 / -12.6% 301709960 / 384935077 / -21.6% 1.03 / 0.93 / +11.5%
tpch_q18/duckdb:vortex-file-compressed 338733806 / 382345712 / -11.4% 478602168 / 383854206 / +24.7% 0.71 / 1.00 / -28.9%
tpch_q19/duckdb:vortex-file-compressed 342602532 / 293814733 / +16.6% 340724487 / 463496997 / -26.5% 1.01 / 0.63 / +58.6%
tpch_q20/duckdb:vortex-file-compressed 510903373 / 678826072 / -24.7% 576417013 / 770727690 / -25.2% 0.89 / 0.88 / +0.6%
tpch_q21/duckdb:vortex-file-compressed 626661200 / 663237665 / -5.5% 664877006 / 679043611 / -2.1% 0.94 / 0.98 / -3.5%
tpch_q22/duckdb:vortex-file-compressed 222194636 / 423969512 / -47.6% 🟢 301490330 / 386492514 / -22.0% 0.74 / 1.10 / -32.8%
duckdb / parquet / ns (0.934x ➖, 1↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
tpch_q01/duckdb:parquet 182082744 / 186444064 / -2.3% 293945994 / 282527813 / +4.0% 0.62 / 0.66 / -6.1%
tpch_q02/duckdb:parquet 436183271 / 474518401 / -8.1% 473875889 / 513358696 / -7.7% 0.92 / 0.92 / -0.4%
tpch_q03/duckdb:parquet 440964316 / 388916387 / +13.4% 591582648 / 630518576 / -6.2% 0.75 / 0.62 / +20.8%
tpch_q04/duckdb:parquet 316850241 / 333484444 / -5.0% 399725420 / 352606432 / +13.4% 0.79 / 0.95 / -16.2%
tpch_q05/duckdb:parquet 582434302 / 587483889 / -0.9% 614231139 / 785134709 / -21.8% 0.95 / 0.75 / +26.7%
tpch_q06/duckdb:parquet 155004287 / 171065019 / -9.4% 193813990 / 227464493 / -14.8% 0.80 / 0.75 / +6.3%
tpch_q07/duckdb:parquet 452136758 / 479532912 / -5.7% 522412581 / 496973228 / +5.1% 0.87 / 0.96 / -10.3%
tpch_q08/duckdb:parquet 600059550 / 605142694 / -0.8% 632224999 / 745229145 / -15.2% 0.95 / 0.81 / +16.9%
tpch_q09/duckdb:parquet 589783719 / 944471828 / -37.6% 🟢 935994118 / 1264518164 / -26.0% 0.63 / 0.75 / -15.6%
tpch_q10/duckdb:parquet 626785647 / 662599778 / -5.4% 658073621 / 673531358 / -2.3% 0.95 / 0.98 / -3.2%
tpch_q11/duckdb:parquet 311954627 / 316939372 / -1.6% 353223283 / 328577210 / +7.5% 0.88 / 0.96 / -8.4%
tpch_q12/duckdb:parquet 255634683 / 310175817 / -17.6% 291630037 / 362225519 / -19.5% 0.88 / 0.86 / +2.4%
tpch_q13/duckdb:parquet 390691088 / 427374154 / -8.6% 473038011 / 597143617 / -20.8% 0.83 / 0.72 / +15.4%
tpch_q14/duckdb:parquet 265358924 / 240100433 / +10.5% 347351423 / 298651575 / +16.3% 0.76 / 0.80 / -5.0%
tpch_q15/duckdb:parquet 238898672 / 262284657 / -8.9% 292735792 / 300349427 / -2.5% 0.82 / 0.87 / -6.5%
tpch_q16/duckdb:parquet 276206425 / 331542209 / -16.7% 281725765 / 321132742 / -12.3% 0.98 / 1.03 / -5.0%
tpch_q17/duckdb:parquet 283699784 / 290567264 / -2.4% 371773743 / 392945962 / -5.4% 0.76 / 0.74 / +3.2%
tpch_q18/duckdb:parquet 414870368 / 392000396 / +5.8% 452303993 / 423335499 / +6.8% 0.92 / 0.93 / -0.9%
tpch_q19/duckdb:parquet 274776527 / 380346270 / -27.8% 500149485 / 373673328 / +33.8% 🔴 0.55 / 1.02 / -46.0%
tpch_q20/duckdb:parquet 470017089 / 434487627 / +8.2% 563494117 / 507857898 / +11.0% 0.83 / 0.86 / -2.5%
tpch_q21/duckdb:parquet 476556580 / 497658230 / -4.2% 569805881 / 553429713 / +3.0% 0.84 / 0.90 / -7.0%
tpch_q22/duckdb:parquet 304527104 / 316207253 / -3.7% 367802881 / 393167450 / -6.5% 0.83 / 0.80 / +2.9%

@joseph-isaacs joseph-isaacs changed the title Reduce V2 scan filtering, planning, and repeated DataFusion file reads Reduce V2 scan filtering, planning, and repeated file reads Oct 27, 2026
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review October 27, 2026 17:19
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026
@joseph-isaacs joseph-isaacs added changelog/performance A performance improvement action/bench-sql Run SQL benchmarks without Vortex Compact on this PR labels Oct 27, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026
@joseph-isaacs
joseph-isaacs marked this pull request as draft October 27, 2026 18:02
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 27, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026
@joseph-isaacs joseph-isaacs changed the title Reduce V2 scan filtering, planning, and repeated file reads Reduce V2 scan filtering, I/O, and batch-tail copying Oct 28, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 28, 2026
Reconcile with merged #10799 and leave overlapping scheduling and read
ownership policies to #10789, #10857, and #10969. Preserve current develop's
structural cache, sparse dictionary threshold, chunk sizing, short-edge
merging, retained segments, and benchmark defaults.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the ji/codex/optimize-scan-v2-10799 branch from 6cea8d0 to 2ad2f11 Compare October 29, 2026 16:40
@joseph-isaacs joseph-isaacs changed the title Reduce V2 scan filtering, I/O, and batch-tail copying Share scan filtering and planning work; reuse file handles and rebalance tails Oct 29, 2026
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 29, 2026 — with ChatGPT Codex Connector
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 29, 2026
joseph-isaacs added a commit that referenced this pull request Nov 3, 2026
…nce tails

Ported from #10844 into the combined V2 scan perf branch. Kept the branch's lazy struct-root expansion and field-path fast path in ExpressionPackRule (adding project_fields, the single-field fast path and PartitionCache after them), the branch's ActiveConjunct filter planner with its dense FilterPlan threshold (dropping 10844's dense_whole/0.2-density path; only infallible cross-column ORs run over the whole chunk range in Scan mode), the 64 KiB local coalescing config next to the cached file handle, and the startup initial_chunks alongside tail rebalancing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBBQHsaCfMVb8tMFxK5
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant