Repository navigation
Stream unfiltered Scan V2 projections in larger chunks - #11142
joseph-isaacs wants to merge 1 commit into
Conversation
Unfiltered projections planned 65,536-row work chunks and gathered each chunk's stored batches into one array, which repeated planning and setup for every chunk and copied across stored boundaries during conversion. Plan up to 1,048,576 rows per projection chunk and emit each stored batch as it is produced, so planning is amortized and batches keep their stored boundaries and encodings. The byte budget still shrinks full-file projections of wide rows, now also when the scan is filtered, and filtered projections still gather their pieces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 1↑ 1↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.157x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.037x ➖, 0↑ 4↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 1↑ 1↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.022x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
|
Summary
Split out of #10964 as an independently measurable piece.
Unfiltered projections planned 65,536-row work chunks and gathered each chunk's stored batches into one array. That repeated planning for every chunk and copied across stored boundaries during conversion.
Changes
Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations against
develop(246a397). Base-vs-base noise: +1.3% geomean, up to ±8% per query.TPC-H has few unfiltered projections. DataFusion TPC-H and ClickBench runs are still in progress locally.
cargo nextest run -p vortex-layout: 737 passed.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code