Repository navigation
Pair DuckDB late-materialization row ids with their files in Scan V2 - #11149
joseph-isaacs wants to merge 2 commits into
Conversation
A positive row selection, such as the row ids of a late-materialization lookup, was scanned like any other: every 65,536-row chunk of the range was planned, and segments were decoded and sliced before the few selected rows were filtered out. Choose `AccessMode::RandomAccess` when a positive selection keeps fewer than one row in sixteen of the scanned range. Plan only the 65,536-row regions holding selected rows, gather selected rows from segments in their stored encoding before decoding, and keep encoded take kernels visible for sparse numeric dictionary lookups as well as string ones. Native arrays keep their zero-copy slices, and dense selections keep the scan path. `ScanBuilder::with_access_mode` overrides the choice. Rows and their order are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
DuckDB's late materialization identifies rows by `(file_index, file_row_number)` and pushes the second scan a filter on each column separately. Every file of a multi-file scan therefore received the row numbers of every file, read rows it did not need, and left DuckDB to discard them in the semi join. When V2 is enabled, expose a `__vortex_row_id` virtual column that packs the file index and row number into one HUGEINT, and declare it, beside `file_index`, as the scan's row id columns. A join filter on that key becomes an exact per-file row selection, so each file skips entirely or reads only its own rows, through V2's sparse random access. SQL predicates on the key stay in DuckDB, which evaluates them exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.026x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.013x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.042x ➖, 0↑ 2↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.957x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.093x ➖, 0↑ 5↓)
duckdb / vortex-file-compressed / ns (0.961x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 1↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Summary
Split out of #10964. Stacked on #11146 (random access), whose commit it includes. Review only the top commit.
DuckDB's late materialization identifies rows by
(file_index, file_row_number)and pushes the second scan a separate filter on each column. Every file of a multi-file scan therefore received the row numbers of every file. Each file read rows it did not need, and DuckDB discarded them in the semi join.Changes
__vortex_row_idHUGEINT virtual column packs the file index and row number. It is declared, besidefile_index, as the scan's row id columns.duckdb_reader_initializenow receives the file's list index.include/vortex.hmatches what cbindgen generates.ORselections, all against V1, plus unit tests for the key codec and filter extraction.API Changes
This adds the user-visible virtual column
__vortex_row_idto V2 DuckDB scans.Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations. Base-vs-base noise: +1.3% geomean, up to ±8% per query.
develop(median / min)TPC-H triggers no late materialization over base tables. ClickBench
SELECT * … ORDER BY … LIMITqueries are the target, and that run is still in progress locally.cargo nextest run -p vortex-duckdb: 285 passed, 2 skipped.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code