Repository navigation
Pair file and row ids for DuckDB V2 late materialization - #10971
joseph-isaacs wants to merge 1 commit into
Conversation
DuckDB late materialization pushes the file index and file-local row number down as independent sets, so a multi-file re-fetch can read rows that pair a file with another file's row number. Under Scan V2, expose a packed HUGEINT `__vortex_row_id` virtual column (file index << 64 | row) and use it, together with the file-index hint for pre-open pruning, as the row-id columns. Each file intersects the exact rows it owns with the existing selection, bounded by its row count, and files with no selected rows are skipped. Arbitrary SQL predicates on the identifier stay in DuckDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gzeoH7sKmzLEiE5q Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.037x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.945x ➖, 21↑ 4↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.191x ➖, 0↑ 4↓)
datafusion / parquet / ns (1.011x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.229x ➖, 0↑ 8↓)
duckdb / parquet / ns (1.045x ➖, 0↑ 3↓)
|
Merging this PR will regress 1 benchmark2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes |
| vector<ColumnBinding> bindings; | ||
| ColumnLifetimeAnalyzer::ExtractColumnBindings(*filter, bindings); | ||
| for (const auto &binding : bindings) { | ||
| if (binding.table_index == get.table_index && |
There was a problem hiding this comment.
We should push this filter and reject it on Rust side based on column id.
| bool is_vortex_scan(const BoundTableFunction &function); | ||
|
|
||
| constexpr inline idx_t COUNT_STAR_PROJ_IDX = std::numeric_limits<idx_t>::max(); | ||
| constexpr inline column_t VORTEX_ROW_ID_COLUMN = UINT64_C(9223372036854775811); |
There was a problem hiding this comment.
Please double-check it's not taken already
| } | ||
|
|
||
| #[rstest] | ||
| #[case::topk("SELECT id FROM ? ORDER BY score DESC, id LIMIT 10", None)] |
There was a problem hiding this comment.
This should be an .slt. I actually plan to remove this file altogether.
|
|
||
| /// The state of an open file that the threads scanning it share. | ||
| pub struct FileScanState { | ||
| file_index: AtomicU64, |
|
Closing as superseded: #10964 contains this change unchanged ( Generated by Claude Code |
Summary
DuckDB late materialization pushes
file_indexandfile_row_numberdown as two independent sets. In a multi-file scan, the re-fetch can then read rows that pair one file with another file's row number. Under Scan V2 this PR switches the row-id columns to a packed HUGEINT key that keeps each file/row pair intact.Split out of #10964. This PR has only the row-id part. The TopK feedback, startup chunks, sparse dictionary export and diagnostics stay in that PR.
Changes
__vortex_row_idHUGEINT virtual column,(file_index << 64) | file_row_number. When Scan V2 is on,get_row_id_columnsreturns{file_index, __vortex_row_id}.file_indexstays in the list so files can still be pruned before they are opened.duckdb_reader_initializenow receives the file'sfile_list_idx. Each file takes the ids that belong to it, intersects them with the existing row selection, and drops rows past its row count. Files left with no rows are skipped. The selection is then used by both the V1 and V2 scan builders.row_idx()for the key column. Before export, it is replaced with the packedi128value as aDecimal(38, 0)array, which DuckDB exports as HUGEINT.IN,=,AND/ORand optional join filters on the key. If any SQL predicate references the key,filter_pushdownis disabled, so DuckDB still evaluates arbitrary HUGEINT expressions exactly.row_id/tests.rs).INwith duplicates and NULL, an empty selection,BETWEENandOR.Generated by Claude Code