Repository navigation
Build each Scan V2 file plan once under concurrent scans - #11139
joseph-isaacs wants to merge 1 commit into
Conversation
Threads that open the same file race to lower its root and optimize the same projection and filter plans. The cache kept the first result but every racing scan paid for planning. Use per-key `OnceCell`s so concurrent scans wait for one build and reuse it, independently per projection and filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.972x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.011x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 1↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.017x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.985x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 1↑ 3↓)
datafusion / parquet / ns (1.058x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.046x ➖, 0↑ 0↓)
|
Summary
Split out of #10964 as an independently measurable piece.
DuckDB threads that open the same file race to lower its root and optimize the same projection and filter plans.
ScanCachekept the first result, but every racing scan still paid for planning. This makes concurrent scans wait for one build and reuse it.