Repository navigation
Group filter conjuncts by shared fields, and decode a shared column once per unit - #11087
joseph-isaacs wants to merge 3 commits into
Conversation
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.014x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.978x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 1↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.004x ➖, 1↑ 1↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.039x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.072x ➖, 0↑ 2↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.016x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 1↓)
|
FilterExpr now groups the conjuncts of a scan filter into the connected components of the graph that links conjuncts reading a common top-level field. Each component is evaluated as a single unit (the conjunction of its members), so conjuncts over the same column compute together on each batch, while units that share no fields stay independently orderable by selectivity. A conjunct that reads the whole scope joins every field-reading conjunct; conjuncts that read no fields stay alone. Pruning plans are still built per original conjunct, so grouping never loses a member's metadata-only pruning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEMPLG2FLT3ZFA1auB Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The conjuncts of one filter unit read the same column, and encodings without compute kernels, such as the Zstd and Pco schemes Vortex Compact picks, make each conjunct decode that column to canonical again. ExecutionCtx::with_canonical_cache memoizes, by array identity, the canonical form of every shared array the context executes, both at the execute_until entry and for slot children. Only arrays referenced from elsewhere are cached, since a uniquely owned array cannot be executed again, and the cache is bounded and lives as long as its context. canonical_cache_stats reports hits, inserts and the canonical bytes the hits reused. The flat reader's filter evaluation and the v2 scan's piece mask enable the cache for their per-batch contexts. A Vortex Compact test filters a Zstd string column and a Pco integer column with two-conjunct units: each hits the cache once, and the cached evaluation is about 35-45% faster locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEMPLG2FLT3ZFA1auB Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
… execute_until Binary (comparisons, arithmetic, AND/OR), Like, Between and FillNull now implement execute_step: instead of executing their inputs themselves, they ask the executor for each input as columnar (canonical for Between) and are stepped again once it is. AND/OR still answer from a deciding constant first, and a null operand still decides a comparison, so neither executes the other side. A filter unit's whole expression tree therefore runs in one execute_until. execute_until keeps a local canonical cache of shared slot children, keyed by array identity: when two parents in the tree request the same child, such as the conjuncts of a unit reading one column, the second gets the first one's canonical form instead of decoding it again. It replaces the opt-in ExecutionCtx cache, so the flat reader and the v2 piece mask no longer enable anything. The cache lives for one call, only holds shared, non-canonical children, and is capped at 64 entries; the test harness exposes per-thread hit counters. Between's execute-parent adaptors read only the bounds now: the executor hands a suspended parent over with the executing input's slot detached, and reading all children panicked on it (caught by vortex-alp's patched_matches_primitive once Between started executing its input in a slot). On Vortex Compact data a two-conjunct unit hits the cache once, and evaluating it as one unit is faster than one conjunct at a time: about 25 vs 39 ms for a Pco u64 column and 93 vs 158 ms for a Zstd string column of 256k rows locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEMPLG2FLT3ZFA1auB Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
844699f to
7df0ccc
Compare
Ported from #10789 into the combined V2 scan perf branch. Kept the branch's ScanCache OnceCell API, chunk sizing (#10964) and filter planner (#11073/#11087); added the cores-per-reader concurrency and inline gate, per-root sessions in chunk tasks, batch-before-poll IO registration, MaskScan (wrapped inside the branch's ActiveConjunct and timed prune planner), one-scan windowed ProjectionMorsel for filtered chunks, the optimizer rules and V2_MIN_ROWS_PER_PARTITION; concurrency only shrinks filtered chunks (floor 64Ki), dropped projected_rows_per_chunk since unfiltered projections keep streaming stored batches in 1Mi chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBQHsaCfMVb8tMFxK5 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
FilterExprused to split a scan filter at every top-levelAND. Each conjunct became its own unit, evaluated one after another on the refined mask, even when several conjuncts read the same column, for examplemc.note LIKE … AND mc.note NOT LIKE …in JOB, or thel_commitdate/l_receiptdatechain in TPC-H q12. Splitting those conjuncts apart loses the partial compute they would otherwise share on each batch.This PR evaluates each connected component of the conjunct–column usage graph as a single unit. In a survey of the
vortex-bench/sqlqueries the graph is always a forest (no cycles). The largest component is 3 conjuncts: TPC-H q12, JOB 05b and JOB 08b. About 50 scans are affected, almost all of them in JOB.Grouping on its own didn't share any decoding. A unit's scalar functions each executed their own inputs, so every conjunct ran its own
execute_untiland decoded the shared column again. This hurt most with encodings that have no compute kernels, such as the Zstd and Pco schemes Vortex Compact picks. Building onScalarFnVTable::execute_step(#11100), the scalar functions on the filter path now execute their inputs through the executor. A unit's whole expression tree therefore runs in oneexecute_until, and a canonical cache local to that call decodes a shared column once.Changes
Conjunct grouping (
vortex-layout)FilterExpr::newgroups conjuncts into the connected components of the graph that links conjuncts reading a common top-level field. Each unit is the conjunction (bound::and_collect) of its members, kept in input order.root()) joins every conjunct that reads a field.FilterExpr::members(idx)returns the original conjuncts in a unit.FilterPlansstill builds one pruning plan per original conjunct, so a member's metadata-only pruning plan isn't lost because of a sibling in the same unit.find_betweenstill runs before grouping, so range pairs are merged first.Iterative filter scalar functions (
vortex-array)Binary,Like,BetweenandFillNullnow implementexecute_step. They ask for each input withexecute_inputinstead of executing it themselves: as columnar, or as canonical forBetween.AND/ORstill answer from a deciding constant first, and a null constant operand still decides a comparison, so neither executes the other operand.execute_parentkernels still run before an input is forced, so compressed-domain kernels (FSSTLIKE, ALP and sequence comparisons, …) still fire.Between's execute-parent adaptors now read only thelowerandupperslots. Step 2a of the executor hands over a suspended parent with the executing input's slot detached, and reading every child panicked on that empty slot.vortex-alp'spatched_matches_primitivehit this onceBetweenstarted executing its input in a slot.Canonical cache in
execute_until(vortex-array)execute_untilkeeps a local cache of the canonical forms of shared slot children, keyed by array identity. When a second parent in the tree requests the same child, it gets the first parent's canonical form instead of decoding it again.ExecutionCtx, and the filter call sites don't opt in to anything.test_harness::canonical_cache_stats().Tests
vortex-layout/src/scan/filter.rs:bound_conjunctssplitting is now tested directly.a > 1,a < c,c = 5), with independent fields kept apart.root()conjunct joins every field-reading conjunct.vortex-arrayexecutor tests:vortex/tests/canonical_cache.rs: a Vortex Compact test. It filters a Zstd string column and a Pco integer column with a two-conjunct unit, and asserts one cache hit and the same mask as evaluating the conjuncts one by one. It also prints the timings. Locally, fastest of 5 runs:u64🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBEMPLG2FLT3ZFA1auB