Skip to content

Push DataFusion dynamic filters into V2 scans - #11227

Closed
joseph-isaacs wants to merge 4 commits into
developfrom
ji/v2-dynamic-filters-min
Closed

joseph-isaacs wants to merge 4 commits into
developfrom
ji/v2-dynamic-filters-min

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

A smaller version of #11215: only the four commits that push dynamic filters down. It drops:

Everything is still gated on scan_v2, so the default V1 path is unchanged.

SF10 has only been benchmarked with the switch in place (DataFusion −11.2%, medium confidence), so this PR needs a TPC-DS SF10 run to confirm that it keeps that gain.

Changes

  1. Push DataFusion TopK and hash-join dynamic filters into persistent V2 scans. The producing operator still enforces them.
  2. Keep the conjuncts of a dynamic filter that convert, instead of dropping the whole filter when the hash-table lookup can't be converted.
  3. Merge a join's key >= min AND key <= max bounds into a single between.
  4. Skip dynamic zone pruning before scans of at most one filtered chunk. The V2 scan tests are updated for this.

Checks: not run locally. CI will run them.


Generated by Claude Code

joseph-isaacs and others added 4 commits November 4, 2026 16:43
Evaluate TopK and hash join dynamic filters in the V2 file scan, so zones and rows are dropped before the engine sees them. The producing operator keeps enforcing them. V1 scans keep them out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's hash table lookup has no Vortex equivalent, so one unconvertible conjunct turned the whole filter into a constant true and its bounds were lost. Convert each conjunct alone and keep the ones that convert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's dynamic filter bounds its probe keys with key >= min AND key <= max. Each
comparison decoded the key column again, and the scan's between rewrite only ran on
static filters, so the bounds now merge into one between when the filter is read.

TPC-DS SF10, 10 join queries against the previous commit: geomean 0.972, Q69 and Q10
0.93, none slower than 1.007.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A scan of at most one filtered chunk evaluates its conjuncts on that chunk next, so
pruning its zones first saves little and delays the chunk, and pruning a dynamic conjunct
proves its current predicate over the zones again. DataFusion splits small join probes into
many such scans. Larger scans keep pruning their dynamic conjuncts: skipping it for every
scan made TPC-DS SF10 Q82 and Q37 3.4-3.7x slower.

TPC-DS, all 99 queries against the previous commit: SF1 geomean 0.97, SF10 0.99 on best
runs and 1.01 on medians; the SF10 outliers were flat when rerun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Copy link
Copy Markdown
Contributor Author

Closing: this change landed in #11215 (merged 2026-11-04, same title), and all of its added lines are already on develop.


Generated by Claude Code

@robert3005
robert3005 deleted the ji/v2-dynamic-filters-min branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant