Repository navigation
Push DataFusion dynamic filters into V2 scans - #11227
Closed
joseph-isaacs wants to merge 4 commits into
Closed
joseph-isaacs wants to merge 4 commits into
joseph-isaacs wants to merge 4 commits into
Conversation
Evaluate TopK and hash join dynamic filters in the V2 file scan, so zones and rows are dropped before the engine sees them. The producing operator keeps enforcing them. V1 scans keep them out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's hash table lookup has no Vortex equivalent, so one unconvertible conjunct turned the whole filter into a constant true and its bounds were lost. Convert each conjunct alone and keep the ones that convert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A join's dynamic filter bounds its probe keys with key >= min AND key <= max. Each comparison decoded the key column again, and the scan's between rewrite only ran on static filters, so the bounds now merge into one between when the filter is read. TPC-DS SF10, 10 join queries against the previous commit: geomean 0.972, Q69 and Q10 0.93, none slower than 1.007. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A scan of at most one filtered chunk evaluates its conjuncts on that chunk next, so pruning its zones first saves little and delays the chunk, and pruning a dynamic conjunct proves its current predicate over the zones again. DataFusion splits small join probes into many such scans. Larger scans keep pruning their dynamic conjuncts: skipping it for every scan made TPC-DS SF10 Q82 and Q37 3.4-3.7x slower. TPC-DS, all 99 queries against the previous commit: SF1 geomean 0.97, SF10 0.99 on best runs and 1.01 on medians; the SF10 outliers were flat when rerun. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Contributor
Author
|
Closing: this change landed in #11215 (merged 2026-11-04, same title), and all of its added lines are already on develop. Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A smaller version of #11215: only the four commits that push dynamic filters down. It drops:
with_skip_unselective_dynamicand the prefetch skip, about 330 lines). The switch was added to recover TPC-DS SF1 slowdowns, but the benchmarks don't show that it did. DataFusion on TPC-DS SF1 was +9.6% for Push DataFusion dynamic filters into V2 scans #11073 without the switch, and +6.7% / +8.9% for Push DataFusion dynamic filters into V2 scans #11215 with it. Both are low-confidence, so the difference is within noise.developas Count each run's selected rows with one pass over the RunEnd filter mask #11221.Everything is still gated on
scan_v2, so the default V1 path is unchanged.SF10 has only been benchmarked with the switch in place (DataFusion −11.2%, medium confidence), so this PR needs a TPC-DS SF10 run to confirm that it keeps that gain.
Changes
key >= min AND key <= maxbounds into a singlebetween.Checks: not run locally. CI will run them.
Generated by Claude Code