Repository navigation
Push DataFusion dynamic filters into V2 scans - #11215
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is #11073 rebased onto
develop, plus an on/off switch for dynamic conjuncts. The switch is ported to Scan V2 from the unselective-skip commit in #10415.On TPC-DS SF1, CI showed #11073 alone with 38 of 99 DataFusion/Vortex queries slower and 2 faster. Most of that is the zone-pruning pass a dynamic filter adds before each scan. This PR checks whether switching off dynamic conjuncts that reject almost nothing recovers that, while keeping the SF10/SF20 gains. Everything is gated on
scan_v2, so the default V1 path is unchanged.Changes
between.v2::ScanBuilder::with_skip_unselective_dynamic. Only the DataFusion persistent opener enables it, because DataFusion pushes dynamic filters down inexactly (PushedDown::No) and the producing TopK or join applies them again. DuckDB is unchanged.find_betweenonly merges comparisons against literals, so a switched-off conjunct never carries a static restriction with it.Checks
cargo test -p vortex-layout -p vortex-datafusion: all passed (826 + 746 + 14 + 1, 2 ignored).cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warnings: clean.cargo +nightly-2026-09-10 fmt -p vortex-layout -p vortex-datafusion: clean.vortex-sqllogictest,vortex-arraytests.action/bench-sqlcovers TPC-DS SF1 for comparison with Push DataFusion dynamic filters into V2 scans #11073. SF10 is not in any PR preset.The switch needs 4 chunks to trip, so it may not help the smallest SF1 scans.
Generated by Claude Code