Skip to content

Push DataFusion dynamic filters into V2 scans - #11215

Merged
joseph-isaacs merged 10 commits into
developfrom
ji/sleepy-clarke-5utr8u
Nov 4, 2026
Merged

joseph-isaacs merged 10 commits into
developfrom
ji/sleepy-clarke-5utr8u

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

This is #11073 rebased onto develop, plus an on/off switch for dynamic conjuncts. The switch is ported to Scan V2 from the unselective-skip commit in #10415.

On TPC-DS SF1, CI showed #11073 alone with 38 of 99 DataFusion/Vortex queries slower and 2 faster. Most of that is the zone-pruning pass a dynamic filter adds before each scan. This PR checks whether switching off dynamic conjuncts that reject almost nothing recovers that, while keeping the SF10/SF20 gains. Everything is gated on scan_v2, so the default V1 path is unchanged.

Changes

  • Commits 1–5 (from Push DataFusion dynamic filters into V2 scans #11073, unchanged):
    1. Give each inline V2 scan root its own IO session.
    2. Push DataFusion dynamic filters into persistent V2 scans.
    3. Keep the conjuncts of a dynamic filter that convert.
    4. Merge the key bounds of a dynamic filter into a between.
    5. Skip dynamic zone pruning before scans of one filtered chunk.
  • Switch off unselective dynamic conjuncts:
    • A conjunct containing dynamic expressions switches off after it keeps at least 99% of its input rows on 4 consecutive chunks at one dynamic version.
    • While off, the scan skips its zone pruning, its prefetch and its evaluation, until its dynamic value changes.
    • Opt-in through v2::ScanBuilder::with_skip_unselective_dynamic. Only the DataFusion persistent opener enables it, because DataFusion pushes dynamic filters down inexactly (PushedDown::No) and the producing TopK or join applies them again. DuckDB is unchanged.
    • Cached filter plans are reused only by scans with the same setting.
    • find_between only merges comparisons against literals, so a switched-off conjunct never carries a static restriction with it.
  • Tests:
    • Unit tests for the switch state: it trips after the streak, resets on a selective evaluation, re-arms on a version change, and never trips without the opt-in.
    • A V2 scan test, with the switch on and off. Once the conjunct is switched off, the next scan reads no more segments than an unfiltered scan. Tightening the bound switches it back on with correct results.

Checks

  • cargo test -p vortex-layout -p vortex-datafusion: all passed (826 + 746 + 14 + 1, 2 ignored).
  • cargo clippy -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warnings: clean.
  • cargo +nightly-2026-09-10 fmt -p vortex-layout -p vortex-datafusion: clean.
  • Not run locally: vortex-sqllogictest, vortex-array tests.
  • Benchmarks: pending. action/bench-sql covers TPC-DS SF1 for comparison with Push DataFusion dynamic filters into V2 scans #11073. SF10 is not in any PR preset.

The switch needs 4 chunks to trip, so it may not help the smallest SF1 scans.


Generated by Claude Code

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants