Skip to content

perf: backport JOIN dynamic-filter and compact pruning optimizations - #39

Merged
discord9 merged 21 commits into
greptimedb-55.1-thin-forkfrom
fix/complete-join-pruning-batch
Nov 10, 2026
Merged

discord9 merged 21 commits into
greptimedb-55.1-thin-forkfrom
fix/complete-join-pruning-batch

Conversation

@discord9

@discord9 discord9 commented Nov 6, 2026 •

Copy link
Copy Markdown

Lane split: this PR stays the JOIN/pruning lane; the aggregate NULL-bound repair is a
separate, independent PR #41 (upstream apache#25579). Both branches are based directly on greptimedb-55.1-thin-fork (e3221b8f7); neither depends on the other.
#37 (cast constraints) remains a separate PR and is not an ancestor of either branch.

Which issue does this PR close?

None. This is a downstack backport of already-merged upstream DataFusion changes; the upstream
reports keep their own ownership, so no Closes line is set.

Rationale for this change

Target workload: a 10m-row FIELD traces JOIN on the UTF8 trace_id key (the original query). The
recent measurement was the same JOIN with an added l.timestamp predicate, on the same binary and
dataset: 1.253–1.815 s original vs 0.112–0.194 s recent (ABBA). The added predicate changes the
work done, so this is a bounded historical observation, not a like-for-like regression number, and
this PR does not claim the original issue is fixed.

It does bound the problem: a dynamic filter is already present at the scan; 10m storage rows are
read and ~151,500 rows reach the probe. The open question is whether usable statistics pruning can
be derived from that filter — not whether the filter exists.

  • #25602's numeric key-range bitmap cannot apply here (UTF8 trace_id); its CASE-branch
    relaxation is a separate candidate.
  • IN/NOT IN compact pruning (#24525, #24526, #24781, #24954, #25012) only engages
    within max_in_list_size, and GT's builder does not raise that cap above the default 20 — the
    features alone are therefore unproven for this workload.
  • Prepared-build (#25491) and memory-accounting (#24319, #25508, #25496), plus the NOT-IN
    and ordered-type capabilities, are retained as the requested preserve package, not as a hard
    prerequisite.

Evidence boundary. Earlier full-12 timings were taken on the combined tree with #37; there is no
performance result for this split branch, and the original scan-volume issue remains unproven as
fixed.

Ordinary-concat accounting boundary. The ordinary hash-join build keeps its input, final
batch, hash-map/row-index, bitmap, computed-key and producer-lease accounting. It no longer
pre-reserves a predicted copy of inputs plus output before Arrow concatenates them. During ordinary
concat, inputs and output may therefore transiently exceed the memory pool; the reservation is
reconciled with the retained batch only after concatenation, and a physical allocation can still
fail. Prepared-build allocation/reconciliation is unchanged. This is an explicit, accepted
tradeoff: the ordinary concat peak guard from the retained memory-accounting backport was removed
rather than extended with more per-type prediction. apache#25496 still retains its final-held and
computed-key accounting here, but this part is not claimed to be a complete or fully faithful
backport of the upstream peak guard.

What changes are included in this PR?

21 commits, 47 files, +11817/−394 against e3221b8f7: 11 upstream backports with original authors,
SHAs and URLs (apache#24319, apache#25508, apache#25496, apache#25491, apache#24525, apache#24526, apache#24781, apache#24954, apache#25012, apache#25500,
apache#25602); 4 immediate adapt commits for fork APIs, imports and probe state (3e4ce27dc,
6e033ea40, 2e16b818c, 7cf6172fe); then local production hardening (join-pruning NULL and
producer-lifetime safety), the retained local regression commit (9 files), SLT alignment and
the final ordinary-concat accounting simplification. A package, not a minimum: no commit, subset
or the whole set is claimed to be the minimal fix, and no new benchmark framework is added.

Direct backport vs fork adapt

Short hashes are listed below. Upstream author/date are preserved in commit metadata; the full
upstream SHA and PR URL are recorded in each backport's message. "± exact" means the actual
added/deleted line contents match upstream, in order per file; normalization is stated when
used. It does not mean whole files are byte-identical. Patch-id equality is stated separately.
"Separate adapt" names the immediately following fork-adaptation commit; "—" means there is no
separate adaptation commit, not that the backport is unmodified.

Upstream PR Commit here Original vs fork deviation Separate adapt
apache#24319 8f22528b5 ± exact, patch-id exact —
apache#25508 12d596120 ± exact after normalization, not byte-exact: two accounting hunks were relocated inline because the upstream helper is absent here 3e4ce27dc (extract that helper + caller wiring)
apache#25496 d658e52ce logic and self-tests faithful, not patch-id exact: context anchors and import wrapping differ, upstream's old-rustdoc deletion has no base to delete, and the new doc line anchors on an existing helper 6e033ea40 (adds the Array import, uses the existing perfect-hash candidate helper)
apache#25491 276f9302b backport with fork mapping, not pure unmodified: benchmark / prepared.rs / test bytes preserved as upstream, but this commit already carries the fork's old self-codec serialization guard mapping and the upstream mark-field context is absent 2e16b818c (older probe atomics, null-aware collector argument, docs, and the existing apache#25508 test move) — the fork mapping is split across both commits; not all of it is here
apache#24525 105af0566 ± exact, patch-id exact —
apache#24526 5b464f88f ± exact in upstream order per file; patch-id differs only on lockfile version context (55.0 → 55.1), which is not a lock dependency version change —
apache#24781 57f27e903 ± exact, patch-id exact —
apache#24954 fc503ee22 ± exact, patch-id exact —
apache#25012 a23cfb1f8 ± exact, patch-id exact —
apache#25500 e8362c6e8 production matcher additions/deletions match upstream; not patch-id exact: rustdoc keeps different older wording, and the same upstream new tests were inserted into the old test layout 7cf6172fe (three-argument → two-argument projection statistics test helper only)
apache#25602 9b71f365b raw content of the new modules matches upstream; not patch-id exact: the fork's self-codec is not destructured, so upstream's added pruning_bitmap: _, destructuring binding is inapplicable and the import anchors are mapped —

Local commits in the same lane, for attribution:

  • a64b1a6c4 — 9 files, +182/−42. Fork production work: the NULL/TRY_CAST/CASE full-match
    proof, the inverse guard, producer-lease reservation, and execution-local visited/bitmap
    ownership. This is our own change, not an upstream PR.
  • 84cd0e433 — 9 files, +1988/−7. Extra, adjusted and moved regression tests, including changes to
    existing tests. Upstream's original self-tests remain in their backport commits above.
  • ca152952e — 1 file, +1/−1. One SQLLogicTest metric line: drops bytes_processed, which has no
    producer in this fork. Only the incidental predicate_cache_records value is ignored; all result columns and the
    substantive prune counters are unchanged. It mirrors our snapshot, not fork production behavior.
  • 17d13cf7f, 90bb0ca7d — local P2 concat-accounting iterations: initially added selected-range
    nested-array and shared-Dictionary estimates, then extended them. These predictor changes are
    superseded by the final simplification and are not part of the final ordinary-concat behavior.
  • 203689a71 — local adaptation: removes the ordinary-concat pre-allocation peak predictor and
    its List/Map/Struct/FixedSizeList/Dictionary recursion; concatenates first, then reconciles only
    the retained batch delta. Input/hash-map/row-index/bitmap/computed-key and producer-lease
    accounting remain. Prepared-build reservation/reconcile is untouched.

This table identifies fork mappings already present in backport commits as well as the separately
committed adaptations. The mapped backports are not described as unmodified cherry-picks. In
particular, apache#25496's ordinary-concat peak-guard portion is not fully retained; its final-held and
computed-key accounting is retained.

Are these changes tested?

  • Preserved SQLLogicTest coverage (not re-run in this split): dynamic_row_group_pruning.slt (build-side key-range bitmap pruning, incl. the
    widening-cast-through case), parquet_filter_pushdown.slt, information_schema.slt alignment.
  • cargo test --offline --locked -p datafusion-physical-plan --lib -- joins::hash_join::exec
    — 414 passed, 0 failed. This includes prepared-build tests, final-held-memory boundary
    tests (limit=held succeeds, limit=held−1 fails and releases the reservation), shared nested
    Dictionary join coverage, and the batch-delta reconciliation test that preserves an extra
    hash-map charge.
  • cargo fmt --all -- --check and git diff --check — PASS. Targeted clippy
    (-p datafusion-physical-plan --all-targets --all-features -- -D warnings) — PASS.
  • A local full extended-suite run on the final head was attempted with the AGENTS command.
    All Rust test binaries that ran passed except four pre-existing SQLLogicTest files whose expected
    planner error wording says Internal error while this tree emits Execution error /
    signature detail: errors.slt, array/array_range.slt, math.slt, and
    spark/math/round.slt. The same four files fail identically on base e3221b8f7; this is
    baseline expectation drift, not a result of this PR. The full suite is therefore not
    reported as green.
  • The earlier full-test evidence belongs to the pre-split combined tree; only the final-head
    local run above is claimed for this split branch.

Are there any user-facing changes?

  • EXPLAIN can show KEY_RANGE_BITMAP(...) dynamic pruning on hash-join scans.
  • The datafusion.execution.parquet.max_in_list_size description now documents extended
    IN/NOT IN eligibility (no new option; default stays 20).

ryux1 and others added 12 commits November 6, 2026 21:38
## Which issue does this PR close?

- Closes apache#24310.

## Rationale for this change

Record batch memory accounting runs on hot execution paths. The current
implementation materializes ArrayData for every array and allocates a
hash set even for small batches, adding measurable overhead to queries
that frequently update memory reservations.

## What changes are included in this PR?

- Traverse Arrow arrays directly and recursively count their backing
buffers without materializing ArrayData.
- Track the first 16 buffer identities inline, then promote to a hash
set for wider batches or counters spanning many batches.
- Preserve shared-buffer deduplication and full buffer-capacity
accounting semantics.
- Add parity coverage against the previous ArrayData traversal for
primitive, binary/view, list/view, fixed-size, struct, union,
dictionary, map, and all legal run-end index layouts.
- Add a focused Criterion benchmark across column counts, row counts,
and primitive/list/struct layouts.

Criterion point estimates from `cargo bench -p datafusion-common --bench
record_batch_memory`, measured sequentially on upstream `main` and this
PR on the same machine:

### Column count (8,192 rows, Int64)

| Columns | main | this PR | Speedup |
|---:|---:|---:|---:|
| 1 | 81.999 ns | 10.645 ns | 7.70x |
| 4 | 308.11 ns | 38.669 ns | 7.97x |
| 16 | 1.2543 us | 173.55 ns | 7.23x |
| 64 | 4.7457 us | 1.3914 us | 3.41x |

### Row count (4 Int64 columns)

| Rows | main | this PR | Speedup |
|---:|---:|---:|---:|
| 1 | 299.20 ns | 39.939 ns | 7.49x |
| 128 | 302.99 ns | 38.914 ns | 7.79x |
| 8,192 | 307.64 ns | 39.740 ns | 7.74x |
| 65,536 | 310.61 ns | 38.994 ns | 7.97x |

### Array layout (4 columns, 8,192 rows)

| Layout | main | this PR | Speedup |
|---|---:|---:|---:|
| Primitive Int64 | 311.01 ns | 40.140 ns | 7.75x |
| List of Int64 | 778.29 ns | 82.865 ns | 9.39x |
| Struct of two Int64 fields | 1.0072 us | 113.70 ns | 8.86x |

Each list row contains two Int64 values. Each struct column contains two
Int64 child fields.

## Are these changes tested?

Yes.

- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-common utils::memory --lib` (10 passed)
- `RUST_BACKTRACE=1 cargo test --profile ci --exclude
datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-cli --workspace --lib --tests --bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
- Focused Criterion comparison shown above

## Are there any user-facing changes?

No API or behavior changes. This reduces CPU and allocation overhead in
record batch memory accounting.

AI assistance: OpenAI Codex assisted with implementation and test
execution. I reviewed the change and its behavior end to end.

Backport provenance:
- Upstream PR: apache#24319
- Upstream commit: c7be84a
- The upstream diff is unchanged, including its own tests and benchmark.
- No fork adaptation is included in this commit. Local Arrow 59 validity-buffer
  regressions are deliberately deferred to the final additional-test section.
- The body above is the original upstream description; its benchmark estimates
  and verification statements describe upstream, not this fork's measurements.

Signed-off-by: discord9 <discord9@163.com>
## Which issue does this PR close?

Addresses the shared accounting bug identified while reviewing
[apache#25491](apache#25491 (comment)).
Split out so it can be reviewed and backported independently of
prepared-build reuse.

## Rationale for this change

Hash joins reserve memory for their hash buckets, but omit the row-index
vector used to link build rows with the same key. As a result, a join
can allocate beyond its admitted budget and underreport its build
memory. For example, 100 million build rows need an additional 400 MB
for this vector that the memory pool currently does not see.

## What changes are included in this PR?

The shared hash-map constructor now reserves the bucket estimate and
row-index storage together before allocating either, and includes both
in the build-memory metric. This covers ordinary CollectLeft and
partitioned joins, using four bytes per row for the u32 representation
and eight for u64. Perfect-hash array maps keep their existing
accounting.

## What is the testing strategy for this PR?

A regression gives the constructor one byte less than the complete
budget and checks that it refuses the allocation without charging the
pool or metric. It also verifies success at the exact limit and release
on drop. The test fails against the original constructor and passes with
this fix.

All 533 hash-join tests, full workspace Clippy with all targets/features
and warnings denied, formatting, and whitespace checks pass on this
branch. The combined prepared-build branch also passed the extended
workspace run with this constructor fix (11,995 Rust tests and 521 SQL
test files; eight tests ignored).

Local checks used Rust 1.98.1 and temporary source overrides for the
requested public releases of Arrow 60.0.0, object_store 0.14.2,
sqlparser 0.63.0, and compression-codecs 0.4.42 because the registry
mirror lacked them. No dependency or lockfile changes are included.
Ordinary-path benchmark comparisons for the combined branch, including
the accounting fix, are reported in apache#25491; they are not an isolated
performance claim for this fix.

## Are there any user-facing changes?

Hash joins using these maps account for their row-index storage in
memory limits and build-memory metrics. A budget that previously
succeeded only because this allocation was omitted can now be rejected
before allocation.

Backport provenance:
- Upstream PR: apache#25508
- Upstream commit: 43f567a
- All upstream additions and deletions are retained, including its regression
  test. Only patch anchors and indentation are relocated: the fork still has
  inline map construction instead of upstream's new_join_hashmap helper.
- No helper extraction or fork interface adaptation is included here. Those
  changes follow in the immediately next commit.
- This original-only intermediate state cannot build test targets: the upstream
  test references that absent helper. Checks apply to the following adapted
  state, not retroactively to this commit.
- The description above records upstream's verification, not new fork results.

Signed-off-by: discord9 <discord9@163.com>
The fork constructs the join map inline, while upstream apache#25508 updates and
regression-tests a new_join_hashmap helper. Extract that existing constructor
and wire its caller, retaining the preceding upstream allocation/accounting
logic unchanged. This supplies the missing interface without mixing it into
the upstream backport.

Upstream PR: apache#25508
Upstream commit: 43f567a
This is a local fork adaptation, not an upstream-authored change.

Verification of this adapted state: cargo fmt --all -- --check; full
all-target/all-feature Clippy with -D warnings; datafusion-physical-plan
nextest (1707 passed). The preceding original-only intermediate test target
was not buildable and is not retrospectively claimed to have passed.

Signed-off-by: discord9 <discord9@163.com>
…ashJoinExec (apache#25496)

## Which issue does this PR close?

- Closes apache#25269.

## Rationale for this change

A hash join reserves every build-side batch as it arrives, then copies
all of them
into a single batch without reserving the copy, while the input batches
stay alive
until the build finishes. Near the memory limit the process therefore
holds about
twice the build side while the memory pool sees it once, so instead of a
`ResourcesExhausted` error the process can be killed by the OS. It also
means
nothing that reacts to pool pressure can rely on what a hash join
reports.

Measured on a 1.25 GB build side (TPC-H SF1 `lineitem`, 16 columns, as
the build
side): the pool was charged ~1.25 GB while peak RSS was ~2.46 GB.

## What changes are included in this PR?

- `concat_build_batches`: reserves the copy before `concat_batches`,
drops the input
batches, then trims the reservation to what the concatenated batch
retains
(`get_record_batch_memory_size`), growing instead if the estimate was
low.
  `build_mem_used` follows every grow and shrink.
- Nothing extra is reserved when there is nothing to copy: concatenating
a single
batch is zero-copy, and for `Utf8View`/`BinaryView` columns only the
views are
  copied while the data buffers stay shared with the inputs.
- `try_create_array_map` becomes `array_map_key_range`: it only decides
whether the
perfect hash join applies and reserves the `ArrayMap`. Both the hash map
and the
`ArrayMap` paths now hand the batches to the helper by value, so the
inputs are
gone before the visited bitmap and the null-aware scope maps are
allocated.
- Join key arrays that do not share the buffers of the build batch (any
on-expression
that is not a plain column) are kept for the whole join and are now
reserved too.
  This uses a new public `RecordBatchMemoryCounter::count_array`.

This does not reduce memory usage: the build still peaks at about twice
the build
side, it is now counted. A memory-limited query that only fit because
the copy was
not counted can now fail with `ResourcesExhausted`.

## What is the testing strategy for this PR?

- `join_build_concat_is_reserved` (CollectLeft and Partitioned, hash map
and
`ArrayMap`): fails with `ResourcesExhausted` at map + 1.5x inputs,
passes at
  map + 3x inputs. It fails on `main` in all four variants.
- `join_build_key_arrays_are_reserved` (hash map and `ArrayMap`): with a
single build
batch and an `a1 + 1` join key, the join fails at inputs + map + half
the key array
and passes with room for it; a plain column key passes at the lower
limit. It fails
  when the reservation is removed.
- Unit tests of the helper: output equals `concat_batches` in both
orders, the copy
is reserved and the reservation then matches the retained batch, a
single batch
  is not reserved twice, view data buffers are not reserved twice.
- Existing `single_partition_join_overallocation` /
`partitioned_join_overallocation`
  are unchanged.

## Are there any user-facing changes?

One public addition: `RecordBatchMemoryCounter::count_array`. Hash joins
running close
to their memory limit may now report `ResourcesExhausted` where they
previously exceeded the limit unnoticed.

Backport provenance:
- Upstream PR: apache#25496
- Upstream commit: 21a3215
- Preserve upstream count_array, concat/accounting helpers and collector, plus
  all original tests. Resolve patch anchors and import wrapping for the fork.
- The upstream rustdoc replacement's old line is absent from the fork; retain
  its added line at the corresponding existing helper without fabricating a
  deletion. Existing fork &bounds context is unchanged.
- No fork candidate guard or missing Array import is mixed into this commit;
  those adaptations follow immediately. This original-only state has not been
  checked for buildability. Validate the adapted state, not this intermediate.
- The original description's tests and benchmarks refer to upstream results.

Signed-off-by: discord9 <discord9@163.com>
Import Array for the upstream concat estimator: this trait was already
imported in the upstream parent but not in the older fork.
Use the fork's existing supported-type candidate helper when checking the
array-map key range, preserving upstream's eligibility requirement.

Upstream PR: apache#25496
Upstream commit: 21a3215
This commit contains only fork adaptation, not upstream-authored changes.

The adapted state compiled in full all-target/all-feature Clippy. Previously
started common/physical-plan nextest also completed: 2289 passed.

Signed-off-by: discord9 <discord9@163.com>
## Which issue does this PR close?

Part of [Comet
apache#6013](apache/datafusion-comet#6013). [Comet
apache#6037](apache/datafusion-comet#6037) uses this
API. The shared row-index accounting fix in
[apache#25508](apache#25508) is included in
the base branch.

## Rationale for this change

Independent joins against the same build snapshot repeat preparation and
retain separate hash tables. `CollectLeft` shares a build among
partitions of one join, but cannot share it across independently created
plans.

For example, eight concurrent requests joining events to the same users
snapshot can share one users lookup while probing their own event
batches. Preparing that lookup once avoids seven redundant builds.
Applications can replace the snapshot while existing consumers finish
with the old one. This supports both Comet broadcast tasks and other
DataFusion applications.

Comet creates independent native joins for its executor tasks, so their
existing builds can overlap across task slots. The intended benefit is
sharing one retained build batch and hash table and avoiding repeated
materialization and hashing work. Removing repeated builds does not
imply a proportional latency improvement: cold consumers wait for
preparation, then probe independently. Latency depends on scheduling,
contention, build size, coordination, and whether a prepared build is
already retained. This PR does not establish end-to-end Comet latency,
process CPU-time, or RSS savings.

## What changes are included in this PR?

`HashJoinExec::prepare_build` returns an immutable
`PreparedHashJoinBuild` that compatible joins attach explicitly:

```rust
let prepared = first_join
    .prepare_build(users_snapshot, durable_pool, config)
    .await?;
let first = first_join.builder()
    .with_prepared_build(Arc::clone(&prepared))
    .build()?;
let second = second_join.builder()
    .with_prepared_build(prepared)
    .build()?;
```

Consumers share build rows, lookup storage, and the retained memory
reservation. Probe progress, residual predicates, and dynamic filters
remain independent. Preparation charges retained data and temporary
working memory; errors and cancellation release unfinished work.
Concatenation alignment is charged once per output buffer, so small
input batches do not multiply that reservation.

Supported builds use non-spilling `CollectLeft` inner joins,
direct-column keys with matching types, and fixed-width or UTF-8 build
columns. Attachment and execution check schema, keys, and null equality,
replaces the unused build subtree with a schema placeholder, and
preserves the handle through supported resets and projections. Prepared
state is process-local and cannot be serialized.

The caller owns snapshot identity, concurrent preparation, and eviction.
Input buffers must survive producer cleanup, and the supplied pool must
cover every consumer. Attach after physical optimization, supply fresh
probe plans and dynamic-filter expressions, and retain a build handle
while any associated filter outlives its plan. Compatibility checks
cannot verify snapshot contents.

## What is the testing strategy for this PR?

Fifteen focused tests cover equivalence with ordinary joins, both
null-equality modes, empty and duplicate keys, batch boundaries,
concurrent consumers with independent filters and predicates, plan
rewrites, public key changes after attachment, admission failures,
small-batch concatenation under a fixed memory limit, cancellation, and
buffer/reservation lifetimes. The latest regressions exercise
under-admission without shrinking the reservation, checked copy/scratch
arithmetic, and IN-list buffer sharing across one/two keys and one/two
input batches. Existing serialization coverage rejects prepared state;
the API example is a doctest. The benchmark checks outputs across 24
cases.

Prepared copy and scratch sizing retain checked arithmetic. A concat
output that exceeds its admitted bound returns an error in every build
profile before shrinking the reservation. Numeric IN-list membership
shares the retained batch's key buffers; charging its full array size
again would duplicate that payload charge. General array/schema metadata
and consumer-local filter expressions are outside the buffer
reservation.

Validation of
[8a964a8](apache@8a964a8)
passed: all 15 focused prepared-build tests in the normal `ci` profile;
`cargo fmt --all`; full workspace Clippy with all targets/features and
`-D warnings`; the complete repository lint suite, including Rust and
HTML documentation builds; and the extended workspace suite (12,159 Rust
tests and all 524 SQL logic files, with 8 Rust tests ignored). All 15
prepared-build tests also passed in the optimized `release-nonlto`
profile.

The public-key-mutation regression was verified to fail before restoring
execution-time validation: the ordinary join returned its two expected
rows while the stale prepared lookup returned none. With the guard
restored, the mutated descriptor is rejected with a planning error.

Local validation used Rust 1.98.1 and temporary source overrides for the
official Arrow 60.0.0 (`ef1fa157`), object_store 0.14.2 (`279572ea`),
and sqlparser 0.63.0 (`85b1a6f2`) releases. Dependency overrides and
generated lockfile changes are not included.

The rebased head has not been benchmarked.

```bash
cargo bench -p datafusion-physical-plan --features test_utils --profile release-nonlto --bench prepared_hash_join
```

## Are there any user-facing changes?

Opt-in Rust APIs and an `EXPLAIN` marker for prepared builds. Ordinary
SQL planning and join selection are unchanged.

Backport provenance:
- Upstream PR: apache#25491
- Upstream commit: acf5c89
- Preserve the upstream prepared-build API, benchmark and original tests.
  The three new Rust files and serialization test are upstream-exact.
- Resolve conflicts against the older fork's existing owner/codec structure.
  Upstream-only neighboring null-mark/ProbeCompletion/RightAnti documentation
  and null-aware parameter context have no counterpart; do not fabricate them.
  The serialization guard uses self.prepared_build before encoding children,
  retaining upstream's rejection and message.
- The immediately following commit contains the older-fork probe-state and
  collector-call adaptations. Additional local tests are deferred to the end.
- The upstream description's tests and benchmark measurements are upstream
  results, not a claim that this original-only intermediate state was tested.

Signed-off-by: discord9 <discord9@163.com>
Use the older fork's execution-local probe counters and NULL-aware atomic
flags instead of upstream-parent fields unavailable here; remove the absent
collector argument. Prepared eligibility remains non-null-aware CollectLeft
INNER only. Preserve the ordinary join's existing semantics.

Document the actual fork ownership and probe state, keep initializer/drop
ordering aligned, and relocate the unchanged apache#25508 test after test imports.
No additional local regression tests are included.

Upstream PR: apache#25491
Upstream commit: acf5c89
This is a local fork adaptation. Compilation will be checked on the final
adapted series; no per-intermediate test-pass claim is made.

Signed-off-by: discord9 <discord9@163.com>
## Why are the changes needed?

### Which issue does this PR close?

Part of apache#10586. This fixes the unsigned string/binary byte-array case;
it does not claim to resolve every Parquet ordering issue. It is also
the correctness prerequisite for apache#24526.

### Rationale for this change

A Parquet scan can skip a file, row group, or page when its statistics
prove that no row can satisfy the filter. If those statistics use a
different comparison order from the query, that proof is invalid:
DataFusion can silently discard a row that should be returned.

For example, suppose a Parquet row group contains `'aé'`, `'az'`, and
`'b'`:

```sql
SELECT s FROM t WHERE s = 'az';
-- Expected: one row containing 'az'
```

Parquet's deprecated byte-array `min`/`max` fields use signed
comparison, whereas Arrow compares strings using unsigned UTF-8 bytes.
The same values therefore have two different orders:

| Comparison | Values in ascending order | Minimum / maximum |
| --- | --- | --- |
| Legacy signed-byte order | `aé < az < b` | `aé` / `b` |
| Arrow's unsigned-byte order | `az < aé < b` | `az` / `b` |

The first UTF-8 byte of `é` is `0xC3`: it sorts before `z`'s `0x7A` as a
signed byte, but after it as an unsigned byte. If DataFusion interprets
the legacy interval `['aé', 'b']` using Arrow's ordering, it sees `'az'
< 'aé'` and can incorrectly skip the row group. Merely checking that
`min <= max` does not help: the two reported endpoints are still in
ascending order.

The newer `min_value`/`max_value` fields and page-index bounds also need
an ordering declaration that the reader understands. A missing or
unknown `column_orders` entry is not enough to justify assuming Arrow's
ordering. The [Parquet statistics
definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift)
and [logical-type ordering
rules](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)
describe these distinctions.

This bug is independently observable with an ordinary equality filter;
it does not require a large `IN` list.

## What changes were proposed in this PR?

### What changes are included in this PR?

The change makes a recognized comparison order a prerequisite for using
unsigned byte-array bounds. At the point where Parquet metadata becomes
Arrow statistics, DataFusion checks whether the column's
physical/logical type and footer establish the expected unsigned order.
It also rejects row-group bounds taken from the deprecated signed-order
fields. When that evidence is missing, min/max is reported as unknown,
so pruning keeps potentially matching data instead of guessing.

The rule is applied at the granularity where the statistics are used. An
unsafe row group prevents DataFusion from claiming a trustworthy bound
for the whole file, but it does not make other row groups' valid bounds
unusable. Page-index bounds are checked against the footer
independently. The same safeguards cover static and runtime row-group
pruning, as well as the inverse predicates used to decide whether every
row already satisfies a filter. Null counts and unrelated columns'
statistics remain available.

The row-group pruning API gains a metadata-aware entry point so callers
can supply the footer needed for this decision. The existing entry point
remains source-compatible and behaves conservatively when that
information is unavailable. Signed logical types such as decimals retain
their existing behavior.

### Are there any user-facing changes?

Queries no longer discard matching data because of these untrustworthy
byte-array bounds. Older files, or files with an unrecognized ordering,
may require more scanning. Modern files with trustworthy bounds retain
min/max pruning. There is no file-format change or breaking public API
change.

## How was this PR tested?

### Are these changes tested?

The regression uses actual serialized Parquet files containing the
example above, with modern, deprecated, missing-order, and unknown-order
metadata. It checks that the matching `az` row survives file, row-group,
runtime, and page pruning, while valid statistics can still eliminate
unrelated data. On unchanged Apache `f1f0449a`, the adapted equality
regression fails because the deprecated-order case loses `az`; the
modern-statistics control passes.

The dedicated Parquet-crate run passed 231 unit tests and four doctests.
Its seven focused ordering tests also cover mixed safe/unsafe row
groups, null counts, fixed-length binary and UUID, signed decimal, and
logical types with undefined ordering. Formatting,
all-targets/all-features Clippy with warnings denied, and
`./dev/rust_lint.sh` passed. The extended workspace run passed 10,666
Rust tests, with eight ignored, and all 503 SQL-logic files.

The existing metadata benchmark was run on Apache `f1f0449a` and this
patch using the same valid modern-footer fixture. Across nine
full-statistics cases there was no material regression; the largest
case, with 256 columns and 128 row groups, measured 1.330 ms before and
1.332 ms after. Both runs used Rust 1.97.0, `release-nonlto`, 20
samples, and separate build directories on an Apple M5 Max.

<details>
<summary>Validation commands</summary>

```sh
cargo test --locked --profile ci -p datafusion-datasource-parquet

cargo bench --locked --profile release-nonlto -p datafusion-datasource-parquet \
  --bench parquet_metadata_statistics -- \
  metadata_full --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot

RUST_BACKTRACE=1 cargo test --locked --profile ci \
  --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \
  --workspace --lib --tests --bins \
  --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```

</details>

Backport provenance:
- Upstream PR: apache#24525
- Upstream commit: 26b40dd
- Upstream self-tests and all additions/removals retained; only unavoidable fork patch context resolved.
- Any fork-specific adaptation is kept in a separate following commit.

Signed-off-by: discord9 <discord9@163.com>
## Why are the changes needed?

### Which issue does this PR close?

Related to apache#8668 and apache#8609; follows apache#24074 and the merged
statistics-ordering fix apache#24525.

### Rationale for this change

Queries often select a sparse set of string identifiers. Parquet min/max
statistics can make these queries much cheaper by ruling out row groups
or pages that cannot contain any requested identifier. For example, this
query asks for 21 IDs, spaced ten apart:

```sql
SET datafusion.execution.parquet.max_in_list_size = 1024;

SELECT *
FROM events
WHERE customer_id IN (
  'id000', 'id010', 'id020', 'id030', 'id040', 'id050', 'id060',
  'id070', 'id080', 'id090', 'id100', 'id110', 'id120', 'id130',
  'id140', 'id150', 'id160', 'id170', 'id180', 'id190', 'id200'
);
```

A row group whose values fall between `id003` and `id007` cannot contain
a match. The default pruning limit is 20, so this list is not eligible
for the `IN` min/max rewrite unless the caller raises the limit. apache#24074
made that limit configurable. With a raised limit, DataFusion can
already reject this row group, but it does so by constructing a growing
expression tree resembling:

```text
(min <= id000 AND id000 <= max)
OR (min <= id010 AND id010 <= max)
OR ...
```

For hundreds or thousands of identifiers, building and evaluating that
tree can become expensive in its own right. Replacing the list with one
enclosing range, `[id000, id200]`, would be cheaper, but would lose the
gaps: that broad range overlaps `[id003, id007]` even though none of the
requested IDs is present there.

The aim is to keep the useful pruning precision of the existing
per-value checks while making large lists cheaper to represent and
evaluate. In the included local microbenchmark, evaluating 1,024 values
against 4,096 intervals falls from 68.8 ms to 0.198 ms. This measures
pruning work only, not end-to-end query speedup.

## What changes were proposed in this PR?

### What changes are included in this PR?

Eligible large string lists are stored as a sorted, deduplicated set of
values inside the pruning predicate, rather than expanded into one
comparison branch per value. For each inclusive statistics interval,
DataFusion finds the first requested value at or after the interval's
minimum, then checks whether that value is also at or before its
maximum.

In the example, the first requested ID at or after `id003` is `id010`.
Since `id010 > id007`, the row group can be skipped. An interval such as
`[id019, id021]` must be kept because it could contain `id020`. This
takes a binary search per interval after sorting the values once per
constructed predicate, and the expression tree no longer grows with the
number of IDs.

The result remains a conservative pruning decision. An overlapping
interval means only that a match is possible; the original `IN`
expression still performs exact row filtering. The original literal
information also remains available to other pruning mechanisms,
including Bloom filters. Inverted or unusable bounds yield UNKNOWN. A
single known bound can still exclude the entire domain; otherwise,
incomplete bounds yield UNKNOWN and keep the container eligible for
reading.

**Limits and scope**

For a nonempty input list of `N` values, before this optimization
deduplicates it, and configured cap `C =
datafusion.execution.parquet.max_in_list_size`:

| Condition | Min/max pruning behavior |
| --- | --- |
| `N <= min(20, C)` | Existing per-value rewrite |
| `20 < N <= C`, positive, non-null literal strings on a direct string
column | Compact sorted-domain lookup |
| `20 < N <= C`, other lists | Existing per-value rewrite, including
non-string lists, `NOT IN`, and NULL-containing lists |
| `N > C` | This `IN` min/max rewrite is skipped; it does not disable
all pruning |

The compact path supports `Utf8`, `LargeUtf8`, `Utf8View`, and
dictionary-encoded string columns. Unsupported expressions keep their
existing handling. The default cap remains 20, and `C = 0` disables this
`IN` min/max rewrite. Other predicates and literal/containment pruning,
including Bloom filters, remain available. Raising the cap can still
produce expensive comparison trees for lists that do not use the compact
path. The lower threshold of 20 preserves the existing expression shape
and limits this PR's scope; it is not a measured performance crossover.

Page-index pruning now receives the same configured cap as row-group
pruning, so raising the limit can benefit both.

An interval search requires bounds in the same comparison order as the
query. The compact path reads the same statistics adapters as the
existing per-value path. The already-merged apache#24525 masks untrusted
Parquet byte-array bounds. For other statistics sources,
`PruningStatistics` requires conservative bounds in Arrow's comparison
order; arbitrary provider bounds are trusted rather than validated by
this expression.

### Are there any user-facing changes?

Users who raise `datafusion.execution.parquet.max_in_list_size` get
cheaper min/max pruning for eligible large string lists, and page
pruning now honors that setting. The configuration default, exact query
results, and existing public APIs are unchanged. This is a focused
optimization for literal string lists, not a general rewrite of every
large `IN` expression.

## How was this PR tested?

### Are these changes tested?

The pruning-crate suite passed 95 tests. The standalone Parquet
regressions use lists of 20, 21, 256, and 1,024 values and check both
exact query results and scan/pruning metrics. They cover gaps inside the
list's enclosing range, row-group pruning, page-only pruning, and the
default and zero-cap controls.

Two correctness regressions exercise the less obvious interactions. A
direct physical-source test uses `NOT IN (..., NULL)`, row-filter
pushdown, and `LIMIT 1`, so logical optimizer folding cannot hide an
incorrect decision to bypass filtering. A real-file test combines this
PR with apache#24525 and verifies that compact page pruning cannot lose a
matching row when the footer's ordering is missing or unknown. Against
unchanged Apache `f1f0449a`, the positive row-group/page tests fail as
expected; the `NOT IN (..., NULL)` control passes.

The benchmark compares the actual raised-cap `IN` path on Apache
`f1f0449a` and this patch, using separate build directories and checking
that both return the same nontrivial pruning results. Local results on
an Apple M5 Max (18 CPUs, 128 GiB), Rust 1.97.0, `release-nonlto`, 20
samples:

| Values | Predicate construction, main → PR | Evaluate 4,096 intervals,
main → PR |
| --- | --- | --- |
| 20 | 31.1 → 27.4 µs | 177.6 → 165.7 µs |
| 21 | 30.5 → 3.52 µs | 183.8 → 98.1 µs |
| 256 | 356.6 → 21.4 µs | 5.53 → 0.144 ms |
| 1,024 | 1.466 → 0.081 ms | 68.8 → 0.198 ms |

A balanced explicit OR tree is included as another comparison: at 1,024
values it takes 8.03 ms to evaluate the same intervals. These
measurements isolate pruning overhead; no end-to-end workload
improvement is claimed.

The latest documentation/comment update passed configuration-doc
generation, formatting, all-targets/all-features Clippy with warnings
denied, and `./dev/rust_lint.sh` (including the workspace Rust
documentation check). Focused validation also passed all 3 Parquet
IN-list integration tests and all 9 Parquet statistics-ordering tests.

The CI follow-up in
[1a2b38c](apache@1a2b38c)
updates the `SHOW ALL VERBOSE` snapshot to match the revised
`max_in_list_size` help text. The stale row caused both the cargo test
and benchmark verification jobs to fail. After the correction, the
targeted `information_schema.slt` test and full extended suite passed:
10,796 Rust tests (8 ignored) and all 505 SQL logic files. Formatting,
Clippy with all targets and features and warnings denied, and
`./dev/rust_lint.sh` also passed.

<details>
<summary>Validation commands</summary>

```sh
cargo test --locked --profile ci -p datafusion-pruning

cargo test --locked --profile ci -p datafusion \
  --test parquet_integration string_in_list_pruning

cargo bench --locked --profile release-nonlto -p datafusion-pruning \
  --bench string_in_list_pruning -- \
  --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot

RUST_BACKTRACE=1 cargo test --locked --profile ci \
  --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \
  --workspace --lib --tests --bins \
  --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```

</details>

Backport provenance:
- Upstream PR: apache#24526
- Upstream commit: 9ad1217
- Upstream self-tests and all additions/removals retained; only unavoidable fork patch context resolved.
- Any fork-specific adaptation is kept in a separate following commit.

Signed-off-by: discord9 <discord9@163.com>
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Part of apache#24711.

## Rationale for this change

Large string NOT IN lists create long expression chains for Parquet
pruning. These expressions are expensive to build and evaluate.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.

Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.

For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->

## What changes are included in this PR?

- Use a compact sorted domain for large, non-null string NOT IN lists.

- Preserve the existing behavior for lists containing NULL.

- Add unit tests and Parquet row-group and page-pruning tests.

- Add a mixed-container regression test.

- Extend the existing benchmark and configuration documentation.

- In a local benchmark with 1,024 values, construction was about 17
times faster and evaluation was about 27 times faster.

<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

Yes

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

No public API changes

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please add the `api
change` label.
-->

Backport provenance:
- Upstream PR: apache#24781
- Upstream commit: eaf7fff
- Upstream self-tests and all additions/removals retained; only unavoidable fork patch context resolved.
- Any fork-specific adaptation is kept in a separate following commit.

Signed-off-by: discord9 <discord9@163.com>
## Which issue does this PR close?

- Closes apache#24711.

## Rationale for this change

apache#24781 added compact pruning for non-NULL `NOT IN` lists. Issue apache#24711
still needs support for NULL-containing lists.

Large string lists can create expensive per-value pruning expressions.
The compact path reduces this cost when `max_in_list_size` permits the
rewrite.

NULL values require special handling. Filter pruning treats FALSE and
UNKNOWN equally, but full-match detection must keep these results
separate.

## What changes are included in this PR?

This PR makes these changes:

- Supports compact pruning for large string `IN` lists that contain
NULL.
- Treats `NOT IN` with NULL as unable to match rows.
- Treats all-NULL `IN` lists as unable to match rows.
- Records whether a pruning predicate is safe for full-match inversion.
- Uses the same configured list-size cap when it builds the inverse
predicate.
- Adds unit tests, Parquet integration tests, documentation, and
benchmarks.

## What is the testing strategy for this PR?

The tests cover positive and negative lists, all-NULL lists, nested `OR`
expressions, configured caps, and Parquet full-match detection.

These commands passed:

```shell
cargo test -p datafusion-pruning
cargo test -p datafusion-datasource-parquet
cargo test -p datafusion --test parquet_integration string_in_list_pruning
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
```

Focused benchmarks also cover compact `IN` and constant-false `NOT IN`
expressions with NULL.

## Are there any user-facing changes?

Large string `IN` and `NOT IN` lists that contain NULL can now use
compact pruning when the cap permits it.

Query results and configuration defaults do not change.

This PR adds two non-breaking `PruningPredicate` accessors:
`max_in_list_size` and `can_be_inverted_for_full_match`.

Backport provenance:
- Upstream PR: apache#24954
- Upstream commit: 33028d5
- Upstream self-tests and all additions/removals retained; only unavoidable fork patch context resolved.
- Any fork-specific adaptation is kept in a separate following commit.

Signed-off-by: discord9 <discord9@163.com>
## Which issue does this PR close?

- Closes apache#24709

## Rationale for this change

Compact pruning for large `IN` lists only supports string columns.

Other ordered types use one min/max comparison for each literal. Large
lists create large expression trees and make pruning more expensive.

These types can use the same compact sorted-domain approach.

## What changes are included in this PR?

This PR adds compact `IN` and `NOT IN` pruning for:

- Variable-length binary types
- Signed and unsigned integers
- Decimal types
- Date and time types
- Timestamp and duration types
- Dictionary-encoded forms of these types

String and binary values share the byte-domain implementation. Primitive
types use typed contiguous arrays.

The compact path preserves NULL behavior and dictionary value NULLs.
Unsupported types keep the existing per-value fallback.

Floating-point types remain unsupported because of NaN and signed zero.
`FixedSizeBinary` and nested types also remain unsupported.

The default `max_in_list_size` value remains 20.

### Benchmarks

With 1,024 literals and 4,096 statistics containers:

- String `IN` evaluation is 37.8x faster.
- String `NOT IN` evaluation is 21.4x faster.
- Int64 `IN` evaluation is 118x faster.
- Int64 `NOT IN` evaluation is 111x faster.
- Predicate construction is 15.9x to 32.1x faster.

Commands:

```shell
cargo bench -p datafusion-pruning --bench string_in_list_pruning
cargo bench -p datafusion-pruning --bench primitive_in_list_pruning
```

## What is the testing strategy for this PR?

The tests cover:

- All supported type families
- Both `IN` and `NOT IN`
- NULL-containing and all-NULL lists
- Dictionary and run-end-encoded values
- Missing, inverted, and truncated statistics
- Parquet row-group and page pruning
- Per-value fallback for unsupported types
- Result equality between compact pruning and the original filter

The following commands passed:

```shell
cargo test -p datafusion-pruning
cargo test -p datafusion --test parquet_integration string_in_list_pruning
cargo test -p datafusion-sqllogictest --test sqllogictests -- information_schema
cargo clippy -p datafusion-pruning -p datafusion --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
```

## Are there any user-facing changes?

Large `IN` and `NOT IN` lists can use compact pruning for more ordered
types.

This behavior applies when
`datafusion.execution.parquet.max_in_list_size` is set above 20. The
default value and SQL results do not change.

This PR does not change public APIs.

Backport provenance:
- Upstream PR: apache#25012
- Upstream commit: f3f420b
- Upstream self-tests and all additions/removals retained; only unavoidable fork patch context resolved.
- Any fork-specific adaptation is kept in a separate following commit.

Signed-off-by: discord9 <discord9@163.com>
@discord9
discord9 force-pushed the fix/complete-join-pruning-batch branch from e25dab8 to 9df12a4 Compare November 6, 2026 13:40
@discord9
discord9 changed the base branch from fix/25531-cast-constraint-propagation to greptimedb-55.1-thin-fork November 6, 2026 13:40
@discord9
discord9 marked this pull request as ready for review November 8, 2026 13:38
haohuaijin and others added 6 commits November 8, 2026 22:11
…he#25500)

## Which issue does this PR close?

Follow-up to apache#25227 and [its
discussion](apache#25227 (comment)).

## Rationale for this change

Some lossless casts unnecessarily discard column statistics or fail to
preserve strict ordering information.

For example, UInt32 → Int64 currently requires both bounds to be exact
to propagate statistics, even though every UInt32 value fits in Int64.
Recognizing this conversion as globally safe preserves available
statistics even when one bound is absent or bounds are inexact.

## What changes are included in this PR?

- Recognize unsigned-to-signed widening casts: UInt8 → Int16 / Int32 /
Int64, UInt16 → Int32 / Int64, and UInt32 → Int64.
- Recognize Utf8 → Utf8View and Binary → LargeBinary / BinaryView.
- Extend the existing `check_bigger_cast` / `is_bigger_cast` predicates
to recognize these lossless conversions.

## What is the testing strategy for this PR?

- Test values, nulls, integer extrema, and strict ordering properties.
- Test empty values, Unicode, long strings, and non-UTF8 binary values.
- Test statistics propagation with exact, inexact, and missing bounds.
- Verify unsafe reverse conversions remain excluded.

Validation on the latest revision: 25 relevant unit tests passed; one
existing test is ignored. Formatting, all-target/all-feature Clippy, and
the standard `rust_lint.sh` suite passed.

## Are there any user-facing changes?

More lossless casts retain statistics and strict ordering information,
enabling existing optimizer rules.

Backport provenance:
- Upstream PR: apache#25500
- Upstream commit: b172f73
- Retain the upstream matcher and its complete self-tests; only the absent fork example-doc context is adjusted.
- Original upstream tests are retained; fork-only adaptation, if any, follows separately.

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 38ea24d)
Use the older fork's two-argument project_column_statistics_through_expr
helper in the upstream regression test. This commit contains only the test
helper adaptation; upstream production logic and tests are in the previous
commit.

Upstream PR: apache#25500
Upstream commit: b172f73

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 5d5cec0)
…25602)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#25291.
- Alternative to apache#25292.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.

Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.

For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->

Avoid scanning redundant files/row-groups/pages in the probe side of
hash joins, based on the values dictated by the build side.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

- wiring up PushdownStrategy::Map/HashTableLookupExpr to carry a
representation of the build side values from a hash join, through
- `KeyRangeBitmap` implementation, which maps the build-side values
array into a finite-sized bitmap bucket, and can answer probing
questions for certain ranges
- extend `build_predicate_expression` to build the associated pruning
expression from `HashTableLookupExpr` using the new
`KeyRangeBitmapPruningExpr`, which implements `PhysicalExpr` on top of
`KeyRangeBitmap`
- also extend `build_predicate_expression` so that it now pushes down
pruning for `CaseExprs`, since that also unlocks the partitioned
hash-join scenario this pr targets

## What is the testing strategy for this PR?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

Briefly describe how this PR is tested, and point to the specific tests
you added. For example: 'This new feature is covered by the
`sqllogictest` cases added in `foo.slt`'.

If this PR does not add tests, explain why. For example, if the change
is already covered by existing tests, please mention it.

You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->

A number of unit tests added, and one SLT added.

Also verified on the query shape that motivated the original issue
```sql
> copy (select i as k, random() as v from generate_series(0, 1999999) t(i))
to '/tmp/fact.parquet'
stored as parquet options ('format.max_row_group_size' '1000');
+---------+
| count   |
+---------+
| 2000000 |
+---------+
1 row(s) fetched.
Elapsed 0.093 seconds.

> create external table fact stored as parquet location '/tmp/fact.parquet';
0 row(s) fetched.
Elapsed 0.008 seconds.

> create table dim as
select i as k from generate_series(0, 1999999) t(i) where i % 10000 < 200;
0 row(s) fetched.
Elapsed 0.014 seconds.

> select count(*), sum(v) from fact join dim on fact.k = dim.k;
select count(*), sum(v) from fact join dim on fact.k = dim.k;
select count(*), sum(v) from fact join dim on fact.k = dim.k;
select count(*), sum(v) from fact join dim on fact.k = dim.k;
+----------+--------------------+
| count(*) | sum(fact.v)        |
+----------+--------------------+
| 40000    | 20121.308621485692 |
+----------+--------------------+
1 row(s) fetched.
Elapsed 0.031 seconds.

+----------+--------------------+
| count(*) | sum(fact.v)        |
+----------+--------------------+
| 40000    | 20121.308621485692 |
+----------+--------------------+
1 row(s) fetched.
Elapsed 0.017 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 20121.30862148569 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 20121.30862148569 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.

> explain analyze select count(*), sum(v) from fact join dim on fact.k = dim.k;
...
|                   |           DataSourceExec: file_groups={12 groups: [[tmp/fact.parquet:0..1927333], [tmp/fact.parquet:1927333..3854666], [tmp/fact.parquet:3854666..5781999], [tmp/fact.parquet:5781999..7709332], [tmp/fact.parquet:7709332..9636665], ...]}, projection=[k, v], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ k@0 >= 0 AND k@0 <= 1990199 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=k_null_count@1 != row_count@2 AND k_max@0 >= 0 AND k_null_count@1 != row_count@2 AND k_min@3 <= 1990199 AND k_null_count@1 != row_count@2 AND KEY_RANGE_BITMAP(k_min@3, k_max@0, 20000/1048576 buckets), required_guarantees=[], metrics=[output_rows=200.0 K, elapsed_compute=9.65ms, output_bytes=25.0 MB, output_batches=200, files_ranges_pruned_statistics=12 total → 12 matched, row_groups_pruned_statistics=2.00 K total → 200 matched, row_groups_pruned_bloom_filter=200 total → 200 matched, page_index_pages_pruned=200 total → 200 matched, page_index_rows_pruned=200.0 K total → 200.0 K matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, bytes_processed=22.1 MB, bytes_scanned=2.1 MB, file_open_errors=0, file_scan_errors=0, files_opened=12, files_processed=12, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_pruned_dynamic_filter=0, predicate_cache_inner_records=0, predicate_cache_records=0, bloom_filter_eval_time=64.23µs, metadata_load_time=345.27µs, page_index_eval_time=1.33ms, row_pushdown_eval_time=36ns, statistics_eval_time=896.06µs, time_elapsed_opening=6.99ms, time_elapsed_processing=24.90ms, time_elapsed_scanning_total=224.99ms, time_elapsed_scanning_until_data=25.01ms, output_rows_skew=1.19%, scan_efficiency_ratio=9.7% (2.24 M/23.13 M)] |
...
```

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>

Backport provenance:
- Upstream PR: apache#25602
- Upstream commit: 4bb9f8d
- Upstream additions/removals and self-tests retained; fork codec/import context is resolved separately.

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit d787bc3)
Local fork correctness changes on top of the twelve upstream backports:
- treat computed NULL/TRY_CAST/CASE results conservatively for full-match
  predicates, and keep the inverse-row-group guard;
- retain dynamic-filter producer reservations through snapshots and
  partitioned remapping;
- make visited-bitmap charges execution-local and carry the accepted
  key-range-bitmap reservation with the derived lookup.

These are local changes, not upstream PR content. Local regression tests are
kept in the following commit.

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 563832c)
Keep the existing local Arrow, Parquet, NULL/full-match, prepared-build and
producer-lifetime coverage. Aggregate producer regressions are carried by
the independent apache#25579 backport, not this JOIN optimization candidate.

Signed-off-by: discord9 <discord9@163.com>
Record the final aggregate/bitmap plan metrics for the existing local
regression, without changing product behavior.

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 9df12a4)
@discord9
discord9 force-pushed the fix/complete-join-pruning-batch branch from 9df12a4 to ca15295 Compare November 8, 2026 14:21
@discord9 discord9 changed the title fix: backport complete join pruning and aggregate NULL-bound recovery perf: backport JOIN dynamic-filter and compact pruning optimizations Nov 8, 2026
@WenyXu WenyXu mentioned this pull request Nov 9, 2026
64 of 69 tasks
The concat copy reservation used ArrayData::get_slice_memory_size for
List/Map/Struct children, which walks the full backing buffer even when
Arrow concat slices child values to the selected offsets. Count only the
range Arrow actually copies, plus the rebuilt offsets/validity, while
leaving view and FixedSizeList behavior unchanged.

Add List, Struct<List>, Map, and ordinary HashJoin regressions with shared
backing arrays under a bounded memory pool.

Signed-off-by: discord9 <discord9@163.com>

@fengjiachun fengjiachun left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a reproducible memory-accounting regression with shared dictionary payloads; details inline.

}
Ok(size)
}
_ => Ok(array.to_data().get_slice_memory_size()?),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we account for shared dictionary values when estimating concat allocations? DictionaryArray reaches this fallback, and get_slice_memory_size() recursively counts its entire values array for every input batch. When the batches share the same dictionary, Arrow 59.2's concat reuses that values buffer and copies only the keys, so this reservation can reject an ordinary hash join that fits in the memory pool.

Reproduced at 17d13cf7f: create a two-row batch with Int32 join keys [1, 100_000] and a Dictionary<Int32, Utf8> payload whose keys are [0, 0] and whose single value is an 8 MiB string; feed its two one-row slices into a CollectLeft INNER join under a 9 MiB GreedyMemoryPool. The join fails requesting an additional 16 MiB with 8 MiB already reserved. A direct concat confirms the output dictionary values are pointer-identical to the input: input buffers total 8,388,688 bytes, the estimated copy is 16,777,248 bytes, and concat retains only 8,388,744 bytes. Please make the estimate reflect dictionary sharing across the input batches and add a bounded-pool regression for this case.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked 90bb0ca7d: the original top-level dictionary reproducer now passes, both through the concat helper and an ordinary hash join. All 451 existing joins::hash_join tests also pass.

There is still an uncovered nested case. shares_dictionary_values() only examines top-level columns, while the Struct, List and FixedSizeList recursion passes false to estimate_concat_allocation. Arrow also preserves shared dictionary values inside those containers, so wrapping the same payload in a container brings back the false resource-exhaustion error.

I reproduced this in three ordinary CollectLeft INNER joins by wrapping the original two-row dictionary in Struct<d: Dictionary<Int32, Utf8>>, List<Dictionary<Int32, Utf8>> (one element per row), or FixedSizeList<Dictionary<Int32, Utf8>, 1>, then passing the same two one-row slices under a 9 MiB pool. All three still request an additional 16 MiB with 8 MiB already reserved and fail. Direct concat retains 8,388,744 bytes for Struct/FixedSizeList and 8,388,760 bytes for List; in every case I verified that the output dictionary values are pointer-identical to the input.

Please detect shared dictionaries recursively across corresponding child arrays, including the selected child ranges, rather than forcing nested dictionaries down the unshared path. A bounded-pool nested-dictionary regression would cover this remaining case.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed at 90bb0ca7d, with the same numbers you measured. A Struct<d: Dictionary<Int32, Utf8>> column whose two one-row inputs are slices of one dictionary holding a single 8 MiB value gives: inputs 8,388,688 B, estimate_concat_allocation sum 16,777,240 B, actual retained after concat 8,388,744 B. shares_dictionary_values(&batches, 1) is false because the top-level column is a Struct, and the Struct/List/FixedSizeList recursion then calls estimate_concat_allocation with false for the nested array, so the Dictionary arm takes keys + full values and the values are charged once per input. Same mechanism your three container repros hit.

Agreed this is a real false ResourcesExhausted, and it is a remaining gap rather than a regression from this commit: before 90bb0ca7d every dictionary took the get_slice_memory_size() fallback, so nested and top-level were equally over-reserved; 90bb0ca7d only fixed the top-level case it can detect.

For the fix, the sharing decision has to travel with the recursion instead of being decided once per top-level column: compute it once per input column, then, at each Struct child / List or Map selected child range / FixedSizeList values, carry down whether the corresponding child arrays of all inputs hold pointer-equal dictionary values (for lists and maps the selected ranges may differ per input, so a range-level check is needed rather than indices), and only then let the Dictionary arm keep keys alone. A nested Struct/List/FixedSizeList bounded-pool regression alongside the existing top-level one would pin it. This is an accounting accuracy issue with no effect on results, so I do not consider it a blocker for this PR either way - your call on whether it lands here or as a follow-up.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked final head 203689a71. The original top-level concat and ordinary-join reproducers, plus all three ordinary-join variants with Struct/List/FixedSizeList-wrapped dictionaries, now pass under the same bounded pools (5/5 independent probes). The existing joins::hash_join suite also passes: 450 passed, 0 failed, including the final-held reservation boundary and extra-charge reconciliation tests.

The reported false ResourcesExhausted issue is addressed. The final change removes the ordinary-concat copy predictor and reconciles the retained batch delta after allocation; it preserves other charges on the reservation and leaves the prepared-build path unchanged. This comes with the explicitly documented tradeoff that ordinary concat can transiently exceed the pool limit before reconciliation. I found no additional blocking issue in this delta. This follow-up validation did not rerun the full SLT suite or workload benchmarks.

Dictionary columns whose values are pointer-shared across every build
input are reused by Arrow concat; reserve only their rebuilt keys.
Detect that sharing per column before reserving. Also recurse through
FixedSizeList values, which Arrow already slices to the selected range,
so nested List backing is not counted once per input.

Add shared-dictionary, ordinary HashJoin, and FixedSizeList<List>
regressions with bounded memory pools.

Signed-off-by: discord9 <discord9@163.com>

@WenyXu WenyXu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I reviewed head 90bb0ca7d against e3221b8f7 and found no correctness defect in the backports or in the fork-local hardening.

Verified

  • Production hunks of apache#24526/apache#24781/apache#24954/apache#25012/apache#25602 are byte-identical to upstream 4bb9f8d31; apache#24525 has only parquet-59.2 API adaptations (INT96 sort order, page-index completeness, one test fixture); apache#24319's memory.rs differs only by the added count_array and tests; apache#25500 only widens check_bigger_cast (the fork set is narrower than upstream head, i.e. more conservative).
  • Local hardening a64b1a6c4: the pre-rewrite TRY_CAST / safe-CAST scan, the CASE-branch nullability check and the inverse-predicate guard only reduce full-match inference, so none can turn UNKNOWN into a proven match. The producer lease survives with_new_children, the pruning-predicate snapshot and partitioned remapping, and the visited-bitmap charge is now execution-local.
  • KeyRangeBitmap: the wrapping_sub(offset) handling for containers entirely below/above the key range, containers straddling min, the hi_off.min(span).max(lo_b) clamp and the deliberately coarse buckets never drop a container that holds a build key; build keys and probe statistics use the same as u64 mapping, and a width/type mismatch degrades to all-NULL (no pruning).
  • Tests: physical-plan joins 1079, hash_join 866, key_range 3, prepared 15; whole datafusion-pruning 118; datafusion-common memory 16 and record_batch 13; statistics_order_tests 10/10; SLT dynamic_row_group_pruning and parquet_filter_pushdown pass. datafusion-datasource-parquet was 210 passed / 16 failed, all of them the uninitialised parquet-testing submodule.

Non-blocking observations

  • exec.rs:2469-2490 (also :2386, :2451): the ordinary-build concat reservation counts a validity bitmap only when an input already has one, but Arrow materialises the bitmap for the whole output column as soon as any input has nulls, so peak reservation can be short by ~ceil(rows/8) per column; the post-concat reconcile fixes only the steady state. prepared.rs:219-234 already compensates for exactly this.
  • exec.rs:2374-2385: the unshared-dictionary estimate covers keys + values, while concat may take merge_dictionary_values, whose bitmaps/interner/indices are not estimated. Both are bounded accounting effects, no behavioural impact.
  • a64b1a6c4, 17d13cf7f and 90bb0ca7d have no upstream counterpart, and upstream has since moved the full-match inversion to datasource-parquet/src/pruning.rs and replaced the pruning cast gate. Worth documenting as fork-local or upstreaming so the next 55.1 sync does not conflict.

Out of scope here (pre-existing in 55.1, fixed upstream by apache#26452 and apache#26982): verify_support_type_for_prune still admits casts that do not keep order, so e.g. WHERE CAST(i AS BOOLEAN) <= false prunes a container holding i = 0. I reproduced it on this head with a throwaway test; it should be a separate backport, not part of this one.

Remove the ordinary-concat pre-allocation peak predictor and its
type-specific recursion. Retain input, final build, hash map, bitmap and
computed-key accounting; after Arrow concatenation, reconcile only the
batch delta against the existing reservation.

This accepts a transient inputs-plus-output peak during ordinary concat:
the pool no longer rejects the copy before it is made, and a physical
allocation may still fail. Prepared-build reservation/reconcile is
unchanged.

Update tests to assert final held memory, preserved extra
reservations, success with shared nested Dictionary payloads, and
post-concat resource failure with RAII release.

Signed-off-by: discord9 <discord9@163.com>
@discord9
discord9 merged commit 8f0e5bb into greptimedb-55.1-thin-fork Nov 10, 2026
discord9 added a commit that referenced this pull request Nov 10, 2026
…on state

#39 introduced `PreparedHashJoinBuild::probe_data`, which builds a fresh
`JoinLeftData` per consuming join. It still created the `probe_threads_counter`
and the two probe-side atomics that the null-aware backports replace with
`ProbeCompletion`, and the `collect_left_input` call it makes predates the two
null-aware flags.

A prepared build is only reachable for ordinary joins: `prepared_key_indices`
requires `INNER`, `CollectLeft` and `!null_aware`, and both `prepare_build` and
`execute` re-check it. The per-join null bitmaps therefore start empty and
`null_aware_build_key_nulls` stays `None`.

The correlation bitmap of a correlated null-aware join is mutable per-execution
state, so it is charged to the same `_visited_reservation` lease as the visited
bitmap rather than to the build reservation. #39 hands that build data out
through `build_lease`, and a dynamic filter can retain it after this join's own
state is dropped; charging the bitmap there would keep paying for memory that is
already released.

`correlated_bitmap_reservation_is_execution_local` pins that down: with the
correlated flag on, dropping `JoinLeftData` returns the pool to the producer's
charge alone, and dropping the retained lease returns it to zero.

Signed-off-by: discord9 <discord9@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants