Repository navigation
perf: speed up non-null dictionary hash scatter - #27025
Rich-T-kid wants to merge 8 commits into
Conversation
Skip clearing the reusable hash buffer when a dictionary with non-null keys and values will overwrite every row. Gather dictionary hashes in independent lane groups so the CPU can overlap the key-indexed loads.
|
run benchmark with_hashes |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
run benchmark with_hashes |
|
Hi @Rich-T-kid, your benchmark configuration could not be parsed (#27025 (comment)). Error: Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
run benchmark with_hashes |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
Existing hash utility tests cover dictionary hashing; retain a concise comment explaining the lane-gather fast path.
|
@kumarUjjawal could you take a look 🙇 |
|
run benchmark with_hashes |
2 similar comments
|
run benchmark with_hashes |
|
run benchmark with_hashes |
kumarUjjawal
left a comment
There was a problem hiding this comment.
Thank you @Rich-T-kid
Left few comments please take a look.
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (cb54f47) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
Keep the buffer-reuse and scatter dispatch checks in one place, validate the unchecked lookup invariant, and cover chunk, tail, multi-column, and buffer-reuse behavior.
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (cb54f47) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
run benchmark with_hashes |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
thank you for the in depth review @kumarUjjawal, I addressed all your comments! added the test back in as well if you could take another look. re-running the benchmakrs as well |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
Restore buffer initialization so the hosted benchmark can isolate the benefit of skipping it from the dictionary scatter changes.
Restoring
|
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing b41d408 (b41d408) to 0a7c4da diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
ref: "0a7c4da0cdcea319338b4150a28108ad9c0fffad"
changed:
ref: "b41d408fd729ac0c7397bdc0ac5118063ca534ec"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing b41d408 (b41d408) to 0a7c4da diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
ref: "0a7c4da0cdcea319338b4150a28108ad9c0fffad"
changed:
ref: "b41d408fd729ac0c7397bdc0ac5118063ca534ec"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
Restoring buffer clearing made the focused dictionary benchmarks 1.04x and 1.16x slower, so retain the guarded reuse path.
|
run benchmark with_hashes |
1 similar comment
|
run benchmark with_hashes |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
this is ready for review again @kumarUjjawal 🙇 |
Use one physical-null predicate for buffer reuse and scatter dispatch, keep collision builds conservative, enforce the unchecked gather bounds in release builds, and cover sliced u8/u16/i64 dictionaries.
|
run benchmark with_hashes |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
|
run benchmark with_hashes |
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#27025 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff Run configurationrun benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"CPU Details (lscpu)Details
Resource Usagewith_hashes — base (merge-base)
with_hashes — branch
File an issue against this benchmark runner |
| array.values().len(), | ||
| "dictionary hashes must cover every dictionary value" | ||
| ); | ||
| assert!( |
There was a problem hiding this comment.
The no-null dictionary fast path now uses assert!(hashes.len() <= keys.len()), which differs both from the dictionary null paths in the same function (min(len) at line 669, zip at line 700) and from every other hasher (assert_eq!).
| let (key_chunks, key_tail) = keys.as_chunks::<LANES>(); | ||
| let (hash_chunks, hash_tail) = hashes.as_chunks_mut::<LANES>(); | ||
| for (keys, hashes) in key_chunks.iter().zip(hash_chunks) { | ||
| let gathered: [u64; LANES] = std::array::from_fn(|lane| unsafe { |
There was a problem hiding this comment.
scatter_non_null_dictionary is a safe fn that reads dict_hashes.get_unchecked(key.as_usize()) with no check on the key. Its SAFETY comment relies on dictionary keys being in bounds, which only holds for validated arrays. Also, the unsafe block wraps the whole closure, including the safe keys[lane] index.
| buffer.clear(); | ||
| // A dictionary with non-null keys and values overwrites every output | ||
| // slot, so retain the initialized storage instead of zeroing it first. | ||
| let reuse_buffer = iter |
There was a problem hiding this comment.
The reuse gate checks nulls on the dictionary before compaction, but the scatter runs on the compacted dictionary. A dictionary whose only null values are never referenced by a key still gets cleared, even though the fast path would overwrite every slot.
|
|
||
| #[cfg(not(feature = "force_hash_collisions"))] | ||
| #[inline(always)] | ||
| fn update_dictionary_hash<const MULTI_COL: bool>(hash: &mut u64, dict_hash: u64) { |
There was a problem hiding this comment.
update_dictionary_hash repeats the MULTI_COL combine logic that already exists inline in scatter_row in hash_dictionary_scatter (lines 658-662).
| // same length as dict_hashes. | ||
| *dict_hashes.get_unchecked(key.as_usize()) | ||
| }); | ||
| if MULTI_COL { |
There was a problem hiding this comment.
The MULTI_COL branch of the 8-lane loop calls update_dictionary_hash::<MULTI_COL>, which branches on MULTI_COL again. The if MULTI_COL { .. } else { *hashes = gathered } split plus the helper adds layers without changing what the code does.
|
|
||
| #[test] | ||
| #[cfg(feature = "force_hash_collisions")] | ||
| fn test_force_hash_collisions_clears_dictionary_buffer() { |
There was a problem hiding this comment.
test_force_hash_collisions_clears_dictionary_buffer only checks that a cfg stub hard-coded to false returns false. It never calls with_hashes, so it tests nothing about clearing the buffer.
| Ok(()) | ||
| }) | ||
| .unwrap(); | ||
| }; |
There was a problem hiding this comment.
The assert_matches test of the clearing path for null dictionaries never dirties the thread-local buffer itself. It only catches a missing clear() because earlier check_non_null_dictionary_hash_fast_path calls happened to leave stale data in the buffer.
| buffer.clear(); | ||
| // A dictionary with non-null keys and values overwrites every output | ||
| // slot, so retain the initialized storage instead of zeroing it first. | ||
| let reuse_buffer = iter |
There was a problem hiding this comment.
with_hashespeeks the iterator a second time inside the TLS closure to computereuse_buffer. It could be computed in the same match iter.peek()that already readsrequired_size`.
Which issue does this PR close?
Part of #26848. This is the non-null path; #26966 handles nullable keys.
Why
For a dictionary with non-null keys and values, scattering hashes overwrites every output row.
with_hasheswas still zeroing its reusable 8,192-row buffer first, and the scatter loop loaded and stored each row before starting the next key-indexed load.What changed
with_hashesnow keeps the initialized buffer contents when the first column is a non-null dictionary, since that column replaces every slot. The non-null scatter loop gathers eight dictionary hashes before updating the output, which gives the CPU independent loads to overlap.Nullable keys or values continue through the existing path. Short hash buffers also keep the existing truncation behavior.
Results
Run 1 · Run 2