Skip to content

perf: speed up non-null dictionary hash scatter - #27025

Open
Rich-T-kid wants to merge 8 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/non-null-dictionary-hash-scatter
Open

Rich-T-kid wants to merge 8 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/non-null-dictionary-hash-scatter

Conversation

@Rich-T-kid

@Rich-T-kid Rich-T-kid commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #26848. This is the non-null path; #26966 handles nullable keys.

Why

For a dictionary with non-null keys and values, scattering hashes overwrites every output row. with_hashes was still zeroing its reusable 8,192-row buffer first, and the scatter loop loaded and stored each row before starting the next key-indexed load.

What changed

with_hashes now keeps the initialized buffer contents when the first column is a non-null dictionary, since that column replaces every slot. The non-null scatter loop gathers eight dictionary hashes before updating the output, which gives the CPU independent loads to overlap.

Nullable keys or values continue through the existing path. Short hash buffers also keep the existing truncation behavior.

Results

Run 1 · Run 2

Skip clearing the reusable hash buffer when a dictionary with non-null keys and values will overwrite every row. Gather dictionary hashes in independent lane groups so the CPU can overlap the key-indexed loads.
@github-actions github-actions Bot added the common Related to common crate label Nov 3, 2026
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6592499441-4263-ftkpf 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.02µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 95.0s
Peak memory 31.5 MiB
Avg memory 5.1 MiB
CPU user 23.7s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 90.0s
Peak memory 29.4 MiB
Avg memory 5.0 MiB
CPU user 23.5s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@adriangbot

Copy link
Copy Markdown

Hi @Rich-T-kid, your benchmark configuration could not be parsed (#27025 (comment)).

Error: invalid configuration: unknown field BENCH_FILTER, expected one of env, baseline, changed, resources, shards at line 2 column 1

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Any benchmark name is accepted: bench.sh suite names (e.g. tpch, clickbench_partitioned, wide_schema) and Criterion bench targets (e.g. sql_planner) are resolved automatically. A name that matches neither fails on the runner.

Per-side configuration (run benchmark tpch followed by):

env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

Pod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal):

resources:
cpu: "16"
memory: "128Gi"
arch: arm64          # arm64 or amd64

Independent benchmark shards (optional; 1–8, default 1):

run benchmark arrow_writer
shards: 4

Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion.


File an issue against this benchmark runner

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6593384459-4264-fb8ds 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.15      3.3±0.02µs        ? ?/sec    1.00      2.8±0.06µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 65.0s
Peak memory 35.3 MiB
Avg memory 7.0 MiB
CPU user 23.4s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 60.0s
Peak memory 29.9 MiB
Avg memory 7.4 MiB
CPU user 23.5s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

Existing hash utility tests cover dictionary hashing; retain a concise comment explaining the lane-gather fast path.
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

@kumarUjjawal could you take a look 🙇

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

2 similar comments
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Rich-T-kid

Left few comments please take a look.

Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6594963661-4285-fbvqw 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (cb54f47) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

Keep the buffer-reuse and scatter dispatch checks in one place, validate the unchecked lookup invariant, and cover chunk, tail, multi-column, and buffer-reuse behavior.
@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (cb54f47) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 65.0s
Peak memory 29.8 MiB
Avg memory 6.7 MiB
CPU user 21.5s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 85.0s
Peak memory 29.3 MiB
Avg memory 5.4 MiB
CPU user 23.6s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6594963493-4286-5r52q 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6594964058-4287-75g47 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

thank you for the in depth review @kumarUjjawal, I addressed all your comments! added the test back in as well if you could take another look.

re-running the benchmakrs as well

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.00µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 65.0s
Peak memory 29.2 MiB
Avg memory 6.6 MiB
CPU user 21.5s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 80.0s
Peak memory 34.4 MiB
Avg memory 5.6 MiB
CPU user 25.8s
CPU sys 0.1s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.09     10.9±0.02µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 75.0s
Peak memory 29.0 MiB
Avg memory 6.1 MiB
CPU user 22.1s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 90.0s
Peak memory 29.3 MiB
Avg memory 4.9 MiB
CPU user 23.3s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6595334800-4301-245tt 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 65.0s
Peak memory 36.6 MiB
Avg memory 7.0 MiB
CPU user 24.8s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 60.0s
Peak memory 29.2 MiB
Avg memory 7.3 MiB
CPU user 23.5s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Restore buffer initialization so the hosted benchmark can isolate the benefit of skipping it from the dictionary scatter changes.
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

Restoring buffer.clear() to measure its impact

Direct A/B: 0a7c4da0c (skip clear) vs. b41d408fd (clear restored).

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
ref: 0a7c4da
changed:
ref: b41d408

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6606243064-4305-xk62z 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing b41d408 (b41d408) to 0a7c4da diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
  ref: "0a7c4da0cdcea319338b4150a28108ad9c0fffad"
changed:
  ref: "b41d408fd729ac0c7397bdc0ac5118063ca534ec"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing b41d408 (b41d408) to 0a7c4da diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
  ref: "0a7c4da0cdcea319338b4150a28108ad9c0fffad"
changed:
  ref: "b41d408fd729ac0c7397bdc0ac5118063ca534ec"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.00     10.0±0.03µs        ? ?/sec    1.04     10.4±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.00      2.8±0.00µs        ? ?/sec    1.16      3.2±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 105.0s
Peak memory 31.7 MiB
Avg memory 4.5 MiB
CPU user 26.1s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 80.0s
Peak memory 27.8 MiB
Avg memory 5.1 MiB
CPU user 20.6s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

Restoring buffer clearing made the focused dictionary benchmarks 1.04x and 1.16x slower, so retain the guarded reuse path.
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

1 similar comment
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6613579983-4344-vlrdt 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6613581773-4345-7l2mw 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.09     10.8±0.02µs        ? ?/sec    1.00      9.9±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.00µs        ? ?/sec    1.00      2.8±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 70.0s
Peak memory 29.6 MiB
Avg memory 6.3 MiB
CPU user 21.2s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 85.0s
Peak memory 29.3 MiB
Avg memory 5.4 MiB
CPU user 23.6s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 70.0s
Peak memory 35.5 MiB
Avg memory 6.6 MiB
CPU user 23.4s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 65.0s
Peak memory 31.9 MiB
Avg memory 7.1 MiB
CPU user 24.9s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@Rich-T-kid
Rich-T-kid marked this pull request as draft November 8, 2026 19:19
@Rich-T-kid
Rich-T-kid marked this pull request as ready for review November 12, 2026 17:48
@Rich-T-kid

Rich-T-kid commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

this is ready for review again @kumarUjjawal 🙇

Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Comment thread datafusion/common/src/hash_utils.rs Resolved
Use one physical-null predicate for buffer reuse and scatter dispatch, keep collision builds conservative, enforce the unchecked gather bounds in release builds, and cover sliced u8/u16/i64 dictionaries.
@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6858853791-4962-bnnhl 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.09     10.9±0.04µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.3±0.00µs        ? ?/sec    1.00      2.8±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 90.0s
Peak memory 29.0 MiB
Avg memory 4.9 MiB
CPU user 21.6s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 85.0s
Peak memory 28.8 MiB
Avg memory 5.4 MiB
CPU user 23.6s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

show benchmark queue

@adriangbot

Copy link
Copy Markdown

Hi @Rich-T-kid, you asked to view the benchmark queue (#27025 (comment)).

Comment Repo PR User Benchmarks Resources Status
#6858430099 apache/arrow-rs #11973 Rich-T-kid ["zip_kernels"] default running
#6858431183 apache/arrow-rs #11973 Rich-T-kid ["merge_kernels"] default running
#6859107075 apache/arrow-rs #11973 Rich-T-kid ["interleave_kernels"] default running
#6859110127 apache/arrow-rs #11973 Rich-T-kid ["zip_kernels"] default running
#6859109055 apache/arrow-rs #11973 Rich-T-kid ["filter_kernels"] default running
#6859110975 apache/arrow-rs #11973 Rich-T-kid ["merge_kernels"] default pending
#6859299632 apache/datafusion #27025 Rich-T-kid ["with_hashes"] default pending
#6859512678 apache/arrow-rs #11973 Rich-T-kid ["concatenate_kernel"] default pending
#6859596521 apache/arrow-rs #11973 Rich-T-kid ["mutable_array"] default pending
#6859852734 apache/arrow-rs #12218 Rich-T-kid ["coalesce_kernels"] default pending
#6859851870 apache/arrow-rs #12218 Rich-T-kid ["coalesce_kernels"] amd64 pending

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6859299632-4970-rzclf 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (1150cfc) to 4c7b307 (merge-base) diff

Run configuration
run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.9±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.3±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric Value
Wall time 65.0s
Peak memory 28.1 MiB
Avg memory 6.5 MiB
CPU user 21.7s
CPU sys 0.0s
Peak spill 0 B

with_hashes — branch

Metric Value
Wall time 60.0s
Peak memory 28.7 MiB
Avg memory 7.1 MiB
CPU user 23.4s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

array.values().len(),
"dictionary hashes must cover every dictionary value"
);
assert!(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The no-null dictionary fast path now uses assert!(hashes.len() <= keys.len()), which differs both from the dictionary null paths in the same function (min(len) at line 669, zip at line 700) and from every other hasher (assert_eq!).

let (key_chunks, key_tail) = keys.as_chunks::<LANES>();
let (hash_chunks, hash_tail) = hashes.as_chunks_mut::<LANES>();
for (keys, hashes) in key_chunks.iter().zip(hash_chunks) {
let gathered: [u64; LANES] = std::array::from_fn(|lane| unsafe {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scatter_non_null_dictionary is a safe fn that reads dict_hashes.get_unchecked(key.as_usize()) with no check on the key. Its SAFETY comment relies on dictionary keys being in bounds, which only holds for validated arrays. Also, the unsafe block wraps the whole closure, including the safe keys[lane] index.

buffer.clear();
// A dictionary with non-null keys and values overwrites every output
// slot, so retain the initialized storage instead of zeroing it first.
let reuse_buffer = iter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reuse gate checks nulls on the dictionary before compaction, but the scatter runs on the compacted dictionary. A dictionary whose only null values are never referenced by a key still gets cleared, even though the fast path would overwrite every slot.


#[cfg(not(feature = "force_hash_collisions"))]
#[inline(always)]
fn update_dictionary_hash<const MULTI_COL: bool>(hash: &mut u64, dict_hash: u64) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update_dictionary_hash repeats the MULTI_COL combine logic that already exists inline in scatter_row in hash_dictionary_scatter (lines 658-662).

// same length as dict_hashes.
*dict_hashes.get_unchecked(key.as_usize())
});
if MULTI_COL {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MULTI_COL branch of the 8-lane loop calls update_dictionary_hash::<MULTI_COL>, which branches on MULTI_COL again. The if MULTI_COL { .. } else { *hashes = gathered } split plus the helper adds layers without changing what the code does.


#[test]
#[cfg(feature = "force_hash_collisions")]
fn test_force_hash_collisions_clears_dictionary_buffer() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_force_hash_collisions_clears_dictionary_buffer only checks that a cfg stub hard-coded to false returns false. It never calls with_hashes, so it tests nothing about clearing the buffer.

Ok(())
})
.unwrap();
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assert_matches test of the clearing path for null dictionaries never dirties the thread-local buffer itself. It only catches a missing clear() because earlier check_non_null_dictionary_hash_fast_path calls happened to leave stale data in the buffer.

buffer.clear();
// A dictionary with non-null keys and values overwrites every output
// slot, so retain the initialized storage instead of zeroing it first.
let reuse_buffer = iter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with_hashespeeks the iterator a second time inside the TLS closure to computereuse_buffer. It could be computed in the same match iter.peek()that already readsrequired_size`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants