Skip to content

fix(array): treat bool nulls as sorting first in non-strict is_sorted - #11437

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/bool-is-sorted-nulls-first
Nov 13, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
ji/bool-is-sorted-nulls-first

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

IsSorted orders nulls first. The non-strict branch of the bool kernel (check_bool_sorted) broke that rule: it walked only the indices of valid values, so it dropped nulls instead of ordering them first. The strict branch already handled nulls correctly. It maps each element to Option, where None < Some(_), which is what the primitive kernel does in both modes.

Two counterexamples showed the bug before this fix:

  1. BoolArray [false, null, true] reported sorted, but the equivalent PrimitiveArray [0, null, 1] reported not sorted.
  2. BoolArray [true, null] reported sorted, but the same values split into the ChunkedArray chunks [true] and [null] reported not sorted. The accumulator's check at the chunk boundary already sees a valid value followed by a null.

The correct answer in both cases is not sorted.

Changes

  • vortex-array/src/aggregate_fn/fns/is_sorted/bool.rs: the strict and non-strict checks now share one Option-mapped iterator. There is no is_nullable_sorted helper in this tree, so this uses the same validity-to-Option mapping the strict branch already used.

  • Regression tests in is_sorted/mod.rs:

    • bool_nulls_sort_first_like_primitive checks bools against the equivalent i32 array, in strict and non-strict mode, with the null in the middle, first and last.
    • bool_valid_then_null_matches_chunked checks that flat [true, null] and chunked [true] + [null] both report not sorted.

    Without the fix, the null-in-the-middle case, the null-last case and the chunked test fail. With the fix, all of them pass.

Relation to #10661 (IsCompoundSorted)

#10661 defines its order as ASC with nulls first, and says it uses the same order as IsSorted. Within a batch, check_chunk_sortedness sends the first key field (and any later field where every pair is still tied) to is_sorted(field, ctx) to use the per-encoding fast paths. Its own pairwise path marks "valid value followed by null" as a violation. Batch boundaries are checked separately by compare_sort_keys, which also orders nulls first.

Before this fix, a nullable bool key column with a null after a valid value inside one batch would pass the is_sorted fast path. That would make IsCompoundSorted report sorted, which contradicts its own documented null rule. This fix makes the fast path follow #10661's semantics, so #10661 needs no code change. A follow-up there could add a nullable-bool key case (for example [false, null, true] as the primary key), which was the uncovered gap. I have not pushed anything to #10661's branch.

Checks

  • cargo test -p vortex-array: all pass (5914 lib tests, plus the other test targets)
  • cargo clippy -p vortex-array --all-targets --all-features -- -D warnings: clean
  • cargo +nightly-2026-09-10 fmt -p vortex-array --check: clean

Generated by Claude Code

The non-strict branch of `check_bool_sorted` iterated only the valid
values, so a null after a valid value was ignored. `[false, null, true]`
reported sorted while the equivalent `[0, null, 1]` integer array did
not, and `[true, null]` disagreed with the same values split into the
chunks `[true]` and `[null]`.

Map each element to `Option` for both strict and non-strict checks, as
the primitive kernel does, so nulls sort first for bools too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDABbwGVk3Grb9vVti
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed

codspeed Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 12.04%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
✅ 1935 untouched benchmarks
⏩ 50 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime decode_chunked_neon[i64, 524288] 15.5 µs 13.8 µs +12.04%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/bool-is-sorted-nulls-first (046d13d) with develop (25f4349)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs joseph-isaacs added changelog/feature A new feature changelog/fix A bug fix and removed changelog/feature A new feature labels Nov 13, 2026
@joseph-isaacs
joseph-isaacs merged commit 6480845 into develop Nov 13, 2026
135 of 141 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/bool-is-sorted-nulls-first branch November 13, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants