Repository navigation
fix(array): treat bool nulls as sorting first in non-strict is_sorted - #11437
Conversation
The non-strict branch of `check_bool_sorted` iterated only the valid values, so a null after a valid value was ignored. `[false, null, true]` reported sorted while the equivalent `[0, null, 1]` integer array did not, and `[true, null]` disagreed with the same values split into the chunks `[true]` and `[null]`. Map each element to `Option` for both strict and non-strict checks, as the primitive kernel does, so nulls sort first for bools too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDABbwGVk3Grb9vVti Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 12.04%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Summary
IsSortedorders nulls first. The non-strict branch of the bool kernel (check_bool_sorted) broke that rule: it walked only the indices of valid values, so it dropped nulls instead of ordering them first. The strict branch already handled nulls correctly. It maps each element toOption, whereNone < Some(_), which is what the primitive kernel does in both modes.Two counterexamples showed the bug before this fix:
BoolArray [false, null, true]reported sorted, but the equivalentPrimitiveArray [0, null, 1]reported not sorted.BoolArray [true, null]reported sorted, but the same values split into theChunkedArraychunks[true]and[null]reported not sorted. The accumulator's check at the chunk boundary already sees a valid value followed by a null.The correct answer in both cases is not sorted.
Changes
vortex-array/src/aggregate_fn/fns/is_sorted/bool.rs: the strict and non-strict checks now share oneOption-mapped iterator. There is nois_nullable_sortedhelper in this tree, so this uses the same validity-to-Optionmapping the strict branch already used.Regression tests in
is_sorted/mod.rs:bool_nulls_sort_first_like_primitivechecks bools against the equivalenti32array, in strict and non-strict mode, with the null in the middle, first and last.bool_valid_then_null_matches_chunkedchecks that flat[true, null]and chunked[true] + [null]both report not sorted.Without the fix, the null-in-the-middle case, the null-last case and the chunked test fail. With the fix, all of them pass.
Relation to #10661 (
IsCompoundSorted)#10661 defines its order as ASC with nulls first, and says it uses the same order as
IsSorted. Within a batch,check_chunk_sortednesssends the first key field (and any later field where every pair is still tied) tois_sorted(field, ctx)to use the per-encoding fast paths. Its own pairwise path marks "valid value followed by null" as a violation. Batch boundaries are checked separately bycompare_sort_keys, which also orders nulls first.Before this fix, a nullable bool key column with a null after a valid value inside one batch would pass the
is_sortedfast path. That would makeIsCompoundSortedreport sorted, which contradicts its own documented null rule. This fix makes the fast path follow #10661's semantics, so #10661 needs no code change. A follow-up there could add a nullable-bool key case (for example[false, null, true]as the primary key), which was the uncovered gap. I have not pushed anything to #10661's branch.Checks
cargo test -p vortex-array: all pass (5914 lib tests, plus the other test targets)cargo clippy -p vortex-array --all-targets --all-features -- -D warnings: cleancargo +nightly-2026-09-10 fmt -p vortex-array --check: cleanGenerated by Claude Code