Skip to content

feat(array) support for desc in IsSorted and IsStrictSorted - #11380

Open
joacoc wants to merge 7 commits into
vortex-data:developfrom
joacoc:is-sort-desc
Open

joacoc wants to merge 7 commits into
vortex-data:developfrom
joacoc:is-sort-desc

Conversation

@joacoc

@joacoc joacoc commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds support for desc sorting for both IsSorted and IsStrictSorted.

Once this lands, the next step is to make IsCompoundSorted use these options, so its sort keys can be desc too.

Changes

  • A new sortedness benchmark (no prev. baseline)
  • Rename is_sorted_impl to is_sorted_with_options
  • Try to use IsSortedOptions and is_sorted_with_options wherever possible

API Changes

  • Adds a sort direction field to IsSortedOptions. Breaking changes:
    • Code that builds IsSortedOptions { strict } must now set direction.
    • Older readers reject the new serialized bytes for descending options (2 and 3).
  • Options are still serialized as one byte, but that single byte now also holds the direction.

Serialization

Byte Direction Strict
0 Asc (0) No (0)
1 Asc (0) Yes (1)
2 Desc (1) No (0)
3 Desc (1) Yes (1)

Two more things worth mentioning:

  1. Desc sortedness isn’t cached. The stats cache can’t differentiate between an agg fn’s options. This only affects desc performance, not correctness. Asc remains cacheable.
  2. Claude noticed that non-strict Boolean checks skip nulls. I left a TODO and a test in bool.rs.

Adds support to use `desc` sorts in `IsSorted`.

Options are still serialized as a single byte but
it is a breaking change, because the options will
now require a new additional field for sort direction.

The stats cache only works for `asc`, because
`Stat::IsSorted` and `Stat::IsStrictSorted` only
describe ascending order. `desc` is always computed,
until the stats cache can differentiate options.

Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Applies for both asc/desc. But there was no previous
benchmark to compare as baseline. Anyways,
I did a benchmark locally and the values were very
similar.

Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
There is a pending issue with booleans,
when there are invalid values/nulls, and
sortedness is asked, due to how arrow handles
bitmaks (thats what Claude is suggesting me
still have to dig into BitIndexIterator) the
wrong sortedness is reported.

Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
The sort kernel for dicts supported only
ascending codes and values, so now it also
supports descending values but asc codes.

The natural way for dicts is to build the
codes incrementally one by one. So adding
support for desc codes didn't make much sense.

This commit also fixes some fn calls around
the array crate.

Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Update calls to use the same function as the rest
of the codebase uses for sorting (`is_sorted_with_options`.)

Also added `desc` cases tests for encondings, and
added a table in sequence to understand better how
the multiplier works.

Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
The benchmark included desc codes, which shouldn't be
considerated because there is no such case in Vortex
for that.

Also added tests in the dict sorted kernel as a
recommendation from Fable. Similar in seq. tests,
for unsigned cases.

Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
@joacoc
joacoc marked this pull request as ready for review November 12, 2026 19:17

Copy link
Copy Markdown
Contributor

Null handling in IsSorted (as of 99a83ab)

Summary of how nulls behave after this PR. Nulls sort first in both directions (asc → NULLS FIRST, desc → NULLS FIRST), with one exception for non-strict Booleans.

Within a batch

IsSortedOptions::is_nullable_sorted turns each element into an Option<T> and relies on None < Some(_):

  • Asc: checks the Options ascending, so every null must come before every value.
  • Desc: does not just flip the comparison, which would put nulls last. It maps Some(v) → Some(Reverse(v)) and checks ascending, so values come out descending and nulls stay first.

Primitive ([primitive.rs#L42-L50](https://github.com/vortex-data/vortex/blob/99a83ab6f0100aeda91463fd5872f062ec2d718e/vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs#L42-L50)), Decimal and VarBinView all go through this path. Shortcuts: all-null → !strict; no nulls → direct is_sorted / is_sorted_by(>=) etc.

Strict pre-check

is_sorted_with_options counts nulls first. Two nulls tie, so: 0 nulls → continue; 1 null → only allowed at index 0 (either direction); ≥2 → not strictly sorted.

Between batches

boundary_is_sorted:

previous next result
null null !strict
null value true
value null false
value value <= / < / >= / > by direction and strict

The Boolean exception

The non-strict path in bool.rs#L39-L54 iterates only set_indices(), i.e. it drops nulls instead of putting them first. That gives answers inconsistent with every other type, and with itself across batches:

  • [false, null, true] is sorted as Booleans, but [0, null, 1] as integers is not.
  • [true, null] is sorted as one batch, but split into [true] + [null] it is not, because boundary_is_sorted puts nulls first.

This affects both directions. It's already flagged by the TODO and pinned by non_strict_skips_nulls. The fix looks like making the non-strict branch use is_nullable_sorted like the strict branch (and like primitive/decimal), then flipping that test. It matters for IsCompoundSorted (#10661) too: its first field goes through is_sorted, while later fields use a nulls-first pair mask, so a Boolean key column could pass one path and fail the other.

FoR kernel

No special handling is needed in for_/compute/is_sorted.rs. The unsigned view reuses encoded.validity(), and FoR's validity is the encoded child's, so null positions are identical. Null slots hold zero but are masked out. A couple of nullable desc cases ([null, 5, 1] → sorted, [5, null, 1] → not) would be worth adding to the FoR tests.


Generated by Claude Code

@codspeed

codspeed Bot commented Nov 13, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 15.16%

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

⚡ 4 improved benchmarks
✅ 1917 untouched benchmarks
🆕 18 new benchmarks
⏩ 50 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation chunked_bool_canonical_into[(1000, 10)] 37.7 µs 29.5 µs +27.98%
⚡ Walltime add_shapes_neon[(32768, PerRowPerRow)] 18.8 µs 16.9 µs +11.39%
⚡ Walltime scalar_subtract_neon 13.8 µs 12.4 µs +11.34%
⚡ Walltime add_i64_nullable_neon 19.7 µs 17.7 µs +10.81%
🆕 CPU Simulation dictionary[(Asc, false)] N/A 342 µs N/A
🆕 CPU Simulation dictionary[(Asc, true)] N/A 348.5 µs N/A
🆕 CPU Simulation dictionary[(Desc, false)] N/A 342.3 µs N/A
🆕 CPU Simulation dictionary[(Desc, true)] N/A 348.2 µs N/A
🆕 CPU Simulation nullable_bool_scan[asc] N/A 189.1 µs N/A
🆕 CPU Simulation nullable_bool_scan[desc] N/A 189.1 µs N/A
🆕 CPU Simulation nullable_primitive_scan[(Asc, false)] N/A 430.7 µs N/A
🆕 CPU Simulation nullable_primitive_scan[(Asc, true)] N/A 291.7 µs N/A
🆕 CPU Simulation nullable_primitive_scan[(Desc, false)] N/A 294 µs N/A
🆕 CPU Simulation nullable_primitive_scan[(Desc, true)] N/A 291.6 µs N/A
🆕 CPU Simulation primitive_scan[(Asc, false)] N/A 190.7 µs N/A
🆕 CPU Simulation primitive_scan[(Asc, true)] N/A 190.7 µs N/A
🆕 CPU Simulation primitive_scan[(Desc, false)] N/A 190.7 µs N/A
🆕 CPU Simulation primitive_scan[(Desc, true)] N/A 190.7 µs N/A
🆕 CPU Simulation utf8_scan[(Asc, false)] N/A 1.3 ms N/A
🆕 CPU Simulation utf8_scan[(Asc, true)] N/A 1.3 ms N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing joacoc:is-sort-desc (99a83ab) with develop (d81bd4c)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005

Copy link
Copy Markdown
Contributor

Should we also add NULLS_FIRST and NULLS_LAST options?

@joacoc

joacoc commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@robert3005 that would make total sense to me. It can be an enum or just a bool (same for direction). It would look like this:

pub struct IsSortedOptions {
    pub strict: bool,
    pub direction: SortDirection,
    pub null_order: NullOrder,
}

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)]
pub enum NullOrder {
    /// Nulls come before all values, whatever the direction.
    #[default]
    First,
    /// Nulls come after all values, whatever the direction.
    Last,
}

Serialization would just use one bit per field:

 bit   7  6  5  4  3      2            1           0
     ┌────────────────┬────────────┬───────────┬────────┐
     │ reserved (0)   │ null order │ direction │ strict │
     └────────────────┴────────────┴───────────┴────────┘
                        0 = first    0 = asc     0 = no
                        1 = last     1 = desc    1 = yes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants