Repository navigation
Conversation
Adds support to use `desc` sorts in `IsSorted`. Options are still serialized as a single byte but it is a breaking change, because the options will now require a new additional field for sort direction. The stats cache only works for `asc`, because `Stat::IsSorted` and `Stat::IsStrictSorted` only describe ascending order. `desc` is always computed, until the stats cache can differentiate options. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Applies for both asc/desc. But there was no previous benchmark to compare as baseline. Anyways, I did a benchmark locally and the values were very similar. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
There is a pending issue with booleans, when there are invalid values/nulls, and sortedness is asked, due to how arrow handles bitmaks (thats what Claude is suggesting me still have to dig into BitIndexIterator) the wrong sortedness is reported. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
The sort kernel for dicts supported only ascending codes and values, so now it also supports descending values but asc codes. The natural way for dicts is to build the codes incrementally one by one. So adding support for desc codes didn't make much sense. This commit also fixes some fn calls around the array crate. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Update calls to use the same function as the rest of the codebase uses for sorting (`is_sorted_with_options`.) Also added `desc` cases tests for encondings, and added a table in sequence to understand better how the multiplier works. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
The benchmark included desc codes, which shouldn't be considerated because there is no such case in Vortex for that. Also added tests in the dict sorted kernel as a recommendation from Fable. Similar in seq. tests, for unsigned cases. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Null handling in
|
| previous | next | result |
|---|---|---|
| null | null | !strict |
| null | value | true |
| value | null | false |
| value | value | <= / < / >= / > by direction and strict |
The Boolean exception
The non-strict path in bool.rs#L39-L54 iterates only set_indices(), i.e. it drops nulls instead of putting them first. That gives answers inconsistent with every other type, and with itself across batches:
[false, null, true]is sorted as Booleans, but[0, null, 1]as integers is not.[true, null]is sorted as one batch, but split into[true]+[null]it is not, becauseboundary_is_sortedputs nulls first.
This affects both directions. It's already flagged by the TODO and pinned by non_strict_skips_nulls. The fix looks like making the non-strict branch use is_nullable_sorted like the strict branch (and like primitive/decimal), then flipping that test. It matters for IsCompoundSorted (#10661) too: its first field goes through is_sorted, while later fields use a nulls-first pair mask, so a Boolean key column could pass one path and fail the other.
FoR kernel
No special handling is needed in for_/compute/is_sorted.rs. The unsigned view reuses encoded.validity(), and FoR's validity is the encoded child's, so null positions are identical. Null slots hold zero but are masked out. A couple of nullable desc cases ([null, 5, 1] → sorted, [5, null, 1] → not) would be worth adding to the FoR tests.
Generated by Claude Code
Merging this PR will improve performance by 15.16%3 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
|
Should we also add NULLS_FIRST and NULLS_LAST options? |
|
@robert3005 that would make total sense to me. It can be an enum or just a bool (same for pub struct IsSortedOptions {
pub strict: bool,
pub direction: SortDirection,
pub null_order: NullOrder,
}
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)]
pub enum NullOrder {
/// Nulls come before all values, whatever the direction.
#[default]
First,
/// Nulls come after all values, whatever the direction.
Last,
}Serialization would just use one bit per field: |
Summary
This PR adds support for
descsorting for bothIsSortedandIsStrictSorted.Once this lands, the next step is to make
IsCompoundSorteduse these options, so its sort keys can be desc too.Changes
is_sorted_impltois_sorted_with_optionsIsSortedOptionsandis_sorted_with_optionswherever possibleAPI Changes
directionfield toIsSortedOptions. Breaking changes:IsSortedOptions { strict }must now setdirection.2and3).Serialization
0123Two more things worth mentioning: