Skip to content

Canonical Decimal encoding #662

Open

Description

@dfa1

Following upstream ticket here:
vortex-data/vortex#10701

This has a nice problem description:
vortex-data/vortex#5820

Activity

  1. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Status and plan (2026-11-08)

    Checked against upstream #10701 / #10733 and the Rust develop source.

    Already at parity

    • Writer: DecimalEncodingEncoder writes every decimal at the canonical width of its precision (DType.Decimal#byteWidth), the rule upstream is moving to.
    • Strict width decoding (upstream FX1, Decode stored DecimalType and PType values strictly vortex-data/vortex#10755): DecimalEncodingDecoder already rejects an unknown values_type.
    • Byte parts: DecimalBytePartsEncodingEncoder matches Rust's current compressor (narrowed_decimal: the narrowest signed integer that holds the values).

    Blocked on upstream

    Doable now: Rust 0.92+ vortex.decimal read rules (vortex-data/vortex#10995, decimal/vtable/plugin/stored.rs)

    Java differs in four places:

    1. Buffer length: Rust requires exactly rows × width bytes; Java only rejects a buffer that is too short.
    2. Validity child: on a non-nullable node it is an error in Rust (fixed_width::deserialize_validity); Java silently ignores it.
    3. Value range on legacy nodes: when the stored width is not the canonical width of the precision, Rust scans the valid values (WidenArray::from_stored_decimal → smallest_decimal_dtype) and fails if one is outside the declared precision. Canonical-width nodes are trusted and not scanned.
    4. Alignment: the buffer must be aligned to the value width (DecimalData::validate).

    Plan (one PR at a time, straight to main)

    1. Bump vortex-jni 0.90.0 → 0.93.0 on its own, and fix or record whatever interop tests it breaks.
    2. Make DecimalEncodingDecoder match the four rules above, with vortex-jni cross-checks: hand-built malformed nodes, plus legacy narrow-width files that must still read.
    3. Keep this issue open for vortex.widen / DecimalByteParts v2 and follow upstream's preview edition.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions