Skip to content

Decode the decimal scalar of a Rust-written flat node's min/max stats #710

Closed

Description

@dfa1

Split from #583. ArrayStats.decodeScalar decodes a node's stats scalar without the column dtype, so a decimal bound arrives as text. #704 stops that from failing a filtered scan (the bound is ignored), but Java then never prunes on it.

To do: find the exact ScalarValue Rust writes for a decimal stat (look it up in the Rust source, not by probing bytes), pass the column dtype to ArrayStats.fromFbs, decode it to BigDecimal, and cross-check with a file whose decimal column has no zone map.

Affects only pruning of decimal columns without a zone map. vortex-jni writes a zone map even for 100 rows, so the path was not reachable from the small file in DecimalBytePartsInteropIntegrationTest; a non-zoned layout is needed to test it.

Activity

  1. added a commit that references this issue on Nov 12, 2026
    7d26daf
  2. dfa1 commented on Nov 12, 2026

    @dfa1
    OwnerAuthor

    Fixed in #740. Rust writes a decimal bound as BytesValue: the little-endian two's-complement bytes of the unscaled value, 1/2/4/8/16/32 bytes (ScalarValue::to_proto / bytes_from_proto, vortex 0.93.0); Java now decodes it against the column dtype.

    The cross-check this ticket asked for isn't possible with vortex-jni 0.93: a probe on a 100-row decimal column (byte-parts shapes, and p20/p38 plain vortex.decimal) found no min/max on the root node of any file it wrote, so the path is covered by unit tests of Rust's encoding instead. Left as is: the inspector still shows these bounds as text (it has no per-node dtype), and nested decimal leaves have no column dtype to decode against.

  3. added a commit that references this issue on Nov 12, 2026
    8acb012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions