Skip to content

Bound decimal storage by precision - #10305

Closed
joseph-isaacs wants to merge 13 commits into
developfrom
ji/decimal-storage-contract
Closed

joseph-isaacs wants to merge 13 commits into
developfrom
ji/decimal-storage-contract

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Decimal arrays currently permit physical storage wider than the smallest type required by their precision. Establish values_type <= DecimalType::smallest_decimal_value_type(decimal_dtype) as the storage contract so compute kernels can rely on this upper bound.

  • Assert the bound in the shared unchecked handle constructor and validate it when assembling array parts.
  • Reject oversized storage in the checked constructors (try_new, try_new_handle) rather than narrowing it; storage within the bound is accepted without copying.
  • Add DecimalData::narrow_to_precision and DecimalArray::try_new_narrowed for importers that receive wider storage than the precision requires. Arrow decimal imports, legacy serialized arrays, and DecimalByteParts assembly narrow explicitly through them; device buffers cannot be narrowed and are rejected. The GPU byte-parts kernel validates its storage instead of constructing unchecked.
  • Keep constant materialization, fill-null, and explicit storage upcasts within the bound.
  • Adapt tests across the workspace (array, arrow, row, byte-parts, sparse, fuzz, CUDA) that built decimals with storage wider than their precision allows, and add regression coverage for precision boundaries, rejection of oversized storage, explicit narrowing (including null payloads and empty arrays), device rejection, legacy decoding, unchecked assertions, and fill-null.

Stack 1/2: based on develop. The dispatch macros and kernel migrations are in #10314, based on this PR.

Validation: cargo clippy -p vortex-array -p vortex-arrow --all-targets --all-features -- -D warnings, cargo fmt --all --check with the pinned nightly, and the tests of vortex-array, vortex-arrow, vortex-row, vortex-sparse, vortex-fuzz and vortex-decimal-byte-parts pass locally with the CI profile. CI is green on the current head, including the CUDA, sanitizer and miri jobs.

Compatibility: constructing a decimal array with storage wider than its precision requires is now an error (a panic from the infallible new constructors). Arrow imports of low-precision Decimal128/Decimal256 columns allocate to narrow their storage. Legacy files with oversized decimal storage still decode, with narrowing on the host; on the GPU they are rejected. As before, constructors require non-null values to fit their declared precision; this change does not add per-value precision validation.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants