Skip to content

Writer: flat array stats should match the ones Rust keeps per encoding node #583

Closed

Description

@dfa1

Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.

What Rust does

FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).

Observed (same 20k-row data through both writers, all stats Exact)

root encoding (Rust) Rust writes Java writes
vortex.primitive, fastlanes.bitpacked min, max, is_constant=false, null_count min, max, null_count
vortex.sequence is_sorted, is_strict_sorted min, max, null_count
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant nothing min, max, null_count
vortex.bool is_constant min, max, null_count
child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) often stats (sorted flags, min/max/sum/null_count) never: Java writes stats on the root only

Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.

Plan

  1. Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
  2. Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.

Expected consequences (intended, per the parity directive)

  • Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So VortexReader.columnStats() string min/max and the SQL MIN/MAX stats shortcut go away for them, as already happens for Rust-written files. JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds (and the emoji test above it) then expect null min/max.
  • Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
  • Numeric columnStats are unaffected, because they come from the zone table.
  • VortexWriter.buildArrayFlatBuffer's comment says null_count is always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.

The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.

Activity

  1. dfa1 commented on Nov 6, 2026

    @dfa1
    OwnerAuthor

    Status after #673, #674, #675, #676, #677:

    • Done: root and child node stats, zone-table shape (no validity on columns without nulls, constant bool by size), Utf8 canonical vortex.varbinview with its node stats.
    • Remaining:
      • is_sorted=true (and int stats) on varbin / FSST offsets children, and FSST lengths shaped as a sequence; both depend on the trained FSST symbol table, which is not ported byte for byte (the zone table of a column whose few bounds Rust's table compresses still differs).
      • Second min/max pass: CascadingCompressor.nodeStats rescans every cascaded integer child; Rust reuses the stats cached on the array. Needs ArrayAndStats (or its min/max) threaded through encodeArray/ChildSlot.
      • Interop coverage for nullable decimal validity children.
      • Reader: decode decimal and string bounds from the zone table.
  2. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Status on #583

    Done:

    Known divergence: the FSST symbol table can't be byte-identical to Rust's. Ties between equal-gain candidates are broken by hash-map iteration order, which depends on the platform's SIMD width. Parity is by symbol set instead: about 90% overlap, and the code stream is within about 1%.

    Still open:

  3. dfa1 commented on Nov 8, 2026

    @dfa1
    OwnerAuthor

    Closing: the plan is done. Stats inheritance is ported for every node of the tree, and NodeStatsInteropIntegrationTest compares every flat root, and the whole tree of the numeric columns, against vortex-jni. The zone-table shape, text as vortex.varbinview, FSST training as fsst-rs, decimal bounds and the single min/max pass are merged.

    What remains is split out:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions