Repository navigation
Description
Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.
What Rust does
FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).
Observed (same 20k-row data through both writers, all stats Exact)
| root encoding (Rust) | Rust writes | Java writes |
|---|---|---|
vortex.primitive, fastlanes.bitpacked |
min, max, is_constant=false, null_count | min, max, null_count |
vortex.sequence |
is_sorted, is_strict_sorted | min, max, null_count |
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant |
nothing | min, max, null_count |
vortex.bool |
is_constant | min, max, null_count |
| child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) | often stats (sorted flags, min/max/sum/null_count) | never: Java writes stats on the root only |
Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.
Plan
- Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
- Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.
Expected consequences (intended, per the parity directive)
- Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So
VortexReader.columnStats()string min/max and the SQLMIN/MAXstats shortcut go away for them, as already happens for Rust-written files.JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds(and the emoji test above it) then expectnullmin/max. - Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
- Numeric
columnStatsare unaffected, because they come from the zone table. VortexWriter.buildArrayFlatBuffer's comment saysnull_countis always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.
The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.
Activity
fix(writer): the root node of a flat segment carries the array stats …
fix(writer): the child nodes of a flat segment carry the array stats …
refactor(writer): narrow the node-stats API and cover the child stats…
fix(writer): the zone table of a column has Rust's shape: no validity…
fix(writer): the zone table of a column has Rust's shape: no validity…
fix(writer): text is a vortex.varbinview unless a scheme is smaller, …
fix(writer): text is a vortex.varbinview unless a scheme is smaller, …
dfa1 commentedon Nov 6, 2026
Status after #673, #674, #675, #676, #677:
vortex.varbinviewwith its node stats.is_sorted=true(and int stats) on varbin / FSST offsets children, and FSST lengths shaped as a sequence; both depend on the trained FSST symbol table, which is not ported byte for byte (the zone table of a column whose few bounds Rust's table compresses still differs).CascadingCompressor.nodeStatsrescans every cascaded integer child; Rust reuses the stats cached on the array. NeedsArrayAndStats(or its min/max) threaded throughencodeArray/ChildSlot.fix(fsst): train the symbol table as Rust's fsst-rs does (#583)
fix(fsst): train the symbol table as Rust's fsst-rs does (#583)
fix(reader): columnStats reports decimal min/max, and decimal columns…
fix(reader): columnStats reports decimal min/max, and decimal columns…
perf(writer): a node's integer min/max reuse the stats the scheme com…
dfa1 commentedon Nov 8, 2026
Status on #583
Done:
vortex.sequence(fix(fsst): train the symbol table as Rust's fsst-rs does (#583) #698). The training is a port of fsst-rs. For few-distinct, no-null columns the encoding tree now matches Rust's.columnStatsand pruning (fix(reader): columnStats reports decimal min/max; decimals compare by value (#583) #695).Known divergence: the FSST symbol table can't be byte-identical to Rust's. Ties between equal-gain candidates are broken by hash-map iteration order, which depends on the platform's SIMD width. Parity is by symbol set instead: about 90% overlap, and the code stream is within about 1%.
Still open:
""rows. Fixing this needs validity inside the encodings, tracked in Writer: string columns written as per-chunk dicts where Rust uses its dict layout (up to 2x larger) #650.ArrayStats.decodeScalar.fix(reader): a decimal flat-node bound read as text no longer fails a…
dfa1 commentedon Nov 8, 2026
Closing: the plan is done. Stats inheritance is ported for every node of the tree, and
NodeStatsInteropIntegrationTestcompares every flat root, and the whole tree of the numeric columns, against vortex-jni. The zone-table shape, text asvortex.varbinview, FSST training as fsst-rs, decimal bounds and the single min/max pass are merged.What remains is split out: