Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Fixed
- The array stats on a flat segment's root node are the ones Rust keeps: integers under `vortex.primitive` and `fastlanes.bitpacked` carry min, max, `is_constant`, `null_count` and the uncompressed size, and every other encoding carries none, so a Java-written string column reports no flat `min`/`max`, as a Rust-written one. ([#583](https://github.com/dfa1/vortex-java/issues/583))
- The compact preset writes text columns as zstd frames of 8192 values, as Rust does, so a row-range scan decompresses only the frames it overlaps. ([#636](https://github.com/dfa1/vortex-java/issues/636))
- A nullable decimal column keeps its validity in its own encoding, or has none when every row is valid, instead of a `vortex.masked` wrapper, and a column of equal decimals is one constant, as Rust writes them: decimal zone-stats tables are smaller. ([#646](https://github.com/dfa1/vortex-java/issues/646))
- `VortexAggregates` summarizes a `U64` column exactly: `MIN`, `MAX` and `SUM` compare and add unsigned instead of failing past 2^63. ([#665](https://github.com/dfa1/vortex-java/issues/665))
Expand Down
20 changes: 20 additions & 0 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,26 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.zstd_buffers` | `ZstdBuffersEncodingDecoder` | — | ✅ | ❌ | Read only (`zstd2026.02.0`, opt-in in Rust): each buffer of any wrapped encoding is Zstd-decompressed at its declared alignment, then the inner encoding decodes with its own children. Needs the optional zstd binding, like `vortex.zstd`. Interop-tested against a Rust-written fixture (`scripts/fixtures/zstd-buffers`) |
| `vortex.onpair` | `OnPairEncodingDecoder` | `OnPairEncodingEncoder` | ✅ | ✅ | Utf8, Binary; `core2026.08.1`, a cascade candidate under the default edition (competes with FSST, as in Rust) |

### Array stats on flat segments

A flat segment's `ArrayNode` tree carries array stats, and Rust's flat layout does not compute them: it keeps
the ones the compressor had cached on the array, and each encoding passes on only some
(#583). The writer mirrors that for the **root node**:

| Column | Root encoding | Stats written |
|---|---|---|
| integer | `vortex.primitive`, `fastlanes.bitpacked` | `min`, `max`, `is_constant` (only without nulls), `null_count`, `uncompressed_size_in_bytes` |
| float | `vortex.primitive` | `is_constant` (only without nulls), `null_count`, `uncompressed_size_in_bytes`; no bounds |
| bool | `vortex.bool` | `is_constant`, `uncompressed_size_in_bytes` |
| any | any other encoding (`vortex.alp`, `vortex.alprd`, `vortex.fsst`, `vortex.onpair`, `vortex.constant`, `fastlanes.for`, `vortex.dict`, strings, ...) | none |

So a Java-written string column has no bound in its flat segments, as a Rust-written one has none:
`VortexReader.columnStats()` reports no string `min`/`max` for either, and a string bound lives only in the
zoned layout's stats table. The decimal bounds the reader can read come from the same two places, and the
zone table's are not decoded yet. Child nodes carry no stats yet, and `vortex.sequence` does not write its sorted
flags yet ([#583](https://github.com/dfa1/vortex-java/issues/583)); `NodeStatsInteropIntegrationTest` compares
every flat root against vortex-jni.

### Decode shape

Per [ADR 0010](../adr/0010-lazy-decode.md) and [ADR 0012](../adr/0012-zero-copy-layout-decoding.md), each
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import io.github.dfa1.vortex.core.fbs.FbsArray;
import io.github.dfa1.vortex.core.fbs.FbsArrayNode;
import io.github.dfa1.vortex.core.fbs.FbsArrayStats;
import io.github.dfa1.vortex.core.fbs.FbsPrecision;
import io.github.dfa1.vortex.core.model.EncodingId;
import io.github.dfa1.vortex.reader.SegmentSpec;
import io.github.dfa1.vortex.reader.VortexReader;
Expand All @@ -25,16 +27,26 @@ private EncodingTrees() {
/// layout around a single chunk is left out: Java keeps one where Rust writes the flat layout
/// directly, and that layout shape is not what the encoding comparisons test.
static List<String> of(Path file) throws IOException {
return of(file, false);
}

/// [#of(Path)] with the array stats each node carries after its id, as `[min max null_count ..]`
/// naming the stats present and their precision: what Rust and Java write into a flat segment.
static List<String> withStats(Path file) throws IOException {
return of(file, true);
}

private static List<String> of(Path file, boolean stats) throws IOException {
List<String> out = new ArrayList<>();
try (VortexReader reader = VortexReader.open(file)) {
layoutTree(reader, reader.layout(), 0, out);
layoutTree(reader, reader.layout(), 0, out, stats);
}
return out;
}

private static void layoutTree(VortexReader reader, Layout layout, int depth, List<String> out) {
private static void layoutTree(VortexReader reader, Layout layout, int depth, List<String> out, boolean stats) {
if (layout.isChunked() && layout.children().size() == 1) {
layoutTree(reader, layout.children().getFirst(), depth, out);
layoutTree(reader, layout.children().getFirst(), depth, out, stats);
return;
}
out.add(" ".repeat(depth) + layout.layoutId().id());
Expand All @@ -43,18 +55,64 @@ private static void layoutTree(VortexReader reader, Layout layout, int depth, Li
MemorySegment seg = reader.rawSegment(spec);
int fbLen = seg.get(LE_INT, seg.byteSize() - 4L);
FbsArray array = FbsArray.getRootAsFbsArray(seg.asSlice(seg.byteSize() - 4L - fbLen, fbLen));
arrayTree(array.root(), reader.footer().arraySpecs(), depth + 1, out);
arrayTree(array.root(), reader.footer().arraySpecs(), depth + 1, out, stats);
return;
}
for (Layout child : layout.children()) {
layoutTree(reader, child, depth + 1, out);
layoutTree(reader, child, depth + 1, out, stats);
}
}

private static void arrayTree(FbsArrayNode node, List<EncodingId> specs, int depth, List<String> out) {
out.add(" ".repeat(depth) + specs.get(node.encoding()).id());
private static void arrayTree(FbsArrayNode node, List<EncodingId> specs, int depth, List<String> out,
boolean stats) {
String line = " ".repeat(depth) + specs.get(node.encoding()).id();
if (stats) {
line += " " + statNames(node.stats());
}
out.add(line);
for (int i = 0; i < node.childrenLength(); i++) {
arrayTree(node.children(i), specs, depth + 1, out);
arrayTree(node.children(i), specs, depth + 1, out, stats);
}
}

/// The stats a node records, by Rust's names, with their values: bounds as hex of the scalar
/// bytes (`~` after the name when inexact), flags and counts as numbers.
private static String statNames(FbsArrayStats stats) {
if (stats == null) {
return "[]";
}
List<String> names = new ArrayList<>();
if (stats.hasMin()) {
names.add("min" + (stats.minPrecision() == FbsPrecision.Exact ? "" : "~") + "=" + hex(stats.minAsSegment()));
}
if (stats.hasMax()) {
names.add("max" + (stats.maxPrecision() == FbsPrecision.Exact ? "" : "~") + "=" + hex(stats.maxAsSegment()));
}
if (stats.hasSum()) {
names.add("sum=" + hex(stats.sumAsSegment()));
}
if (stats.hasIsConstant()) {
names.add("is_constant=" + stats.isConstant());
}
if (stats.hasIsSorted()) {
names.add("is_sorted=" + stats.isSorted());
}
if (stats.hasIsStrictSorted()) {
names.add("is_strict_sorted=" + stats.isStrictSorted());
}
if (stats.hasNullCount()) {
names.add("null_count=" + stats.nullCount());
}
if (stats.hasNanCount()) {
names.add("nan_count=" + stats.nanCount());
}
if (stats.hasUncompressedSizeInBytes()) {
names.add("uncompressed_size=" + stats.uncompressedSizeInBytes());
}
return names.toString();
}

private static String hex(java.lang.foreign.MemorySegment segment) {
return java.util.HexFormat.of().formatHex(segment.toArray(java.lang.foreign.ValueLayout.JAVA_BYTE));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -918,8 +918,8 @@ static Stream<Arguments> stringEncoders() {
@MethodSource("stringEncoders")
void javaWriter_jniReader_utf8ChunkStats_inUtf8ByteOrder(EncodingEncoder encoder, @TempDir Path tmp)
throws IOException {
// Given — each string encoder writes the chunk's min/max, and with zone maps off those are
// the only bounds in the file. The emoji is a surrogate pair: below U+E000 in UTF-16, above
// Given — a string column's bounds live in the zone map (Rust keeps none in a flat segment's
// array stats), and a filtered read prunes against them. The emoji is a surrogate pair: below U+E000 in UTF-16, above
// it in UTF-8, the order Rust compares in, so a UTF-16 max (U+E000) would let a reader prune
// the chunk holding it. The shared-prefix ids made the FSST table keep two symbols in one of
// Rust's hash slots, and vortex-jni aborted the JVM rebuilding it to filter.
Expand All @@ -932,23 +932,16 @@ void javaWriter_jniReader_utf8ChunkStats_inUtf8ByteOrder(EncodingEncoder encoder
keys[500] = "\uE000";
Path file = tmp.resolve("java_utf8_chunk_stats.vtx");
try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
var sut = VortexWriter.create(ch, STRING_SCHEMA, WriteOptions.defaults().withZoneMaps(false),
List.of(encoder))) {
var sut = VortexWriter.create(ch, STRING_SCHEMA, WriteOptions.defaults(), List.of(encoder))) {
sut.writeChunk(Map.of(ColumnName.of("s"), keys));
}

// When
List<Object> result = readColumnFiltered(file, "s", Expression.binary(Expression.BinaryOp.EQ,
Expression.column("s"), Expression.literal(emoji)));
io.github.dfa1.vortex.reader.ArrayStats stats;
try (var reader = io.github.dfa1.vortex.reader.VortexReader.open(file)) {
stats = reader.columnStats().get(ColumnName.of("s"));
}

// Then
assertThat(result).map(Object::toString).containsExactly(emoji);
assertThat(stats.min()).isEqualTo("EVT0000000000");
assertThat(stats.max()).isEqualTo(emoji);
}

@Test
Expand Down Expand Up @@ -980,11 +973,12 @@ void javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds(@TempDir Path tmp
stats = reader.columnStats().get(ColumnName.of("k"));
}

// Then — vortex-jni accepts the flat stats' precision fields, and the column's min/max are
// the full keys: only the zone map truncates, so SQL MIN/MAX may answer from these stats.
// Then — the matching row survives the truncated zone bounds. As in a Rust-written file, no
// flat array stats hold a string bound, and the zone map's are truncated, so the column
// reports no exact min/max.
assertThat(result).extracting(Object::toString).containsExactly(target);
assertThat(stats.min()).isEqualTo(prefix + String.format("%08d", 0));
assertThat(stats.max()).isEqualTo(prefix + String.format("%08d", 3 * batch - 1));
assertThat(stats.min()).isNull();
assertThat(stats.max()).isNull();
}

@Test
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
package io.github.dfa1.vortex.integration;

import dev.vortex.api.Session;
import dev.vortex.api.VortexWriter;
import dev.vortex.arrow.ArrowAllocation;
import dev.vortex.jni.NativeLoader;
import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.core.model.DTypes;
import io.github.dfa1.vortex.writer.WriteOptions;
import io.github.dfa1.vortex.writer.encode.NullableData;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.BigIntVector;
import org.apache.arrow.vector.BitVector;
import org.apache.arrow.vector.Float8Vector;
import org.apache.arrow.vector.IntVector;
import org.apache.arrow.vector.VarCharVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;

import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.List;
import java.util.Map;
import java.util.Random;

import static org.assertj.core.api.Assertions.assertThat;

/// The array stats on the root node of every flat segment, against what vortex-jni writes for the same
/// data (#583). Rust's flat layout keeps only the stats the compressor had cached on the array, and each
/// encoding passes on some of them: integers keep min, max, `is_constant`, `null_count` and the
/// uncompressed size under `vortex.primitive` and `fastlanes.bitpacked`; floats and strings keep none
/// under ALP, FSST or onpair. `vortex.sequence` is compared by encoding only: its sorted flags are not
/// written yet.
class NodeStatsInteropIntegrationTest {

private static final Session SESSION = Session.create();
private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator();
private static final int N = 20_000;

static {
NativeLoader.loadJni();
}

@Test
void javaWritesTheRootStatsJniWrites(@TempDir Path tmp) throws Exception {
// Given — the same rows through both writers: sorted, random and small-range integers, floats,
// low- and high-cardinality strings, bools, and a nullable integer
Random r = new Random(7);
long[] sorted = new long[N];
long[] random = new long[N];
int[] small = new int[N];
double[] rf = new double[N];
double[] sf = new double[N];
String[] low = new String[N];
String[] high = new String[N];
boolean[] bool = new boolean[N];
int[] nullable = new int[N];
boolean[] valid = new boolean[N];
for (int i = 0; i < N; i++) {
sorted[i] = i;
random[i] = r.nextLong();
small[i] = r.nextInt(100);
rf[i] = r.nextDouble() * 1000;
sf[i] = i * 0.5;
low[i] = "k" + r.nextInt(10);
high[i] = "value-" + Long.toString(r.nextLong() & 0xffffffffL, 36);
bool[i] = r.nextBoolean();
valid[i] = i % 7 != 0;
nullable[i] = valid[i] ? r.nextInt(5_000) : 0;
}
Schema arrow = new Schema(List.of(
Field.notNullable("sorted", new ArrowType.Int(64, true)), Field.notNullable("random", new ArrowType.Int(64, true)),
Field.notNullable("small", new ArrowType.Int(32, true)),
Field.notNullable("rf", new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE)),
Field.notNullable("sf", new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE)),
Field.notNullable("low", ArrowType.Utf8.INSTANCE), Field.notNullable("high", ArrowType.Utf8.INSTANCE),
Field.notNullable("bool", ArrowType.Bool.INSTANCE),
Field.nullable("nullable", new ArrowType.Int(32, true))));
Path jni = tmp.resolve("jni.vortex");
try (VortexWriter w = VortexWriter.builder(SESSION, jni.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build();
VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR)) {
BigIntVector a = (BigIntVector) root.getVector("sorted");
BigIntVector b = (BigIntVector) root.getVector("random");
IntVector c = (IntVector) root.getVector("small");
Float8Vector d = (Float8Vector) root.getVector("rf");
Float8Vector e = (Float8Vector) root.getVector("sf");
VarCharVector f = (VarCharVector) root.getVector("low");
VarCharVector g = (VarCharVector) root.getVector("high");
BitVector h = (BitVector) root.getVector("bool");
IntVector nv = (IntVector) root.getVector("nullable");
root.allocateNew();
for (int i = 0; i < N; i++) {
a.setSafe(i, sorted[i]);
b.setSafe(i, random[i]);
c.setSafe(i, small[i]);
d.setSafe(i, rf[i]);
e.setSafe(i, sf[i]);
f.setSafe(i, low[i].getBytes(StandardCharsets.UTF_8));
g.setSafe(i, high[i].getBytes(StandardCharsets.UTF_8));
h.setSafe(i, bool[i] ? 1 : 0);
if (valid[i]) {
nv.setSafe(i, nullable[i]);
} else {
nv.setNull(i);
}
}
root.setRowCount(N);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema as = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, as);
w.writeBatch(array.memoryAddress(), as.memoryAddress());
}
}
Path java = tmp.resolve("java.vortex");
DType.Struct schema = new DType.Struct(
List.of("sorted", "random", "small", "rf", "sf", "low", "high", "bool", "nullable").stream().map(ColumnName::of).toList(),
List.of(DTypes.I64, DTypes.I64, DTypes.I32, DTypes.F64, DTypes.F64, DTypes.UTF8, DTypes.UTF8, DTypes.BOOL, DTypes.I32.asNullable()), false);
try (FileChannel ch = FileChannel.open(java, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
var w = io.github.dfa1.vortex.writer.VortexWriter.create(ch, schema, WriteOptions.defaults())) {
w.writeChunk(Map.of(ColumnName.of("sorted"), sorted, ColumnName.of("random"), random,
ColumnName.of("small"), small, ColumnName.of("rf"), rf, ColumnName.of("sf"), sf,
ColumnName.of("low"), low, ColumnName.of("high"), high, ColumnName.of("bool"), bool,
ColumnName.of("nullable"), new NullableData(nullable, valid)));
}
List<String> expected = flatRoots(EncodingTrees.withStats(jni));

// When
List<String> result = flatRoots(EncodingTrees.withStats(java));

// Then
assertThat(result).isEqualTo(expected);
}

/// The root node line of every flat segment, in file order; a sequence's stats are blanked.
private static List<String> flatRoots(List<String> tree) {
List<String> roots = new java.util.ArrayList<>();
for (int i = 0; i + 1 < tree.size(); i++) {
if (tree.get(i).strip().equals("vortex.flat")) {
String root = tree.get(i + 1).strip();
roots.add(root.startsWith("vortex.sequence") ? "vortex.sequence" : root);
}
}
return roots;
}
}
Loading
Loading