Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- `EncodeContext` has a new last component, `trace`, so code calling its canonical constructor passes `EncodeTrace.NONE`. ([#601](https://github.com/dfa1/vortex-java/issues/601))

### Fixed
- A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650))

### Added
- JDK Flight Recorder events for file open, HTTP range requests, segment decode and encode, chunk filtering, writer back-pressure and the compressor's scheme choices, which mirror Rust's compressor trace. ([#601](https://github.com/dfa1/vortex-java/issues/601))

Expand Down
2 changes: 1 addition & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
| `vortex.map` | `MapEncodingDecoder` | `MapEncodingEncoder` | ✅ | ✅ | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately |
| `vortex.fixed_size_list` | `FixedSizeListEncodingDecoder` | `FixedSizeListEncodingEncoder` | ✅ | ✅ | |
| `vortex.zstd` | `ZstdEncodingDecoder` | `ZstdEncodingEncoder` | ✅ | ✅ | Primitive, Utf8, Binary |
| `vortex.masked` | `MaskedEncodingDecoder` | `MaskedEncodingEncoder` | ✅ | ✅ | NullableData carrier. A nullable primitive or decimal column is not masked when cascading: as in Rust, its validity is a plain `vortex.bool` child of the encoding that stores the values (`vortex.primitive`, `fastlanes.bitpacked`, the run values of `vortex.runend`, the indices of `fastlanes.rle`, the codes of `vortex.dict`, the left parts of `vortex.alprd`, `vortex.pco` over the valid values only), passed down by `fastlanes.for`, `vortex.zigzag` and `vortex.alp` (which patches only valid exceptions), and one whose valid rows hold one value is a `vortex.constant` masked by a plain bitmap. Other nullable columns: inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it |
| `vortex.masked` | `MaskedEncodingDecoder` | `MaskedEncodingEncoder` | ✅ | ✅ | NullableData carrier. A nullable primitive or decimal column is not masked when cascading: as in Rust, its validity is a plain `vortex.bool` child of the encoding that stores the values (`vortex.primitive`, `fastlanes.bitpacked`, the run values of `vortex.runend`, the indices of `fastlanes.rle`, the codes of `vortex.dict`, the left parts of `vortex.alprd`, `vortex.pco` over the valid values only), passed down by `fastlanes.for`, `vortex.zigzag` and `vortex.alp` (which patches only valid exceptions), and one whose valid rows hold one value is a `vortex.constant` masked by a plain bitmap. Other nullable columns: inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 or Binary column without a null is not masked when cascading (including under a `ColumnEncoding` override): it has no validity at all, as Rust's all-valid string array has ([#650](https://github.com/dfa1/vortex-java/issues/650)); at cascade depth 0 it is still masked. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it |
| `vortex.decimal` | `DecimalEncodingDecoder` | `DecimalEncodingEncoder` | ✅ | ✅ | Precision 1–76 (i8 … i256 storage by precision), negative scales. Written from `BigDecimal[]`. A nullable column keeps its validity as an optional `vortex.bool` child, not a `vortex.masked` wrapper ([#646](https://github.com/dfa1/vortex-java/issues/646)) |
| `vortex.decimal_byte_parts` | `DecimalBytePartsEncodingDecoder`| `DecimalBytePartsEncodingEncoder`| ✅ | ✅ | Write: one mantissa, no low parts, so a decimal is split only when every unscaled value fits a long (any precision, as Rust's `narrowed_decimal` finds from the data). As Rust's `DecimalScheme`, the mantissa is narrowed to the smallest of i8–i64 holding the values and cascaded (frame of reference, bit-packing, …); a chunk of fewer than two values stays `vortex.decimal`, as Rust leaves it un-narrowed. The byte parts are kept only when strictly smaller than the plain decimal, and a column of equal values is one `vortex.constant`. A nullable column keeps its validity on the mantissa child, or has none when every row is valid, rather than in a `vortex.masked` wrapper ([#646](https://github.com/dfa1/vortex-java/issues/646)) |
| `vortex.datetimeparts` | `DateTimePartsEncodingDecoder` | `DateTimePartsEncodingEncoder` | ✅ | ✅ | |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
package io.github.dfa1.vortex.integration;

import dev.vortex.api.DataSource;
import dev.vortex.api.ImmutableScanOptions;
import dev.vortex.api.Expression;
import dev.vortex.api.Partition;
import dev.vortex.api.Scan;
import dev.vortex.api.ScanOptions;
import dev.vortex.api.Session;
import dev.vortex.api.VortexWriter;
import dev.vortex.arrow.ArrowAllocation;
import dev.vortex.jni.NativeLoader;
import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.core.model.DTypes;
import io.github.dfa1.vortex.writer.WriteOptions;
import io.github.dfa1.vortex.writer.encode.NullableData;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.FieldVector;
import org.apache.arrow.vector.VarCharVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;

import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Random;
import java.util.function.IntPredicate;
import java.util.function.IntFunction;
import java.util.stream.Stream;

import static org.assertj.core.api.Assertions.assertThat;

/// Each column goes through vortex-jni and vortex-java, and the two files must hold the same
/// encoding tree, data and zone stats table alike. Each shape steers the cascade to a different
/// winner.
/// A nullable text column without a null is written without a validity, as Rust writes an all-valid
/// string array (#650), instead of under `vortex.masked`. Columns with nulls are still masked, and
/// differ from Rust in the text scheme itself (#650), so the encoding tree is compared with
/// vortex-jni's for the no-null shape only; every shape is read back by Java and by vortex-jni,
/// full, filtered and row-ranged, compact files included.
class NullableTextInteropIntegrationTest {

private static final Session SESSION = Session.create();
private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator();
private static final ColumnName V = ColumnName.of("v");
private static final int ROWS = 4 * 8192;
private static final long[] RANDOM = new Random(7).longs(ROWS).toArray();

static {
NativeLoader.loadJni();
}

static Stream<Arguments> shapes() {
IntPredicate every7 = i -> i % 7 == 0;
IntFunction<String> few = i -> "value-" + Math.floorMod(RANDOM[i], 10);
IntFunction<String> many = i -> "some longer text value number " + Math.floorMod(RANDOM[i], 5000) + " end";
IntFunction<String> unique = i -> "row-" + RANDOM[i];
return Stream.of(
shape("few distinct, no nulls", few, _ -> false),
shape("many distinct, no nulls", many, _ -> false),
shape("unique, no nulls", unique, _ -> false),
shape("few distinct", few, every7),
shape("many distinct", many, every7),
shape("unique", unique, every7),
shape("one null", few, i -> i == 5),
shape("95% null", few, i -> Math.floorMod(RANDOM[i] >>> 20, 100) < 95),
shape("runs of nulls", few, i -> (i / 500) % 3 == 0));
}

/// The shapes whose whole tree is Rust's. The others still differ beyond the validity, in the
/// scheme chosen for the text itself (#650: the probe applies to numbers only; FSST's lengths).
static Stream<Arguments> treeShapes() {
return shapes().filter(a -> a.get()[0].equals("unique, no nulls"));
}

@ParameterizedTest(name = "{0}")
@MethodSource("treeShapes")
void javaWritesTheEncodingTreeJniWrites(String name, IntFunction<String> values, IntPredicate isNull,
@TempDir Path tmp) throws Exception {
// Given
Path jniFile = tmp.resolve("jni.vortex");
Path javaFile = tmp.resolve("java.vortex");
writeWithJni(jniFile, values, isNull);
List<String> expected = EncodingTrees.of(jniFile);

// When
writeWithJava(javaFile, values, isNull, WriteOptions.defaults());

// Then
assertThat(EncodingTrees.of(javaFile)).isEqualTo(expected);
}

@ParameterizedTest(name = "{0}")
@MethodSource("shapes")
void jniReadsJavaFile(String name, IntFunction<String> values, IntPredicate isNull, @TempDir Path tmp)
throws Exception {
assertJniReads(values, isNull, WriteOptions.defaults(), tmp);
}

/// Compact writes add Zstd; vortex-jni cannot write them, so only its read of Java's file checks them.
@ParameterizedTest(name = "{0}")
@MethodSource("shapes")
void jniReadsJavaCompactFile(String name, IntFunction<String> values, IntPredicate isNull, @TempDir Path tmp)
throws Exception {
assertJniReads(values, isNull, WriteOptions.defaults().withCompact(true), tmp);
}

/// Columns without a null: those are the plain values a Java read returns as strings.
static Stream<Arguments> noNullShapes() {
return shapes().filter(a -> a.get()[0].toString().endsWith("no nulls"));
}

@ParameterizedTest(name = "{0}")
@MethodSource("noNullShapes")
void javaReadsItsOwnFile(String name, IntFunction<String> values, IntPredicate isNull, @TempDir Path tmp)
throws Exception {
// Given
Path file = tmp.resolve("java.vortex");
writeWithJava(file, values, isNull, WriteOptions.defaults());

// When
List<String> result = readWithJava(file);

// Then
List<String> expected = new ArrayList<>();
for (int i = 0; i < ROWS; i++) {
expected.add(values.apply(i));
}
assertThat(result).isEqualTo(expected);
}

private static void assertJniReads(IntFunction<String> values, IntPredicate isNull, WriteOptions options,
Path tmp) throws Exception {
// Given
Path file = tmp.resolve("java.vortex");
writeWithJava(file, values, isNull, options);
List<String> expected = new ArrayList<>();
for (int i = 0; i < ROWS; i++) {
expected.add(isNull.test(i) ? null : values.apply(i));
}
String threshold = values.apply(ROWS / 2);

// When
List<String> result = readWithJni(file, ScanOptions.builder());
// a filtered and a row-range read prune and slice, which a full scan never does
List<String> filtered = readWithJni(file, ScanOptions.builder().filter(Expression.binary(
Expression.BinaryOp.EQ, Expression.column("v"), Expression.literal(threshold))));
List<String> range = readWithJni(file, ScanOptions.builder().rowRangeBegin(1000).rowRangeEnd(20_000));

// Then
assertThat(result).isEqualTo(expected);
assertThat(filtered).isEqualTo(expected.stream().filter(threshold::equals).toList());
assertThat(range).isEqualTo(expected.subList(1000, 20_000));
}

private static Arguments shape(String name, IntFunction<String> values, IntPredicate isNull) {
return Arguments.of(name, values, isNull);
}

private static void writeWithJni(Path path, IntFunction<String> values, IntPredicate isNull) throws Exception {
Schema schema = new Schema(List.of(Field.nullable("v", new ArrowType.Utf8())));
try (VortexWriter writer = VortexWriter.builder(SESSION, path.toAbsolutePath().toUri().toString(), schema,
ALLOCATOR).build();
VectorSchemaRoot root = VectorSchemaRoot.create(schema, ALLOCATOR)) {
VarCharVector vector = (VarCharVector) root.getVector("v");
vector.allocateNew();
for (int i = 0; i < ROWS; i++) {
if (isNull.test(i)) {
vector.setNull(i);
} else {
vector.setSafe(i, values.apply(i).getBytes(StandardCharsets.UTF_8));
}
}
vector.setValueCount(ROWS);
root.setRowCount(ROWS);
try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR);
ArrowSchema arrowSchema = ArrowSchema.allocateNew(ALLOCATOR)) {
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, arrowSchema);
writer.writeBatch(array.memoryAddress(), arrowSchema.memoryAddress());
}
}
}

private static void writeWithJava(Path path, IntFunction<String> values, IntPredicate isNull,
WriteOptions options) throws IOException {
boolean[] valid = new boolean[ROWS];
String[] strings = new String[ROWS];
for (int i = 0; i < ROWS; i++) {
valid[i] = !isNull.test(i);
strings[i] = valid[i] ? values.apply(i) : null;
}
DType.Struct schema = new DType.Struct(List.of(V), List.of(DTypes.UTF8.asNullable()), false);
try (var channel = FileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
var writer = io.github.dfa1.vortex.writer.VortexWriter.create(channel, schema, options)) {
writer.writeChunk(Map.of(V, new NullableData(strings, valid)));
}
}

private static List<String> readWithJava(Path file) throws IOException {
List<String> out = new ArrayList<>();
try (var reader = io.github.dfa1.vortex.reader.VortexReader.open(file);
var scan = reader.scan(io.github.dfa1.vortex.reader.ScanOptions.all())) {
while (scan.hasNext()) {
try (var chunk = scan.next()) {
io.github.dfa1.vortex.reader.array.VarBinArray array = chunk.column(V);
for (int i = 0; i < chunk.rowCount(); i++) {
out.add(array.getString(i));
}
}
}
}
return out;
}

/// Partitions come back in vortex-jni's order, which is the file's for a single-chunk column (all
/// the shapes here); a multi-chunk shape would need them ordered.
private static List<String> readWithJni(Path file, ImmutableScanOptions.Builder options) throws IOException {
ScanOptions opts = options.projection(Expression.select(new String[]{"v"}, Expression.root())).build();
List<String> out = new ArrayList<>();
DataSource ds = DataSource.open(SESSION, file.toAbsolutePath().toUri().toString());
Scan scan = ds.scan(opts);
while (scan.hasNext()) {
Partition partition = scan.next();
try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) {
while (reader.loadNextBatch()) {
VectorSchemaRoot root = reader.getVectorSchemaRoot();
FieldVector vec = root.getVector("v");
for (int i = 0; i < root.getRowCount(); i++) {
out.add(vec.isNull(i) ? null : vec.getObject(i).toString());
}
}
}
}
return out;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -313,9 +313,11 @@ private EncodingEncoder resolveOverride(DType dtype, Object data, EncodingEncode
// and so does a nullable struct or list, which carry it as a child of their own (#648)
boolean keepsValidity = options.allowedCascading() > 0
&& (CascadingCompressor.keepsValidity(dtype) || CascadingCompressor.carriesValidity(dtype));
// a text column without a null has no validity to store, whatever the cascade picks
boolean noValidity = options.allowedCascading() > 0 && CascadingCompressor.isUnmaskedText(dtype, data);
if (encodingOverride == null
&& data instanceof NullableData
&& !(dtype instanceof DType.Extension) && !keepsValidity) {
&& !(dtype instanceof DType.Extension) && !keepsValidity && !noValidity) {
EncodingEncoder nullableCapable = nullableCapableEncoder(dtype);
return nullableCapable != null ? nullableCapable : new MaskedEncodingEncoder();
}
Expand Down Expand Up @@ -395,7 +397,8 @@ public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
registry, editionExcluded).withTrace(ctx.trace());
// a decimal, struct or list keeps its validity in its own encoding, as on the default path
boolean masked = data instanceof NullableData && !(dtype instanceof DType.Extension)
&& !(options.allowedCascading() > 0 && CascadingCompressor.carriesValidity(dtype));
&& !(options.allowedCascading() > 0 && (CascadingCompressor.carriesValidity(dtype)
|| CascadingCompressor.isUnmaskedText(dtype, data)));
if (masked) {
return new MaskedEncodingEncoder().encode(dtype, data, columnCtx);
}
Expand Down
Loading
Loading