From 76762dabf73cc9af5470d2a424af80b3ea14e2c1 Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Sat, 7 Nov 2026 22:00:46 +0100 Subject: [PATCH] fix(writer): a nullable text column without a null is written without validity (#650) Rust imports a nullable string array whose null count is zero as all-valid, with no validity child, so the cascade never wraps it. Java wrapped it in vortex.masked with a constant validity. Route it to the compressor, as a nullable primitive without nulls. NullableTextInteropIntegrationTest compares encoding trees with vortex-jni (the no-null unique shape matches now; the other shapes still differ in the text scheme itself, tracked in #650) and has vortex-jni read every Java file, filtered and ranged. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV --- CHANGELOG.md | 3 + docs/compatibility.md | 2 +- .../NullableTextInteropIntegrationTest.java | 252 ++++++++++++++++++ .../dfa1/vortex/writer/SegmentEncoder.java | 7 +- .../writer/encode/CascadingCompressor.java | 17 +- .../vortex/writer/NullableTextMaskTest.java | 99 +++++++ 6 files changed, 375 insertions(+), 5 deletions(-) create mode 100644 integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java create mode 100644 writer/src/test/java/io/github/dfa1/vortex/writer/NullableTextMaskTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index b80d6af77..68eecf33d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - `EncodeContext` has a new last component, `trace`, so code calling its canonical constructor passes `EncodeTrace.NONE`. ([#601](https://github.com/dfa1/vortex-java/issues/601)) +### Fixed +- A nullable text column without a null is written without a validity, as Rust writes it, instead of under `vortex.masked` (with cascading on). ([#650](https://github.com/dfa1/vortex-java/issues/650)) + ### Added - JDK Flight Recorder events for file open, HTTP range requests, segment decode and encode, chunk filtering, writer back-pressure and the compressor's scheme choices, which mirror Rust's compressor trace. ([#601](https://github.com/dfa1/vortex-java/issues/601)) diff --git a/docs/compatibility.md b/docs/compatibility.md index 16647f6ad..260f755db 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -136,7 +136,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul | `vortex.map` | `MapEncodingDecoder` | `MapEncodingEncoder` | ✅ | ✅ | Single `entries` child, a bare `vortex.listview` of non-nullable `{key, value}` structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither `ChunkedArrayCombiner` nor `ScanIterator`'s row-window slicing has a `MapArray` arm yet, so a chunked map column or a split-window scan fails as `VortexException`. Pre-existing gap shared with `vortex.listview`; tracked separately | | `vortex.fixed_size_list` | `FixedSizeListEncodingDecoder` | `FixedSizeListEncodingEncoder` | ✅ | ✅ | | | `vortex.zstd` | `ZstdEncodingDecoder` | `ZstdEncodingEncoder` | ✅ | ✅ | Primitive, Utf8, Binary | -| `vortex.masked` | `MaskedEncodingDecoder` | `MaskedEncodingEncoder` | ✅ | ✅ | NullableData carrier. A nullable primitive or decimal column is not masked when cascading: as in Rust, its validity is a plain `vortex.bool` child of the encoding that stores the values (`vortex.primitive`, `fastlanes.bitpacked`, the run values of `vortex.runend`, the indices of `fastlanes.rle`, the codes of `vortex.dict`, the left parts of `vortex.alprd`, `vortex.pco` over the valid values only), passed down by `fastlanes.for`, `vortex.zigzag` and `vortex.alp` (which patches only valid exceptions), and one whose valid rows hold one value is a `vortex.constant` masked by a plain bitmap. Other nullable columns: inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it | +| `vortex.masked` | `MaskedEncodingDecoder` | `MaskedEncodingEncoder` | ✅ | ✅ | NullableData carrier. A nullable primitive or decimal column is not masked when cascading: as in Rust, its validity is a plain `vortex.bool` child of the encoding that stores the values (`vortex.primitive`, `fastlanes.bitpacked`, the run values of `vortex.runend`, the indices of `fastlanes.rle`, the codes of `vortex.dict`, the left parts of `vortex.alprd`, `vortex.pco` over the valid values only), passed down by `fastlanes.for`, `vortex.zigzag` and `vortex.alp` (which patches only valid exceptions), and one whose valid rows hold one value is a `vortex.constant` masked by a plain bitmap. Other nullable columns: inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 or Binary column without a null is not masked when cascading (including under a `ColumnEncoding` override): it has no validity at all, as Rust's all-valid string array has ([#650](https://github.com/dfa1/vortex-java/issues/650)); at cascade depth 0 it is still masked. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it | | `vortex.decimal` | `DecimalEncodingDecoder` | `DecimalEncodingEncoder` | ✅ | ✅ | Precision 1–76 (i8 … i256 storage by precision), negative scales. Written from `BigDecimal[]`. A nullable column keeps its validity as an optional `vortex.bool` child, not a `vortex.masked` wrapper ([#646](https://github.com/dfa1/vortex-java/issues/646)) | | `vortex.decimal_byte_parts` | `DecimalBytePartsEncodingDecoder`| `DecimalBytePartsEncodingEncoder`| ✅ | ✅ | Write: one mantissa, no low parts, so a decimal is split only when every unscaled value fits a long (any precision, as Rust's `narrowed_decimal` finds from the data). As Rust's `DecimalScheme`, the mantissa is narrowed to the smallest of i8–i64 holding the values and cascaded (frame of reference, bit-packing, …); a chunk of fewer than two values stays `vortex.decimal`, as Rust leaves it un-narrowed. The byte parts are kept only when strictly smaller than the plain decimal, and a column of equal values is one `vortex.constant`. A nullable column keeps its validity on the mantissa child, or has none when every row is valid, rather than in a `vortex.masked` wrapper ([#646](https://github.com/dfa1/vortex-java/issues/646)) | | `vortex.datetimeparts` | `DateTimePartsEncodingDecoder` | `DateTimePartsEncodingEncoder` | ✅ | ✅ | | diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java new file mode 100644 index 000000000..01166aac3 --- /dev/null +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/NullableTextInteropIntegrationTest.java @@ -0,0 +1,252 @@ +package io.github.dfa1.vortex.integration; + +import dev.vortex.api.DataSource; +import dev.vortex.api.ImmutableScanOptions; +import dev.vortex.api.Expression; +import dev.vortex.api.Partition; +import dev.vortex.api.Scan; +import dev.vortex.api.ScanOptions; +import dev.vortex.api.Session; +import dev.vortex.api.VortexWriter; +import dev.vortex.arrow.ArrowAllocation; +import dev.vortex.jni.NativeLoader; +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.writer.WriteOptions; +import io.github.dfa1.vortex.writer.encode.NullableData; +import org.apache.arrow.c.ArrowArray; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.FieldVector; +import org.apache.arrow.vector.VarCharVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import java.io.IOException; +import java.nio.channels.FileChannel; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.function.IntPredicate; +import java.util.function.IntFunction; +import java.util.stream.Stream; + +import static org.assertj.core.api.Assertions.assertThat; + +/// Each column goes through vortex-jni and vortex-java, and the two files must hold the same +/// encoding tree, data and zone stats table alike. Each shape steers the cascade to a different +/// winner. +/// A nullable text column without a null is written without a validity, as Rust writes an all-valid +/// string array (#650), instead of under `vortex.masked`. Columns with nulls are still masked, and +/// differ from Rust in the text scheme itself (#650), so the encoding tree is compared with +/// vortex-jni's for the no-null shape only; every shape is read back by Java and by vortex-jni, +/// full, filtered and row-ranged, compact files included. +class NullableTextInteropIntegrationTest { + + private static final Session SESSION = Session.create(); + private static final BufferAllocator ALLOCATOR = ArrowAllocation.rootAllocator(); + private static final ColumnName V = ColumnName.of("v"); + private static final int ROWS = 4 * 8192; + private static final long[] RANDOM = new Random(7).longs(ROWS).toArray(); + + static { + NativeLoader.loadJni(); + } + + static Stream shapes() { + IntPredicate every7 = i -> i % 7 == 0; + IntFunction few = i -> "value-" + Math.floorMod(RANDOM[i], 10); + IntFunction many = i -> "some longer text value number " + Math.floorMod(RANDOM[i], 5000) + " end"; + IntFunction unique = i -> "row-" + RANDOM[i]; + return Stream.of( + shape("few distinct, no nulls", few, _ -> false), + shape("many distinct, no nulls", many, _ -> false), + shape("unique, no nulls", unique, _ -> false), + shape("few distinct", few, every7), + shape("many distinct", many, every7), + shape("unique", unique, every7), + shape("one null", few, i -> i == 5), + shape("95% null", few, i -> Math.floorMod(RANDOM[i] >>> 20, 100) < 95), + shape("runs of nulls", few, i -> (i / 500) % 3 == 0)); + } + + /// The shapes whose whole tree is Rust's. The others still differ beyond the validity, in the + /// scheme chosen for the text itself (#650: the probe applies to numbers only; FSST's lengths). + static Stream treeShapes() { + return shapes().filter(a -> a.get()[0].equals("unique, no nulls")); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("treeShapes") + void javaWritesTheEncodingTreeJniWrites(String name, IntFunction values, IntPredicate isNull, + @TempDir Path tmp) throws Exception { + // Given + Path jniFile = tmp.resolve("jni.vortex"); + Path javaFile = tmp.resolve("java.vortex"); + writeWithJni(jniFile, values, isNull); + List expected = EncodingTrees.of(jniFile); + + // When + writeWithJava(javaFile, values, isNull, WriteOptions.defaults()); + + // Then + assertThat(EncodingTrees.of(javaFile)).isEqualTo(expected); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("shapes") + void jniReadsJavaFile(String name, IntFunction values, IntPredicate isNull, @TempDir Path tmp) + throws Exception { + assertJniReads(values, isNull, WriteOptions.defaults(), tmp); + } + + /// Compact writes add Zstd; vortex-jni cannot write them, so only its read of Java's file checks them. + @ParameterizedTest(name = "{0}") + @MethodSource("shapes") + void jniReadsJavaCompactFile(String name, IntFunction values, IntPredicate isNull, @TempDir Path tmp) + throws Exception { + assertJniReads(values, isNull, WriteOptions.defaults().withCompact(true), tmp); + } + + /// Columns without a null: those are the plain values a Java read returns as strings. + static Stream noNullShapes() { + return shapes().filter(a -> a.get()[0].toString().endsWith("no nulls")); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("noNullShapes") + void javaReadsItsOwnFile(String name, IntFunction values, IntPredicate isNull, @TempDir Path tmp) + throws Exception { + // Given + Path file = tmp.resolve("java.vortex"); + writeWithJava(file, values, isNull, WriteOptions.defaults()); + + // When + List result = readWithJava(file); + + // Then + List expected = new ArrayList<>(); + for (int i = 0; i < ROWS; i++) { + expected.add(values.apply(i)); + } + assertThat(result).isEqualTo(expected); + } + + private static void assertJniReads(IntFunction values, IntPredicate isNull, WriteOptions options, + Path tmp) throws Exception { + // Given + Path file = tmp.resolve("java.vortex"); + writeWithJava(file, values, isNull, options); + List expected = new ArrayList<>(); + for (int i = 0; i < ROWS; i++) { + expected.add(isNull.test(i) ? null : values.apply(i)); + } + String threshold = values.apply(ROWS / 2); + + // When + List result = readWithJni(file, ScanOptions.builder()); + // a filtered and a row-range read prune and slice, which a full scan never does + List filtered = readWithJni(file, ScanOptions.builder().filter(Expression.binary( + Expression.BinaryOp.EQ, Expression.column("v"), Expression.literal(threshold)))); + List range = readWithJni(file, ScanOptions.builder().rowRangeBegin(1000).rowRangeEnd(20_000)); + + // Then + assertThat(result).isEqualTo(expected); + assertThat(filtered).isEqualTo(expected.stream().filter(threshold::equals).toList()); + assertThat(range).isEqualTo(expected.subList(1000, 20_000)); + } + + private static Arguments shape(String name, IntFunction values, IntPredicate isNull) { + return Arguments.of(name, values, isNull); + } + + private static void writeWithJni(Path path, IntFunction values, IntPredicate isNull) throws Exception { + Schema schema = new Schema(List.of(Field.nullable("v", new ArrowType.Utf8()))); + try (VortexWriter writer = VortexWriter.builder(SESSION, path.toAbsolutePath().toUri().toString(), schema, + ALLOCATOR).build(); + VectorSchemaRoot root = VectorSchemaRoot.create(schema, ALLOCATOR)) { + VarCharVector vector = (VarCharVector) root.getVector("v"); + vector.allocateNew(); + for (int i = 0; i < ROWS; i++) { + if (isNull.test(i)) { + vector.setNull(i); + } else { + vector.setSafe(i, values.apply(i).getBytes(StandardCharsets.UTF_8)); + } + } + vector.setValueCount(ROWS); + root.setRowCount(ROWS); + try (ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); + ArrowSchema arrowSchema = ArrowSchema.allocateNew(ALLOCATOR)) { + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, arrowSchema); + writer.writeBatch(array.memoryAddress(), arrowSchema.memoryAddress()); + } + } + } + + private static void writeWithJava(Path path, IntFunction values, IntPredicate isNull, + WriteOptions options) throws IOException { + boolean[] valid = new boolean[ROWS]; + String[] strings = new String[ROWS]; + for (int i = 0; i < ROWS; i++) { + valid[i] = !isNull.test(i); + strings[i] = valid[i] ? values.apply(i) : null; + } + DType.Struct schema = new DType.Struct(List.of(V), List.of(DTypes.UTF8.asNullable()), false); + try (var channel = FileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var writer = io.github.dfa1.vortex.writer.VortexWriter.create(channel, schema, options)) { + writer.writeChunk(Map.of(V, new NullableData(strings, valid))); + } + } + + private static List readWithJava(Path file) throws IOException { + List out = new ArrayList<>(); + try (var reader = io.github.dfa1.vortex.reader.VortexReader.open(file); + var scan = reader.scan(io.github.dfa1.vortex.reader.ScanOptions.all())) { + while (scan.hasNext()) { + try (var chunk = scan.next()) { + io.github.dfa1.vortex.reader.array.VarBinArray array = chunk.column(V); + for (int i = 0; i < chunk.rowCount(); i++) { + out.add(array.getString(i)); + } + } + } + } + return out; + } + + /// Partitions come back in vortex-jni's order, which is the file's for a single-chunk column (all + /// the shapes here); a multi-chunk shape would need them ordered. + private static List readWithJni(Path file, ImmutableScanOptions.Builder options) throws IOException { + ScanOptions opts = options.projection(Expression.select(new String[]{"v"}, Expression.root())).build(); + List out = new ArrayList<>(); + DataSource ds = DataSource.open(SESSION, file.toAbsolutePath().toUri().toString()); + Scan scan = ds.scan(opts); + while (scan.hasNext()) { + Partition partition = scan.next(); + try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) { + while (reader.loadNextBatch()) { + VectorSchemaRoot root = reader.getVectorSchemaRoot(); + FieldVector vec = root.getVector("v"); + for (int i = 0; i < root.getRowCount(); i++) { + out.add(vec.isNull(i) ? null : vec.getObject(i).toString()); + } + } + } + } + return out; + } +} diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentEncoder.java b/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentEncoder.java index 2cd9bc0dc..24b0483ed 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentEncoder.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/SegmentEncoder.java @@ -313,9 +313,11 @@ private EncodingEncoder resolveOverride(DType dtype, Object data, EncodingEncode // and so does a nullable struct or list, which carry it as a child of their own (#648) boolean keepsValidity = options.allowedCascading() > 0 && (CascadingCompressor.keepsValidity(dtype) || CascadingCompressor.carriesValidity(dtype)); + // a text column without a null has no validity to store, whatever the cascade picks + boolean noValidity = options.allowedCascading() > 0 && CascadingCompressor.isUnmaskedText(dtype, data); if (encodingOverride == null && data instanceof NullableData - && !(dtype instanceof DType.Extension) && !keepsValidity) { + && !(dtype instanceof DType.Extension) && !keepsValidity && !noValidity) { EncodingEncoder nullableCapable = nullableCapableEncoder(dtype); return nullableCapable != null ? nullableCapable : new MaskedEncodingEncoder(); } @@ -395,7 +397,8 @@ public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) { registry, editionExcluded).withTrace(ctx.trace()); // a decimal, struct or list keeps its validity in its own encoding, as on the default path boolean masked = data instanceof NullableData && !(dtype instanceof DType.Extension) - && !(options.allowedCascading() > 0 && CascadingCompressor.carriesValidity(dtype)); + && !(options.allowedCascading() > 0 && (CascadingCompressor.carriesValidity(dtype) + || CascadingCompressor.isUnmaskedText(dtype, data))); if (masked) { return new MaskedEncodingEncoder().encode(dtype, data, columnCtx); } diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/CascadingCompressor.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/CascadingCompressor.java index 60a200812..3182ad304 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/CascadingCompressor.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/CascadingCompressor.java @@ -231,6 +231,18 @@ public static boolean carriesValidity(DType dtype) { return dtype instanceof DType.Struct || dtype instanceof DType.List || dtype instanceof DType.Decimal; } + /// Whether `data` is a nullable text column without a null, which has no validity to store: as + /// Rust imports a string array whose null count is zero (`Validity::AllValid`), it is written + /// as the plain values, never under a `vortex.masked`. + /// + /// @param dtype the column's dtype + /// @param data the column's carrier + /// @return `true` for a [NullableData] of utf8 or binary values with every row valid + public static boolean isUnmaskedText(DType dtype, Object data) { + return (dtype instanceof DType.Utf8 || dtype instanceof DType.Binary) + && data instanceof NullableData nullable && nullable.nullCount() == 0; + } + /// Entry point: encode `data` using the best cascading strategy. /// /// Cascade parameters (depth, sampling, exclusions) are taken from `ctx`. @@ -246,9 +258,10 @@ public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) { // null count is zero (`Validity::AllValid`). Deeper down a NullableData keeps its bitmap // even when every bit is set, as a filtered or sampled Rust array does. // An extension's storage is that array too. - if ((keepsValidity(dtype) || dtype instanceof DType.Extension || dtype instanceof DType.Decimal) + if (isUnmaskedText(dtype, data) + || (keepsValidity(dtype) || dtype instanceof DType.Extension || dtype instanceof DType.Decimal) && data instanceof NullableData nullable && nullable.nullCount() == 0) { - return encodeWithCtx(dtype, nullable.values(), ctx); + return encodeWithCtx(dtype, ((NullableData) data).values(), ctx); } return encodeWithCtx(dtype, data, ctx); } diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/NullableTextMaskTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/NullableTextMaskTest.java new file mode 100644 index 000000000..d6f7ceeb1 --- /dev/null +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/NullableTextMaskTest.java @@ -0,0 +1,99 @@ +package io.github.dfa1.vortex.writer; + +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.core.model.EncodingId; +import io.github.dfa1.vortex.reader.VortexReader; +import io.github.dfa1.vortex.writer.encode.CascadingCompressor; +import io.github.dfa1.vortex.writer.encode.NullableData; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import java.io.IOException; +import java.nio.channels.FileChannel; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import static org.assertj.core.api.Assertions.assertThat; + +/// A nullable text column without a null has no validity to store, as Rust's all-valid string array +/// has none (#650): with cascading on it is never written under a `vortex.masked`. +class NullableTextMaskTest { + + private static final ColumnName V = ColumnName.of("v"); + private static final int ROWS = 5_000; + + static Stream writes() { + // no global dict: its layout has no mask either, and these columns are low cardinality + WriteOptions defaults = WriteOptions.defaults().withGlobalDict(false); + WriteOptions fsst = defaults + .withColumnEncoding(V, ColumnEncoding.candidates(EncodingId.VORTEX_FSST, EncodingId.VORTEX_VARBIN)); + return Stream.of( + Arguments.of("no null, cascading", DTypes.UTF8.asNullable(), 0, defaults, false), + Arguments.of("binary, no null, cascading", DTypes.BINARY.asNullable(), 0, defaults, false), + Arguments.of("one null, cascading", DTypes.UTF8.asNullable(), 1, defaults, true), + Arguments.of("no null, depth 0", DTypes.UTF8.asNullable(), 0, WriteOptions.cascading(0).withGlobalDict(false), true), + Arguments.of("no null, column override", DTypes.UTF8.asNullable(), 0, fsst, false), + Arguments.of("one null, column override", DTypes.UTF8.asNullable(), 1, fsst, true)); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("writes") + void write_masksOnlyWhenThereIsValidityToStore(String name, DType dtype, int nulls, WriteOptions options, + boolean masked, @TempDir Path tmp) throws IOException { + // Given + boolean binary = dtype instanceof DType.Binary; + boolean[] validity = new boolean[ROWS]; + Object values; + if (binary) { + byte[][] bytes = new byte[ROWS][]; + for (int i = 0; i < ROWS; i++) { + validity[i] = i >= nulls; + bytes[i] = validity[i] ? ("row-" + i % 50).getBytes() : null; + } + values = bytes; + } else { + String[] strings = new String[ROWS]; + for (int i = 0; i < ROWS; i++) { + validity[i] = i >= nulls; + strings[i] = validity[i] ? "row-" + i % 50 : null; + } + values = strings; + } + Path file = tmp.resolve("text.vtx"); + + // When + try (var channel = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var sut = VortexWriter.create(channel, new DType.Struct(List.of(V), List.of(dtype), false), options)) { + sut.writeChunk(Map.of(V, new NullableData(values, validity))); + } + + // Then + try (var reader = VortexReader.open(file)) { + assertThat(reader.footer().arraySpecs().contains(EncodingId.VORTEX_MASKED)).isEqualTo(masked); + } + } + + @Test + void isUnmaskedText_isOnlyANullableTextCarrierWithEveryRowValid() { + // Given + boolean[] allValid = {true, true}; + boolean[] oneNull = {true, false}; + Object text = new String[]{"a", "b"}; + + // When / Then + assertThat(CascadingCompressor.isUnmaskedText(DTypes.UTF8.asNullable(), new NullableData(text, allValid))).isTrue(); + assertThat(CascadingCompressor.isUnmaskedText(DTypes.BINARY.asNullable(), new NullableData(text, allValid))).isTrue(); + assertThat(CascadingCompressor.isUnmaskedText(DTypes.UTF8.asNullable(), new NullableData(text, oneNull))).isFalse(); + assertThat(CascadingCompressor.isUnmaskedText(DTypes.UTF8.asNullable(), text)).isFalse(); + assertThat(CascadingCompressor.isUnmaskedText(DTypes.I64.asNullable(), + new NullableData(new long[]{1, 2}, allValid))).isFalse(); + } +}