Repository navigation
fix(writer): a nullable text column without a null is written without validity (#650) - #692
Merged
Merged
Conversation
… validity (#650) Rust imports a nullable string array whose null count is zero as all-valid, with no validity child, so the cascade never wraps it. Java wrapped it in vortex.masked with a constant validity. Route it to the compressor, as a nullable primitive without nulls. NullableTextInteropIntegrationTest compares encoding trees with vortex-jni (the no-null unique shape matches now; the other shapes still differ in the text scheme itself, tracked in #650) and has vortex-jni read every Java file, filtered and ranged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of #650 (does not close it).
Rust imports a nullable string array with no null as all-valid, so it has no validity child. Java wrapped it in
vortex.maskedwith a constant validity. It now goes to the compressor unwrapped, also under aColumnEncodingoverride; at cascade depth 0 it is still masked.NullableTextInteropIntegrationTest: vortex-jni vs Java encoding tree (no-null unique shape matches; it failed before), and vortex-jni reads every Java file, full, filtered and ranged, compact included.NullableTextMaskTest: masked only when there is validity to store.Still open on #650: nullable text with nulls (validity inside FSST/OnPair/VarBin/Dict), FSST's lengths child (
sequencevsprimitive), the string probe. Struct fields and list elements of nullable text keep their previous path (no cross-check yet).🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV