Skip to content

Add finish-time metadata and schema-less batches to the JNI writer - #11455

Open
robert3005 wants to merge 2 commits into
developfrom
rk/jni-finish-metadata
Open

robert3005 wants to merge 2 commits into
developfrom
rk/jni-finish-metadata

Conversation

@robert3005

@robert3005 robert3005 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Apache Hudi knows its footer values (bloom filter, min and max keys) only after the last row, but vortex-jni accepts metadata segments only at create time. The Rust writer serializes these segments only with the footer, so this PR lets a caller add them at finish.

It also adds a batch write with no schema argument. writeBatch(long, long) releases the schema on each call, so a caller that reuses one exported schema crashes on the second batch.

Changes

  • vortex-file: VortexWriteOptions::deferred_metadata() returns a DeferredMetadata handle. The write reads it after the input ends and merges it with the create-time segments. A late key replaces the earlier key, and the limits apply to the merged set. Writer::finish_with_metadata and BlockingWriter::finish_with_metadata use it.
  • vortex-jni: NativeWriter.finish takes a nullable metadata map. New NativeWriter.writeArray imports the array with the writer's schema and checks buffer, child and dictionary counts first. It cannot find a mismatch in value width.
  • Java: VortexWriter#finish(Map<String, byte[]>) and VortexWriter#writeBatch(long). The existing finish(), close() and writeBatch(long, long) do not change.
  • Tests: Rust and Java tests for merge and override, invalid merged metadata, the stream destination, multiple schema-less batches, and a mismatched array.

API Changes

Breaking: dev.vortex.jni.NativeWriter.finish(long) is now finish(long, Map<String, byte[]>). Its only caller in the repo is VortexWriter.

🤖 Generated with Claude Code

Java writers can now add file metadata segments when they finish the
file. A caller can then write values that it knows only after the last
batch, for example a bloom filter of the written keys. The writer reads
these segments after the input ends and merges them with the
create-time segments. A late key replaces the create-time key. The
metadata limits apply to the merged set, and a violation is an error
from finish.

VortexWriter#writeBatch(long) writes a batch with no schema argument.
Native code imports the array with the schema of the writer, so a
caller does not export and release a schema for each batch. Native code
checks the buffer and child counts of the array before the import.

BREAKING: NativeWriter.finish(long) is now finish(long, Map).
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the changelog/break A breaking API change label Nov 13, 2026
…ch APIs

Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant