Skip to content

feat(spark): let VortexOutputWriter store footer metadata - #77

Merged
ChristopherWirt merged 1 commit into
developfrom
cwirt/spark-writer-footer-metadata
Nov 3, 2026
Merged

ChristopherWirt merged 1 commit into
developfrom
cwirt/spark-writer-footer-metadata

Conversation

@ChristopherWirt

Copy link
Copy Markdown
Collaborator

Problem

VortexOutputWriter has no way to set footer metadata, although VortexWriter.Builder supports it. Callers that write files outside Spark's commit protocol and need their own footer keys must keep a custom writer. The event store keeps BoundedVortexFileWriter for this reason alone, now that #60, #62 and #70 cover its batch limits and stats.

Change

  • Add a VortexOutputWriter constructor that takes Map<String, byte[]> metadata and passes it to VortexWriter.Builder.metadata.
  • The existing constructor delegates with no metadata, so Spark's write path is unchanged.
  • Add storesFooterMetadata: writes a file with one metadata key, then reads both the key and the row back.

Validation

  • git diff --check passes.
  • Gradle tests and spotless were not run locally; CI runs them.

🤖 Generated with Claude Code

Adds a VortexOutputWriter constructor that takes footer metadata segments
and passes them to the JNI writer builder, so callers writing files outside
Spark's commit protocol can record their own metadata without a custom writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt
ChristopherWirt merged commit 90799d1 into develop Nov 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants