Skip to content

feat(spark): expose completed output writer summaries - #60

Merged
ChristopherWirt merged 2 commits into
developfrom
cwirt/spark-writer-summary
Nov 2, 2026
Merged

ChristopherWirt merged 2 commits into
developfrom
cwirt/spark-writer-summary

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #59.

Problem

VortexOutputWriter.close() throws away the VortexWriteSummary that VortexWriter.finish() returns. To get a file's size or row count, a caller has to open the file again.

Change

When close() succeeds, the writer keeps an immutable, serializable VortexOutputWriterSummary, available from summary():

Accessor Meaning
fileSize() / rowCount() Exact values, from VortexWriter.finish()
batchCount() Batches submitted, including the final partial batch
peakBatchArrowBytes() Largest single batch's Arrow buffer capacity, including nested buffers
peakBufferedBytes() Largest native bufferedBytes() value, sampled after each batch
  • The summary is published only after native finish, Arrow cleanup and sink close all succeed.
  • If a write or close fails, summary() throws IllegalStateException.
  • Closing again keeps the same summary.
  • Writes after close are rejected.
  • Both memory figures are approximate signals, not bounds on process memory. The README and javadoc spell out what each one leaves out.

Tests

VortexOutputWriterSummaryTest covers:

  • empty, full and partial batches
  • nested buffer accounting
  • idempotent close
  • failed writes, failed finish and failed sink close
  • Java serialization, and Spark Kryo serialization with spark.kryo.unsafe both on and off

🤖 Generated with Claude Code

ChristopherWirt and others added 2 commits November 2, 2026 11:05
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Fold the Arrow buffer walk into one recursive helper, give
VortexOutputWriterSummary value semantics, and drop redundant test
assertions and helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt
ChristopherWirt merged commit cb719c5 into develop Nov 2, 2026
1 check passed
ChristopherWirt added a commit that referenced this pull request Nov 3, 2026
## Problem

`VortexOutputWriter` has no way to set footer metadata, although
`VortexWriter.Builder` supports it. Callers that write files outside
Spark's commit protocol and need their own footer keys must keep a
custom writer. The event store keeps `BoundedVortexFileWriter` for this
reason alone, now that #60, #62 and #70 cover its batch limits and
stats.

## Change

- Add a `VortexOutputWriter` constructor that takes `Map<String, byte[]>
metadata` and passes it to `VortexWriter.Builder.metadata`.
- The existing constructor delegates with no metadata, so Spark's write
path is unchanged.
- Add `storesFooterMetadata`: writes a file with one metadata key, then
reads both the key and the row back.

## Validation

- `git diff --check` passes.
- Gradle tests and spotless were not run locally; CI runs them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Spark] Expose a write summary from VortexOutputWriter

2 participants