Skip to content

[Spark] Expose a write summary from VortexOutputWriter #59

Closed

Description

@ChristopherWirt

Problem

VortexOutputWriter.close() finishes the file but discards the VortexWriteSummary from VortexWriter.finish(). A caller can't learn anything about the file it wrote without opening it again.

Proposal

After close(), expose what the write did, e.g. VortexWriteSummary summary():

  • file size and row count;
  • number of batches;
  • peak Arrow bytes of one batch;
  • peak bufferedBytes() reported by the native writer.

Why

  • Callers can collect file stats from tasks (sizes, rows per file) to check layout and skew, without listing and reopening files.
  • Peak batch and buffered bytes show writer memory per file. A regression in native memory retention, such as JNI writer keeps imported Arrow batches alive until the file finishes #46, then shows up as a number that moves rather than as OOM kills.

Together with a byte bound on batches, this would let a caller that names and places its own files (writing through a NativeWritable it provides) use VortexOutputWriter directly, instead of reimplementing it on top of package-private ArrowFieldWriter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions