You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Spark] Expose a write summary from VortexOutputWriter #59
VortexOutputWriter.close() finishes the file but discards the VortexWriteSummary from VortexWriter.finish(). A caller can't learn anything about the file it wrote without opening it again.
Proposal
After close(), expose what the write did, e.g. VortexWriteSummary summary():
file size and row count;
number of batches;
peak Arrow bytes of one batch;
peak bufferedBytes() reported by the native writer.
Why
Callers can collect file stats from tasks (sizes, rows per file) to check layout and skew, without listing and reopening files.
Together with a byte bound on batches, this would let a caller that names and places its own files (writing through a NativeWritable it provides) use VortexOutputWriter directly, instead of reimplementing it on top of package-private ArrowFieldWriter.
Activity