You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Spark] Limit VortexOutputWriter batches by bytes, not just rows #11126
VortexOutputWriter cuts a batch only by row count (vortex.write.batch.size). With wide or deeply nested rows, nothing limits the Arrow memory of one batch, and a run of large rows makes one very large batch.
Writing wide event records at 8,192 rows per batch, one batch peaked at about 99 MiB of Arrow. The native writer can have several batches in flight, so per-task memory depends on the data rather than on configuration.
A smaller row count doesn't fix this: it wastes batches on narrow data and still leaves no limit on wide data.
Proposed solution
An optional vortex.write.batch.bytes write option:
After each complete row, flush the batch if its Arrow allocations have reached the threshold, as well as at the existing row limit.
Count the open batch's own backing buffers, nested buffers included. An allocator baseline would undercount once native code releases older batches.
Unset by default, so behaviour doesn't change.
It limits batch size, not total process memory: a single row can still exceed the threshold.
Additional context
Together with a write summary (#11125), this would let a caller check and limit writer memory per file through configuration alone.
I'd like to implement this as proposed: an optional vortex.write.batch.bytes for the Spark writer, unset by default so existing behaviour does not change. After each complete row, the open batch is flushed when its own Arrow buffers (nested included) reach the threshold, in addition to the row limit. A single row can still exceed it. I'll open a PR with a test on wide string rows shortly. Happy to adjust if someone is already working on it.
I have #11357 that fixes the size estimation. I have used iceberg rolling writer to make sure this behaves as expected. After this merges we can adopt it in spark
Activity
jackylee-ch commentedon Nov 12, 2026
I'd like to implement this as proposed: an optional
vortex.write.batch.bytesfor the Spark writer, unset by default so existing behaviour does not change. After each complete row, the open batch is flushed when its own Arrow buffers (nested included) reach the threshold, in addition to the row limit. A single row can still exceed it. I'll open a PR with a test on wide string rows shortly. Happy to adjust if someone is already working on it.robert3005 commentedon Nov 12, 2026
I have #11357 that fixes the size estimation. I have used iceberg rolling writer to make sure this behaves as expected. After this merges we can adopt it in spark