Skip to content

[Spark] Limit VortexOutputWriter batches by bytes, not just rows #11126

Open

Description

@ChristopherWirt

Problem or motivation

VortexOutputWriter cuts a batch only by row count (vortex.write.batch.size). With wide or deeply nested rows, nothing limits the Arrow memory of one batch, and a run of large rows makes one very large batch.

Writing wide event records at 8,192 rows per batch, one batch peaked at about 99 MiB of Arrow. The native writer can have several batches in flight, so per-task memory depends on the data rather than on configuration.

A smaller row count doesn't fix this: it wastes batches on narrow data and still leaves no limit on wide data.

Proposed solution

An optional vortex.write.batch.bytes write option:

  • After each complete row, flush the batch if its Arrow allocations have reached the threshold, as well as at the existing row limit.
  • Count the open batch's own backing buffers, nested buffers included. An allocator baseline would undercount once native code releases older batches.
  • Unset by default, so behaviour doesn't change.

It limits batch size, not total process memory: a single row can still exceed the threshold.

Additional context

Together with a write summary (#11125), this would let a caller check and limit writer memory per file through configuration alone.

Activity

  1. jackylee-ch commented on Nov 12, 2026

    @jackylee-ch
    Contributor

    I'd like to implement this as proposed: an optional vortex.write.batch.bytes for the Spark writer, unset by default so existing behaviour does not change. After each complete row, the open batch is flushed when its own Arrow buffers (nested included) reach the threshold, in addition to the row limit. A single row can still exceed it. I'll open a PR with a test on wide string rows shortly. Happy to adjust if someone is already working on it.

  2. robert3005 commented on Nov 12, 2026

    @robert3005
    Contributor

    I have #11357 that fixes the size estimation. I have used iceberg rolling writer to make sure this behaves as expected. After this merges we can adopt it in spark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions