Skip to content

feat(spark): adapt Arrow capacity hints to batch utilization - #70

Merged
ChristopherWirt merged 4 commits into
developfrom
cwirt/spark-arrow-capacity-policy
Nov 3, 2026
Merged

ChristopherWirt merged 4 commits into
developfrom
cwirt/spark-arrow-capacity-policy

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Arrow keeps sizing hints after a batch is exported and cleared. Retaining useful hints avoids
regrowth on steady data, but an unusually large string or list can inflate allocations for
every later batch. Native imports those buffers zero-copy and holds the unused capacity too.

Behavior

Adds vortex.write.reset.capacity, validated on the driver:

Value After a row-limit flush
auto (default) Preserve useful hints. Reset a variable-width leaf or list/map subtree only when the completed batch uses less than 25% of buffer capacity. Recurse through structs and keep row-bounded fixed-width hints.
true Reset every vector to a small starting capacity.
false Keep all hints.

Every mode resets all hints after a byte-triggered flush, preserving develop behavior and
avoiding repeated one-row batches. Hint changes affect future allocations and leave exported
buffers intact.

The adaptive check tolerates ordinary Arrow rounding and growth, so steady variable-width data
keeps useful capacity. It inspects buffer lengths once per batch, with no per-value scan.
After usage drops, the first smaller batch can still inherit a large allocation; the following
batch uses the reset hint. This is the memory tradeoff for avoiding unconditional regrowth.

Matrix and default choice

207 fresh-JVM measured writes: 23 workloads × 3 current modes × 3 repetitions, using the
same fixtures and pinned release JNI library. Output byte counts match across all modes
and repetitions.

Cumulative JVM allocation in MiB, not peak live heap:

Case false auto true
tiny-flat 3.31 3.31 3.32
narrow-flat 2.73 2.74 4.01
narrow-largebatch 1.51 1.51 1.93
wide-flat 5.40 5.40 7.85
short-text 1.72 1.74 2.86
long-text 3.02 3.03 3.18
long-text-largebatch 2.98 2.97 2.97
nested 2.20 2.23 2.80
nested-outlier 4.33 4.00 4.03
map-null 0.47 0.48 0.63
small-outlier 1.26 1.17 1.17
narrow-long 81.45 81.59 121.97
wide-long 39.39 39.40 59.69
wide-struct 41.36 41.20 62.32
max-batch 8.65 8.65 9.27
byte-limited-outlier 0.28 0.28 0.28
byte-limited-steady 0.32 0.32 0.32
short-text-long 6.08 6.12 11.08
long-text-long 3.85 3.77 5.45
nested-long 17.14 17.33 21.54
map-null-long 3.22 3.28 4.61
periodic-outlier 5.21 4.94 4.96
byte-limited-periodic 4.88 4.98 5.03

On the longer steady workloads, auto has JVM allocation close to false: short text uses
6.12 versus 6.08 MiB; long text 3.77 versus 3.85 MiB; nested data 17.33 versus 17.14 MiB;
maps 3.28 versus 3.22 MiB. false remains a good option for known steady data because it
avoids the utilization check. For nested-long, false measured 121 ms versus 135 ms for
auto; timing ranges overlap and these are short local runs.

Sampled retained Arrow in MiB shows the protection and its cost:

Case false auto true
nested-outlier 320.13 3.87 2.62
small-outlier 73.13 2.26 1.13
periodic-outlier 320.13 21.30 11.34
byte-limited-outlier 1.13 1.13 1.13
byte-limited-steady 0.13 0.13 0.13
byte-limited-periodic 11.22 11.22 11.22

The reason to keep auto as the default is protection against severe capacity inflation when
values change and the byte limit is unset, while retaining useful sizing on steady data.
It allows up to four times the observed used bytes and can retain one inflated batch after
usage drops. true retains the aggressive reset option. The byte-limited cases benefit all
modes equally and do not justify choosing auto over false.

Benchmark harness, full 23-case allocation/memory/timing matrices and reproduction commands:
benchmark branch.
Raw medians, timing ranges and source/library hashes:
adaptive results.

Timings are three short local runs with a discarding sink. Some ranges are wide, and sampled
native retention depends on scheduling. Treat speed differences as exploratory.

Validation

  • 69 focused writer and summary tests freshly pass on each of Spark 3.5/Scala 2.12,
    Spark 3.5/Scala 2.13 and Spark 4.0/Scala 2.13 (207 tests total).
  • Tests verify retaining capacity on full batches and shrinking after values get smaller
    for strings, binary, lists, maps and struct fields, with value readback and buffer release.
  • Existing regressions still cover root/nested fixed-width hints, one-off outliers and
    unconditional resets after byte-triggered flushes in all three modes.
  • All 207 current-mode benchmark writes match output byte counts across policies and repetitions.
  • Complete-review: four independent reviewers found no actionable issues; primary revalidation
    confirmed buffer-lifetime and reset semantics.
  • 12 additional benchmark smoke writes cover steady strings/nested data, recurring outliers
    and byte-triggered resets; all output byte counts match across modes.
  • Java compile/Error Prone, Spotless and patch whitespace checks pass.
  • Not run: Rust checks or a Sphinx build; no Rust or documentation configuration changed.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>

@gitRasheed gitRasheed left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mechanism holds up. clear() keeps the last capacity, allocateNew() reuses it, and the reset only touches Java-side hints, never buffers native already holds. Two things before merge, plus a test gap:

  1. auto resets the fixed-width children of top-level structs. resetInitialCapacity (VortexOutputWriter.java:229) only looks at root vectors. A root StructVector isn't a BaseFixedWidthVector, so it gets reset, and StructVector.setInitialCapacity passes that on to every child. A struct's fixed-width fields are row-bounded just like root ones, so on struct-heavy schemas, which is our wide nested shape, auto behaves like true and pays the regrowth cost the PR gives as the reason not to default to true. Recursing through struct children with the same rule would fix it: reset whole subtrees only at lists and maps, plus variable-width leaves.

  2. false with vortex.write.batch.bytes gives one-row batches on steady data. After a byte-triggered flush the hint is kept (line 254), so the next allocateNew() allocates at least the budget again, and the first row already hits batchBytes. That is what capacityPolicyAlsoAppliesAfterAByteTriggeredFlush asserts for false. No outlier is needed. One long column at batch.size 8192 crosses the budget once while doubling, then every later batch is one row in a budget-sized allocation, and native keeps 8192 of them per block. Keeping develop's unconditional reset after a byte flush, so false only means "keep hints across row-limit flushes", or rejecting false with batch.bytes, avoids it. It's worth settling before anyone sets it.

  3. No test tells auto from true, or covers || reachedBytes: both new tests use one array<long> column, where the modes act the same. Dropping either the instanceof BaseFixedWidthVector check or || reachedBytes passes every test. A one-long-column case with a small batch.bytes (assert a one-row batch after the byte flush under auto/true), plus a struct-of-long case once 1 is fixed, would catch both.

Minor: the default now regrows string/list/map vectors from 64 values each batch on steady data. Your own numbers show a little more JVM allocation for short-text and nested. Worth a sentence in the description.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt

Copy link
Copy Markdown
Collaborator Author

Making improvements to auto

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt ChristopherWirt changed the title feat(spark): choose Arrow capacity resets by schema feat(spark): adapt Arrow capacity hints to batch utilization Nov 3, 2026
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>

@gitRasheed gitRasheed left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed: structs keep their fixed-width hints, byte-triggered flushes always reset, and the new tests tell auto from true and cover the byte path on steady data. The excess check before clear() is a nice touch.

@ChristopherWirt

Copy link
Copy Markdown
Collaborator Author

@gitRasheed this PR had moved auto to adaptive approach which now works pretty well in nearly all cases for staying - almost not worth having a setting

@ChristopherWirt
ChristopherWirt merged commit ee7a572 into develop Nov 3, 2026
1 check passed
@gitRasheed

Copy link
Copy Markdown
Collaborator

Agreed, the adaptive auto reads well. If it holds up after the release, we could drop true/false and keep auto as the only behaviour. That's one less option to document.

ChristopherWirt added a commit that referenced this pull request Nov 3, 2026
## Problem

`VortexOutputWriter` has no way to set footer metadata, although
`VortexWriter.Builder` supports it. Callers that write files outside
Spark's commit protocol and need their own footer keys must keep a
custom writer. The event store keeps `BoundedVortexFileWriter` for this
reason alone, now that #60, #62 and #70 cover its batch limits and
stats.

## Change

- Add a `VortexOutputWriter` constructor that takes `Map<String, byte[]>
metadata` and passes it to `VortexWriter.Builder.metadata`.
- The existing constructor delegates with no metadata, so Spark's write
path is unchanged.
- Add `storesFooterMetadata`: writes a file with one metadata key, then
reads both the key and the row back.

## Validation

- `git diff --check` passes.
- Gradle tests and spotless were not run locally; CI runs them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants