Repository navigation
feat(spark): adapt Arrow capacity hints to batch utilization - #70
Conversation
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
gitRasheed
left a comment
There was a problem hiding this comment.
The mechanism holds up. clear() keeps the last capacity, allocateNew() reuses it, and the reset only touches Java-side hints, never buffers native already holds. Two things before merge, plus a test gap:
-
autoresets the fixed-width children of top-level structs.resetInitialCapacity(VortexOutputWriter.java:229) only looks at root vectors. A rootStructVectorisn't aBaseFixedWidthVector, so it gets reset, andStructVector.setInitialCapacitypasses that on to every child. A struct's fixed-width fields are row-bounded just like root ones, so on struct-heavy schemas, which is our wide nested shape,autobehaves liketrueand pays the regrowth cost the PR gives as the reason not to default totrue. Recursing through struct children with the same rule would fix it: reset whole subtrees only at lists and maps, plus variable-width leaves. -
falsewithvortex.write.batch.bytesgives one-row batches on steady data. After a byte-triggered flush the hint is kept (line 254), so the nextallocateNew()allocates at least the budget again, and the first row already hitsbatchBytes. That is whatcapacityPolicyAlsoAppliesAfterAByteTriggeredFlushasserts forfalse. No outlier is needed. One long column at batch.size 8192 crosses the budget once while doubling, then every later batch is one row in a budget-sized allocation, and native keeps 8192 of them per block. Keeping develop's unconditional reset after a byte flush, sofalseonly means "keep hints across row-limit flushes", or rejectingfalsewithbatch.bytes, avoids it. It's worth settling before anyone sets it. -
No test tells
autofromtrue, or covers|| reachedBytes: both new tests use onearray<long>column, where the modes act the same. Dropping either theinstanceof BaseFixedWidthVectorcheck or|| reachedBytespasses every test. A one-long-column case with a smallbatch.bytes(assert a one-row batch after the byte flush underauto/true), plus a struct-of-long case once 1 is fixed, would catch both.
Minor: the default now regrows string/list/map vectors from 64 values each batch on steady data. Your own numbers show a little more JVM allocation for short-text and nested. Worth a sentence in the description.
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
|
Making improvements to auto |
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
gitRasheed
left a comment
There was a problem hiding this comment.
All three addressed: structs keep their fixed-width hints, byte-triggered flushes always reset, and the new tests tell auto from true and cover the byte path on steady data. The excess check before clear() is a nice touch.
|
@gitRasheed this PR had moved auto to adaptive approach which now works pretty well in nearly all cases for staying - almost not worth having a setting |
|
Agreed, the adaptive auto reads well. If it holds up after the release, we could drop true/false and keep auto as the only behaviour. That's one less option to document. |
## Problem `VortexOutputWriter` has no way to set footer metadata, although `VortexWriter.Builder` supports it. Callers that write files outside Spark's commit protocol and need their own footer keys must keep a custom writer. The event store keeps `BoundedVortexFileWriter` for this reason alone, now that #60, #62 and #70 cover its batch limits and stats. ## Change - Add a `VortexOutputWriter` constructor that takes `Map<String, byte[]> metadata` and passes it to `VortexWriter.Builder.metadata`. - The existing constructor delegates with no metadata, so Spark's write path is unchanged. - Add `storesFooterMetadata`: writes a file with one metadata key, then reads both the key and the row back. ## Validation - `git diff --check` passes. - Gradle tests and spotless were not run locally; CI runs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Christopher Wirt <cwirt@quantcast.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Problem
Arrow keeps sizing hints after a batch is exported and cleared. Retaining useful hints avoids
regrowth on steady data, but an unusually large string or list can inflate allocations for
every later batch. Native imports those buffers zero-copy and holds the unused capacity too.
Behavior
Adds
vortex.write.reset.capacity, validated on the driver:auto(default)truefalseEvery mode resets all hints after a byte-triggered flush, preserving
developbehavior andavoiding repeated one-row batches. Hint changes affect future allocations and leave exported
buffers intact.
The adaptive check tolerates ordinary Arrow rounding and growth, so steady variable-width data
keeps useful capacity. It inspects buffer lengths once per batch, with no per-value scan.
After usage drops, the first smaller batch can still inherit a large allocation; the following
batch uses the reset hint. This is the memory tradeoff for avoiding unconditional regrowth.
Matrix and default choice
207 fresh-JVM measured writes: 23 workloads × 3 current modes × 3 repetitions, using the
same fixtures and pinned release JNI library. Output byte counts match across all modes
and repetitions.
Cumulative JVM allocation in MiB, not peak live heap:
On the longer steady workloads,
autohas JVM allocation close tofalse: short text uses6.12 versus 6.08 MiB; long text 3.77 versus 3.85 MiB; nested data 17.33 versus 17.14 MiB;
maps 3.28 versus 3.22 MiB.
falseremains a good option for known steady data because itavoids the utilization check. For
nested-long,falsemeasured 121 ms versus 135 ms forauto; timing ranges overlap and these are short local runs.Sampled retained Arrow in MiB shows the protection and its cost:
The reason to keep
autoas the default is protection against severe capacity inflation whenvalues change and the byte limit is unset, while retaining useful sizing on steady data.
It allows up to four times the observed used bytes and can retain one inflated batch after
usage drops.
trueretains the aggressive reset option. The byte-limited cases benefit allmodes equally and do not justify choosing
autooverfalse.Benchmark harness, full 23-case allocation/memory/timing matrices and reproduction commands:
benchmark branch.
Raw medians, timing ranges and source/library hashes:
adaptive results.
Timings are three short local runs with a discarding sink. Some ranges are wide, and sampled
native retention depends on scheduling. Treat speed differences as exploratory.
Validation
Spark 3.5/Scala 2.13 and Spark 4.0/Scala 2.13 (207 tests total).
for strings, binary, lists, maps and struct fields, with value readback and buffer release.
unconditional resets after byte-triggered flushes in all three modes.
confirmed buffer-lifetime and reset semantics.
and byte-triggered resets; all output byte counts match across modes.