Skip to content

fix(jni): retain Java Arrow buffers independently - #11069

Merged
robert3005 merged 6 commits into
vortex-data:developfrom
quantcast:cwirt/upstream-jni-independent-buffers
Nov 3, 2026
Merged

robert3005 merged 6 commits into
vortex-data:developfrom
quantcast:cwirt/upstream-jni-independent-buffers

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Tracking issue: #11068

Problem: An Arrow C Data Interface import gives every buffer in a batch one shared release callback. The writer keeps small slices of each batch (e.g. validity bitmaps) until finish, so each slice keeps its whole batch allocated. In a 615-column Spark job this held ~4.7 GB until finish.

Fix: A new VortexWriter.writeBatch(VectorSchemaRoot, BufferAllocator) overload retains each Java Arrow allocation separately. It points the imported Rust buffers at those owners, with the same addresses and no copy. A retained bitmap now keeps only its own allocation alive. Spark uses the new overload.

Result: In a synthetic nested-string workload (16 × 8,192 rows), memory held before finish drops from ~65 MiB to 8.1 MiB. Everything is released at finish. The production workload has not been rerun yet.

Changes

  • Java: Add the writeBatch(root, allocator) overload. It validates the root against the writer's stored schema, so there is no schema export per batch.
  • Rust (vortex-jni/src/buffer): Retain each Java allocation and make it the owner of the matching imported buffers. The last Rust reference releases it, including from native worker threads.
    • Arrow's alignment copies of unaligned inputs are kept.
    • A buffer with no matching Java owner fails the import instead of quietly keeping the whole batch.
  • Spark: VortexOutputWriter uses the new overload.
  • Fixes: Both writeBatch overloads are now serialized with finish. The builder now releases its temporary ArrowSchema (previously leaked).

API notes

  • writeBatch(long, long) is unchanged and still uses the producer's shared release callback.
  • Dictionary-encoded roots still require the address-based API.
  • Buffers backed by the same Java allocation still share its lifetime.

Testing

New tests cover independent buffer lifetimes, cleanup after a partial import or sink failure, empty batches, string views, unaligned inputs, schema mismatch, release from native threads, and concurrent write/finish. The string and nested-map round-trip tests now run against both APIs.

On the fork (quantcast#49, same change):

  • JNI tests: 18 passed (testParquetVariantRoundTrip was already @Disabled)
  • Rust ownership tests: 3 passed
  • Spark 2.13 VortexDataSourceWriteTest: 11 passed
  • Clippy (-D warnings), pinned rustfmt, Java format checks and git diff --check passed

This branch ports that change onto develop. The Spark conflict was resolved by keeping upstream's row-building code and replacing only the export/write block. Only git diff --check has been run on the port.

Known issue (not fixed here)

String-view concatenation reads metadata from null view slots. The view round-trip test initializes those slots to work around it.

AI assistance

Disclosed per the AI policy: AI tools helped write and review the source change. Codex prepared this upstream port, issue and PR description at my request. I am the responsible contributor.

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two comments

Comment thread vortex-jni/src/buffer/mod.rs Resolved
@robert3005

Copy link
Copy Markdown
Contributor

We can merge this once you rebase - I have been merging fixes to the jni bindings

ChristopherWirt and others added 3 commits November 2, 2026 19:06
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Only native code should retain or release the wrapped allocation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt
ChristopherWirt force-pushed the cwirt/upstream-jni-independent-buffers branch from 46de578 to 8e2e3b6 Compare November 2, 2026 19:06
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 2, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 2, 2026 19:10
@codspeed

codspeed Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1877 untouched benchmarks
⏩ 468 skipped benchmarks1


Comparing quantcast:cwirt/upstream-jni-independent-buffers (8793582) with develop (eed0212)

Open in CodSpeed

Footnotes

  1. 468 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
auto-merge was automatically disabled November 2, 2026 20:28

Head branch was pushed to by a user without write access

@robert3005
robert3005 enabled auto-merge (squash) November 3, 2026 12:19
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
auto-merge was automatically disabled November 3, 2026 13:17

Head branch was pushed to by a user without write access

@robert3005
robert3005 merged commit 0b0ac41 into vortex-data:develop Nov 3, 2026
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants