Skip to content

fix: [branch-1.2] split a coalesced broadcast before a string or binary vector passes 1 GiB (#7193) - #7646

Merged
andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-7193-branch-1.2
Nov 17, 2026
Merged

andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-7193-branch-1.2

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #7193 to branch-1.2. Closes #7192 on this branch.

Rationale for this change

coalesceBroadcastBatches appends the whole broadcast into one VectorSchemaRoot. Arrow Java's VectorAppender grows a string or binary data buffer by doubling it and cannot go past 1 GiB, so a broadcast join fails with OversizedAllocationException once one such column of the build side holds more than that, where Spark runs the same query. branch-1.2 has the same unbounded coalescing in Utils.coalesceBroadcastBatches. See #7193 and #7192 for the full analysis and repro.

What changes are included in this PR?

Cherry-pick of 4e54f1c (git cherry-pick -x) with no conflicts and no adaptations. coalesceBroadcastBatches starts a new coalesced batch when the next input batch would take a string or binary vector (nested ones included) past 2^30 bytes of values, and an OversizedAllocationException from the appender now leaves the broadcast uncoalesced. A broadcast below that size is still one batch. It includes the source PR's UtilsSuite test.

Other release branches: branch-1.2 is the newest release branch, so there is no newer one to port to. Whether branch-1.1 needs a backport is not part of this PR.

How are these changes tested?

On this branch, with JDK 17 and the default Spark profile (4.1): ./mvnw test-compile passes, and UtilsSuite passes (15 tests, 0 failures) including the new test. I did not rerun the 1 GiB repro from #7192. The test cannot fail on branch-1.2 without the fix in the usual way, because it passes the new maxValueBytes parameter, so with spark/src/main reverted to branch-1.2's version the test does not compile. I ran the default Spark profile only.

comphead and others added 2 commits November 16, 2026 10:33
…ses 1 GiB (apache#7193)

coalesceBroadcastBatches appended the whole broadcast into one VectorSchemaRoot. Arrow Java's
VectorAppender grows a string or binary data buffer by doubling it and cannot go past 1 GiB, so a
broadcast join failed with OversizedAllocationException once one such column of the build side
held more than that (apache#7192).

Start a new coalesced batch when the next input batch would take a string or binary vector,
nested ones included, past 2^30 bytes of values. A broadcast below that is still one batch. An
OversizedAllocationException from the appender now leaves the broadcast uncoalesced, as an
IllegalArgumentException already does.

(cherry picked from commit 4e54f1c)

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@andygrove
andygrove merged commit c24668b into apache:branch-1.2 Nov 17, 2026
138 checks passed
@andygrove
andygrove deleted the backport-7193-branch-1.2 branch November 17, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants