Repository navigation
API: Prevent intermediate overflow in integer truncation - #18112
laserninja wants to merge 3 commits into
Conversation
Use a nonnegative remainder without overflowing the width addition so matching partitions are not incorrectly pruned. Fixes apache#18111 Generated-by: Codex
Verify that SQL inserts remain visible to range filters across Parquet, Avro, and ORC. Generated-by: Codex
Fokko
left a comment
There was a problem hiding this comment.
It is a bit of an edge case, but I believe it is a bug. Thanks @laserninja for fixing this. Can you follow up on my comment?
| } | ||
|
|
||
| @TestTemplate | ||
| void scanWithLargeIntegerTruncateWidth() { |
There was a problem hiding this comment.
Can we add this one for the other Spark versions as well?
There was a problem hiding this comment.
Added the regression test for Spark 3.5, 4.0, and 4.2. All nine added cases passed across Parquet, Avro, etc. Updated validation notes too.
Add the scan regression to Spark 3.5, 4.0, and 4.2 so each supported engine version verifies matching rows survive partition pruning. Generated-by: Codex
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
wgtmac
left a comment
There was a problem hiding this comment.
We have a similar fix at apache/iceberg-cpp#1042 but I think they are still partial fixes and need more clarification on the spec side.
|
|
||
| public static byte truncateByte(int width, byte value) { | ||
| return (byte) (value - (((value % width) + width) % width)); | ||
| return (byte) (value - Math.floorMod(value, width)); |
There was a problem hiding this comment.
Line 656 in e5f8ab1
This fixes the implementation, but the spec above still doesn’t say the arithmetic must use unbounded integers. Could we clarify the spec as r = floorMod(v, W) and v - r computed with unbounded arithmetic? Otherwise other implementations can still produce different partition values.
There was a problem hiding this comment.
Another issue is that truncateInt(10, Integer.MIN_VALUE) still returns Integer.MAX_VALUE - 1 after this PR: the final mathematical result is outside int, so wrapping breaks ordering and can cause incorrect partition pruning as reported in #9767 and #13105.
Do we need a spec change for clarification?
There was a problem hiding this comment.
+1 to updating the spec accordingly.
Fixes #18111.
Truncating
Integer.MAX_VALUE - 1with widthInteger.MAX_VALUEcurrently produces a negative partition key. A scan filtered byid >= 0then omits the matching file.Use
Math.floorModin the int and narrow-integer helpers to avoid overflowing the intermediate remainder-plus-width addition. This preserves public APIs and existing behavior when the final mathematical result is outside the target type's range.Validation on JDK 17:
:iceberg-api:testsuite and targeted core truncation and table scan regressions passed. The new regressions failed before the fix.:iceberg-api:spotlessCheck :iceberg-core:spotlessCheckpassed.:iceberg-api:revapi :iceberg-core:revapipassed.Spark SQL regression (Spark 4.1): create an integer table partitioned by
truncate(2147483647, id), insert2147483646, and queryWHERE id >= 0. On the original implementation the query returns no rows in Parquet, Avro, and ORC; with the fix it returns the inserted row in all three formats. The new Spark test, formatting, and Spark test checkstyle pass locally on JDK 17.Additional local validation at
28aff7bfb: the full CI all-module build command (./gradlew -DallModules build -x test -x javadoc -x integrationTest) passed on JDK 17, including formatting and style checks across the engine versions. Apache RAT (dev/check-license),./gradlew checkAllRuntimeDeps -q -DallModules=true, and./gradlew -Pquick=true javadocalso passed. This build excludes tests; the Spark regression results above are from a separate test run.Review follow-up: the same Spark SQL regression now covers Spark 3.5 (Scala 2.12), 4.0, and 4.2 (Scala 2.13), in addition to 4.1. All nine added version/format cases passed locally on JDK 17, together with
spotlessCheckandcheckstyleTestfor the three changed Spark modules.AI Disclosure