You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Check if struct type has duplicate keys when converting to ArrowSchema #2457
In the failed test case of #2444, I found that the struct type with duplicate keys will be deduplicated when converted to arrowType, which causes RowToColumanr to lose some columns.
error:
Job aborted due to stage failure: Task 0 in stage 7.0 failed 1 times, most recent failure: Lost task 0.0 in stage 7.0 (TID 9) (10.5.155.10 executor driver): java.lang.IndexOutOfBoundsException: Index 2 out of bounds for length 2
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
at java.base/java.util.Objects.checkIndex(Objects.java:361)
at java.base/java.util.ArrayList.get(ArrayList.java:427)
at org.apache.comet.vector.CometStructVector.getChild(CometStructVector.java:55)
at org.apache.spark.sql.vectorized.ColumnarRow.isNullAt(ColumnarRow.java:101)
This is actually related to the default CONFLICT_REPLACE conflict policy of StructVector, which will replace the existing field with the newer one when there is a conflict. We can change this behavior by adding -Darrow.struct.conflict.policy=CONFLICT_APPEND.
This appears to have been fixed through multiple layers of validation that now prevent duplicate struct field names from reaching the Arrow schema conversion:
CreateNamedStruct (structs.scala) — rejects duplicate field names, falls back to Spark (commit abd9f85)
Shuffle (QueryPlanSerde.scala) — rejects structs with duplicate field names (commit 9203f17)
@andygrove Thanks for providing this informations. Duplicate fields should be a relatively rare case, and falling back to spark when encountering them should be sufficient. I will close this issue.
Activity
wForget commentedon Sep 26, 2025
This is actually related to the default
CONFLICT_REPLACEconflict policy of StructVector, which will replace the existing field with the newer one when there is a conflict. We can change this behavior by adding-Darrow.struct.conflict.policy=CONFLICT_APPEND.https://github.com/apache/arrow-java/blob/f38e72f5a46e30f44c28ff407eb522a5e5c266c0/vector/src/main/java/org/apache/arrow/vector/complex/AbstractStructVector.java#L65-L75
wForget commentedon Sep 26, 2025
But I would prefer to reject this behavior, like spark does.
https://github.com/apache/spark/blob/cb283404d3b229f18b8508793098aa6759af46a7/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala#L212
andygrove commentedon Mar 27, 2026
This appears to have been fixed through multiple layers of validation that now prevent duplicate struct field names from reaching the Arrow schema conversion:
structs.scala) — rejects duplicate field names, falls back to Spark (commit abd9f85)QueryPlanSerde.scala) — rejects structs with duplicate field names (commit 9203f17)CometScanRule.scala) — rejects duplicates in case-insensitive mode (commit f6d84b1)errors.rs) — converts DataFusion deduplication errors to proper Spark errors instead of crashing (commit 9929745)@wForget can this issue be closed?
wForget commentedon Mar 30, 2026
@andygrove Thanks for providing this informations. Duplicate fields should be a relatively rare case, and falling back to spark when encountering them should be sufficient. I will close this issue.