Skip to content

Check if struct type has duplicate keys when converting to ArrowSchema #2457

Closed

Description

@wForget

Describe the bug

In the failed test case of #2444, I found that the struct type with duplicate keys will be deduplicated when converted to arrowType, which causes RowToColumanr to lose some columns.

error:

Job aborted due to stage failure: Task 0 in stage 7.0 failed 1 times, most recent failure: Lost task 0.0 in stage 7.0 (TID 9) (10.5.155.10 executor driver): java.lang.IndexOutOfBoundsException: Index 2 out of bounds for length 2
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
	at java.base/java.util.Objects.checkIndex(Objects.java:361)
	at java.base/java.util.ArrayList.get(ArrayList.java:427)
	at org.apache.comet.vector.CometStructVector.getChild(CometStructVector.java:55)
	at org.apache.spark.sql.vectorized.ColumnarRow.isNullAt(ColumnarRow.java:101)

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. wForget commented on Sep 26, 2025

    @wForget
    MemberAuthor

    This is actually related to the default CONFLICT_REPLACE conflict policy of StructVector, which will replace the existing field with the newer one when there is a conflict. We can change this behavior by adding -Darrow.struct.conflict.policy=CONFLICT_APPEND.

    https://github.com/apache/arrow-java/blob/f38e72f5a46e30f44c28ff407eb522a5e5c266c0/vector/src/main/java/org/apache/arrow/vector/complex/AbstractStructVector.java#L65-L75

  2. added
    priority:highCrashes, panics, segfaults, major functional breakage
    on Mar 27, 2026
  3. andygrove commented on Mar 27, 2026

    @andygrove
    Member

    This appears to have been fixed through multiple layers of validation that now prevent duplicate struct field names from reaching the Arrow schema conversion:

    1. CreateNamedStruct (structs.scala) — rejects duplicate field names, falls back to Spark (commit abd9f85)
    2. Shuffle (QueryPlanSerde.scala) — rejects structs with duplicate field names (commit 9203f17)
    3. Native DataFusion scan (CometScanRule.scala) — rejects duplicates in case-insensitive mode (commit f6d84b1)
    4. Rust error handling (errors.rs) — converts DataFusion deduplication errors to proper Spark errors instead of crashing (commit 9929745)

    @wForget can this issue be closed?

  4. wForget commented on Mar 30, 2026

    @wForget
    MemberAuthor

    @andygrove Thanks for providing this informations. Duplicate fields should be a relatively rare case, and falling back to spark when encountering them should be sufficient. I will close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:highCrashes, panics, segfaults, major functional breakage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions