Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
Run test test("basic parquet write") and output the plan
*(1) CometColumnarToRow
+- CometNativeWrite children {
plan_id: 43
scan {
fields {
}
fields {
type_id: INT8
}
fields {
type_id: INT16
}
fields {
type_id: INT32
}
fields {
type_id: INT64
}
fields {
type_id: FLOAT
}
fields {
type_id: DOUBLE
}
fields {
type_id: DECIMAL
type_info {
decimal {
precision: 10
scale: 2
}
}
}
fields {
type_id: DECIMAL
type_info {
decimal {
precision: 36
scale: 18
}
}
}
fields {
type_id: DATE
}
fields {
type_id: TIMESTAMP
}
fields {
type_id: TIMESTAMP_NTZ
}
fields {
type_id: STRING
}
fields {
type_id: BYTES
}
source: "WriteFiles"
}
}
plan_id: 43
parquet_writer {
output_path: "file:/private/var/folders/6f/_s1vnnd55zgfkx7zlwnrnv0h0000gn/T/spark-00eba07b-474a-4bc4-a277-b3ef73dc3511/output.parquet"
compression: Snappy
column_names: "c0"
column_names: "c1"
column_names: "c2"
column_names: "c3"
column_names: "c4"
column_names: "c5"
column_names: "c6"
column_names: "c7"
column_names: "c8"
column_names: "c9"
column_names: "c10"
column_names: "c11"
column_names: "c12"
column_names: "c13"
}
, file:/private/var/folders/6f/_s1vnnd55zgfkx7zlwnrnv0h0000gn/T/spark-00eba07b-474a-4bc4-a277-b3ef73dc3511/output.parquet, org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol@402d76c7, 1c0d1729-74bb-41ad-96a1-f743b308ba3e
+- CometScan [native_iceberg_compat] parquet [c0#56,c1#57,c2#58,c3#59,c4#60L,c5#61,c6#62,c7#63,c8#64,c9#65,c10#66,c11#67,c12#68,c13#69] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/6f/_s1vnnd55zgfkx7zlwnrnv0h0000gn/T/spark-2a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c0:boolean,c1:tinyint,c2:smallint,c3:int,c4:bigint,c5:float,c6:double,c7:decimal(10,2),c8:...
The transition node shouldnot be there.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
[-]The Comet writer adds a trailing `Comet`[/-][+]The Comet writer adds unnecessary transition `CometColumnarToRow`[/+]comphead commentedon Dec 22, 2025