Skip to content

The Comet writer adds unnecessary transition CometColumnarToRow #2958

Closed

Description

@comphead

Describe the bug

Run test test("basic parquet write") and output the plan

*(1) CometColumnarToRow
+- CometNativeWrite children {
  plan_id: 43
  scan {
    fields {
    }
    fields {
      type_id: INT8
    }
    fields {
      type_id: INT16
    }
    fields {
      type_id: INT32
    }
    fields {
      type_id: INT64
    }
    fields {
      type_id: FLOAT
    }
    fields {
      type_id: DOUBLE
    }
    fields {
      type_id: DECIMAL
      type_info {
        decimal {
          precision: 10
          scale: 2
        }
      }
    }
    fields {
      type_id: DECIMAL
      type_info {
        decimal {
          precision: 36
          scale: 18
        }
      }
    }
    fields {
      type_id: DATE
    }
    fields {
      type_id: TIMESTAMP
    }
    fields {
      type_id: TIMESTAMP_NTZ
    }
    fields {
      type_id: STRING
    }
    fields {
      type_id: BYTES
    }
    source: "WriteFiles"
  }
}
plan_id: 43
parquet_writer {
  output_path: "file:/private/var/folders/6f/_s1vnnd55zgfkx7zlwnrnv0h0000gn/T/spark-00eba07b-474a-4bc4-a277-b3ef73dc3511/output.parquet"
  compression: Snappy
  column_names: "c0"
  column_names: "c1"
  column_names: "c2"
  column_names: "c3"
  column_names: "c4"
  column_names: "c5"
  column_names: "c6"
  column_names: "c7"
  column_names: "c8"
  column_names: "c9"
  column_names: "c10"
  column_names: "c11"
  column_names: "c12"
  column_names: "c13"
}
, file:/private/var/folders/6f/_s1vnnd55zgfkx7zlwnrnv0h0000gn/T/spark-00eba07b-474a-4bc4-a277-b3ef73dc3511/output.parquet, org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol@402d76c7, 1c0d1729-74bb-41ad-96a1-f743b308ba3e
   +- CometScan [native_iceberg_compat] parquet [c0#56,c1#57,c2#58,c3#59,c4#60L,c5#61,c6#62,c7#63,c8#64,c9#65,c10#66,c11#67,c12#68,c13#69] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/6f/_s1vnnd55zgfkx7zlwnrnv0h0000gn/T/spark-2a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c0:boolean,c1:tinyint,c2:smallint,c3:int,c4:bigint,c5:float,c6:double,c7:decimal(10,2),c8:...

The transition node shouldnot be there.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. changed the title [-]The Comet writer adds a trailing `Comet`[/-] [+]The Comet writer adds unnecessary transition `CometColumnarToRow`[/+] on Dec 22, 2025
  2. comphead commented on Dec 22, 2025

    @comphead
    ContributorAuthor
    Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions