Skip to content

Native Iceberg writes: support hdfs data locations #7637

Open

Description

@andygrove

What is the problem the feature request solves?

Native Iceberg writes fall back to iceberg-java for an hdfs data location. The gate in CometIcebergNativeWrite admits no hdfs scheme, so every write to an Iceberg table on HDFS runs on the JVM writer.

This no longer waits on iceberg-rust. apache/iceberg-rust#3111 added the hdfs-native storage backend, and #5898 already reads Iceberg tables on HDFS natively through it (#5894). The remaining work is on Comet's side.

Describe the potential solution

Admit hdfs data locations in the native write gate, and route writes through the same hdfs-native backend and configuration the native scan uses. Keep the cases the native scan declines falling back, in particular a cluster with Kerberos and a location whose nameservice cannot be resolved from the Hadoop configuration.

Additional context

Part of #5649. Recorded under "Plans for each fallback" in the Iceberg user guide (#6961).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions