You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg writes: support hdfs data locations #7637
Native Iceberg writes fall back to iceberg-java for an hdfs data location. The gate in CometIcebergNativeWrite admits no hdfs scheme, so every write to an Iceberg table on HDFS runs on the JVM writer.
This no longer waits on iceberg-rust. apache/iceberg-rust#3111 added the hdfs-native storage backend, and #5898 already reads Iceberg tables on HDFS natively through it (#5894). The remaining work is on Comet's side.
Describe the potential solution
Admit hdfs data locations in the native write gate, and route writes through the same hdfs-native backend and configuration the native scan uses. Keep the cases the native scan declines falling back, in particular a cluster with Kerberos and a location whose nameservice cannot be resolved from the Hadoop configuration.
Additional context
Part of #5649. Recorded under "Plans for each fallback" in the Iceberg user guide (#6961).
Activity