Repository navigation
Description
Describe the bug
CometScanRule.validateIcebergFileScanTasks checks Iceberg file schemes against a hardcoded set (file, s3, s3a, gs, gcs, oss, abfss, abfs, wasbs, wasb at CometScanRule.scala:1082), and its fallback message repeats that list to users. But the Iceberg execution path resolves storage through its own factory (iceberg_scan.rs storage_factory_for), which supports only file, s3, s3a, gs, oss via OpenDAL. And the JNI probe every other scheme gate delegates to (isNativelyReadableScheme → object_store::ObjectStoreScheme::parse) recognizes a third, different set.
Net effect: an Iceberg table under gcs, abfs, abfss, wasb, or wasbs passes validation, gets claimed, and every task dies with CometNativeException: Unsupported storage scheme: ... — while stock Spark reads the same table fine. Meanwhile oss actually works at execution despite the JNI probe saying it's unsupported, so the three lists disagree in both directions.
Steps to reproduce
No cloud account needed — back the scheme with local disk the way ParquetReadFromFakeHadoopFsSuite does:
// register a local-backed filesystem for the wasb scheme
spark.conf.set("spark.hadoop.fs.wasb.impl", "org.apache.hadoop.fs.RawLocalFileSystem")
// hadoop catalog with a wasb warehouse
// spark.sql.catalog.hadoop_catalog = org.apache.iceberg.spark.SparkCatalog
// spark.sql.catalog.hadoop_catalog.type = hadoop
// spark.sql.catalog.hadoop_catalog.warehouse = wasb://fake-container/tmp/warehouse
sql("CREATE TABLE hadoop_catalog.t (id INT, s STRING) USING iceberg")
sql("INSERT INTO hadoop_catalog.t VALUES (1,'a'),(2,'b'),(3,'c')")
sql("SELECT * FROM hadoop_catalog.t").collect()
With spark.comet.enabled=false: [1,a],[2,b],[3,c].
With Comet + spark.comet.scan.icebergNative.enabled=true (plan shows CometIcebergNativeScan ... wasb://fake-container/...): every task fails with
org.apache.comet.CometNativeException: Unsupported storage scheme: wasb
(Ensure spark.comet.libhdfs.schemes doesn't include wasb, since that conf routes schemes elsewhere.)
Expected behavior
Schemes the Iceberg native path can't execute should be declined at plan time so the scan falls back to Spark, and the fallback message should list what's actually supported. One source of truth: the validator should ask the same layer that executes — either expose the Iceberg storage factory's supported set through the existing JNI probe pattern, or derive the check from it directly — rather than maintaining a third hand-written list.
Additional context
The claimed-but-fails set on this path is {gcs, abfs, abfss, wasb, wasbs}; oss works at execution but is rejected by the generic JNI probe, so aligning the lists fixes both directions. Same drift class as the scheme/config gates reworked in #5365.
Activity
abhinav-phi commentedon Aug 29, 2026
Hi! I'd like to work on this.
Plan: I'll review the scheme allowlist in the Iceberg native scan path, make it strictly report/decline schemes it can't actually execute instead of claiming them, and add tests for each supported/unsupported scheme.
Could you please assign this to me? Thanks!
take
/assign