Repository navigation
Description
Problem. #6031 (for #6207) gave native Parquet reads one credential per policy location through CometS3LocationScopedCredentialProvider and left the Iceberg path unchanged. On that path a provider still gets one credential per table.
load_file_io builds each FileIO with a credential bridge bound to one reference path. For scans that is the table's metadata location (iceberg_scan.rs:181); for writes it is the data location (iceberg_write.rs:484). The bridge sends that bucket and path on every getCredentialsForPath call (iceberg_common.rs:351). iceberg-rust then attaches that same loader to the operator it builds for every file (s3_config_build in iceberg-storage-opendal). As a result, every data and delete file a scan reads, and every file a write produces, is signed with the reference path's credential.
That is correct when one policy covers everything a table touches. It is wrong when a table's files span locations with different policies:
- Data files outside the table location. This includes
write.data.path, the object-storage layout, and files added byadd_files,migrateorsnapshotthat stay where they were. - A narrower policy nested under the table location, for example a separate policy for
warehouse/db/t/data/region=eu. - Data files in another bucket, because the bridge's bucket is also fixed by the reference path.
Reads of those files fail with 403 when the reference credential does not cover them. When a broader credential does cover a path that a narrower policy restricts, the read succeeds even though the provider's own per-path answer for that file would refuse it.
Proposal. Honor getPolicyLocations on the Iceberg path too. No new API is needed.
- When the configured provider is location-scoped,
storage_factory_forreturns a CometStorageFactorythat wrapsOpenDalStorageFactory::S3, the same wayBlobHostPromotingS3StorageFactorydoes today. - Every iceberg-rust
Storagemethod receives the path it operates on. The wrapper routes each call by that path to the longest covering location in the path's bucket, using the routingLocationScopedObjectStorealready has. It then delegates to an inner OpenDAL S3 storage whose loader is a bridge bound to that location. Inner storages are built on first use. new_inputandnew_outputreturn files bound to the wrapper, so every read goes through the routing.- A read that fails with 403 refreshes the bucket's locations, at most once per snapshot. It retries if the path now routes to a different location, as on the Parquet path. On this path a provider exception reaches S3 as an unsigned request, so a 403 is the signal for both cases. Writes and deletes route by path without retry.
- Providers that implement only the base interface keep today's behavior.
Compatibility. This adds no public types or methods, but it changes documented behavior. The CometS3LocationScopedCredentialProvider Javadoc and the user guide both say Iceberg reads do not use locations. After this change, location-scoped providers receive:
getPolicyLocationscalls for the buckets Iceberg reads and writes touch;- location paths in
getCredentialsForPathcalls from the Iceberg path.
Both fall within the existing contract: a location's credential must authorize every path the location is the longest match for, and getPolicyLocations may be called from any thread on the driver or executors. Maintainers may still prefer to put the change behind a config for one release.
Related. #6207 and #6031 cover the Parquet path. #6293 covers blocking JVM calls on Tokio workers; some of the getPolicyLocations and bridge calls this adds would run inside async Storage methods. A design sketch follows in a comment.
Activity
snmvaughan commentedon Sep 30, 2026
Design sketch: location-scoped credentials on the Iceberg path
Code references are against
apache/datafusion-cometmainat9603ad128andapache/iceberg-rustatbb1e4a4, the revision Comet pins.Current behavior
The Parquet path routes each request to a policy location (
parquet/objectstore/location_scoped.rs). The Iceberg path does not:load_file_io(execution/operators/iceberg_common.rs:226) builds aFileIOfrom a reference path. That path is the metadata location for scans (iceberg_scan.rs:181) and the data location for writes (iceberg_write.rs:484).build_s3_credential_loader(iceberg_common.rs:308) creates oneCometS3CredentialBridgefrom the reference path's bucket andurl.path()(iceberg_common.rs:351), then wraps it in aCustomAwsCredentialLoader.s3_config_build(crates/storage/opendal/src/s3.rs:129) builds an operator for each file operation. It takes the bucket from the file's path but pushes the same loader onto every operator's credential chain (s3.rs:149).provide_credential(cloud/s3/credential_bridge.rs:376) always callsgetCredentialsForPathwith the reference bucket and path.FileIOcache (iceberg_common.rs:108, 64 entries) keepsreference_pathin its key, so two tables never share a bridge.The result is one credential identity per table and access mode. The bridge's own doc comment describes this as "per-table-location (Iceberg)" (
credential_bridge.rs:69).Goals
getPolicyLocations.CometS3CredentialProvider.Non-goals
getCredentialsForPath; what is kept is the location list and one inner storage per location that has been used.Design
Where it plugs in
storage_factory_for(iceberg_common.rs:57) already substitutes a Comet factory for S3-compatible aliases.BlobHostPromotingS3StorageFactoryandBlobHostPromotingS3Storage(parquet/objectstore/s3_blob_fs_support.rs:181,:210) wrapOpenDalStorageFactory::S3and delegate everyStoragemethod by path. The new pair follows the same pattern:LocationScopedS3StorageFactoryis#[typetag::serde(name = "CometLocationScopedS3StorageFactory")]. Its runtime state is#[serde(skip)], ascustomized_credential_loadalready is. Serde exists only to satisfy theStoragesupertraits; Comet never serializes storage during a scan.LocationScopedS3StorageimplementsStorage(iceberg-rustcrates/iceberg/src/io/storage/mod.rs:72).build_s3_credential_loadercreates the reference bridge as it does today, then asks it forpolicy_locations(), ascreate_storedoes on the Parquet path.Nonemeans a base provider. Keep today's single-loaderOpenDalStorageFactory::S3.Some(locations)means a location-scoped provider. Return the new factory, seeded with the reference bucket's snapshot.For an S3-compatible alias scheme, the host-promotion wrapper stays outermost, so routing sees the promoted host as the bucket.
Routing
State per storage instance:
LocationIndex: canonical location mapped to credential path, plus a generation counter and the last refresh failure.(bucket, credential_path)map of innerArc<dyn Storage>.For each call, the storage:
pathinto a bucket and key.LocationIndex::route(location_scoped.rs:110): longest covering location, compared segment by segment after percent-decoding, with/as the implicit root.(bucket, credential_path).LocationIndexandrouteare currently private tolocation_scoped.rsand keyed byobject_store::path::Path. Move them to a small shared module used by both paths. KeepPathfor canonicalization, since it is already the decoding rule the SPI documents.Inner storages and bridges
An inner storage is
OpenDalStorageFactory::S3 { customized_credential_load: Some(loader) }.build(config). Itsloaderwraps a bridge bound to(bucket, credential_path). It is built on first use and kept for the life of the outer storage.Bridges are derived from the reference bridge, so they reuse its dispatcher handle and trigger no
ensureInitializedcall or class loading on the Tokio worker that usually builds them:for_path(credential_bridge.rs:144).for_location(bucket, path)derivation. On the Iceberg path the dispatch key is the catalog name, so one registration already serves every bucket a table touches.Snapshot lifecycle and blocking
build_s3_credential_loader. That runs on the thread that callsload_file_io, the same place the Parquet path'screate_storefetches.Storagemethod, on a Tokio worker. That is a blocking JVM call of the kind #6293 describes, so wrap it intokio::task::block_in_placeas #6261 does foracquireMemory. The per-requestgetCredentialsForPathcalls are already in #6293's list and are unchanged here.FileIOin the executor cache. A cache hit therefore reuses the snapshot until a 403 refreshes it, or until the entry is evicted and rebuilt.Refresh and retry
The semantics mirror
LocationScopedObjectStore:refreshatlocation_scoped.rs:225and the per-generation sharing described in the contributor guide.CredentialProviderError, but here reqsign'sProvideCredentialChainswallows provider exceptions. This is the existing "error message fidelity" caveat: the request goes out unsigned and S3 rejects it with 403. The storage must detect permission-denied through the source chain of iceberg'sError, which wraps opendal'sErrorKind::PermissionDenied.exists,metadata,read, and eachFileRead::read.reader()andnew_input()must stay retryable after they return.new_input(path)therefore returnsInputFile::new(Arc<Self>, path)(iceberg-rustfile_io.rs:331) rather than the inner storage'sInputFile.reader(path)returns aFileReadwrapper that holds the current route. On a 403 it refreshes, reopens through the new route if the route changed, and retries the range once.Writes and deletes
write,writer, andnew_outputroute by path withAccessMode::Writebridges, and do not retry, matching the Parquet store's non-read operations.deleteanddelete_streamroute per path.delete_streamgroups paths by route before delegating.delete_prefixroutes by the prefix itself, aslistdoes on the Parquet path. If the prefix spans several locations, the single credential may not cover all of them. That is acceptable for the current callers, but see the open questions.FileIO cache
The cache key stays as it is. The base provider still needs
reference_pathin the key, and keeping it for location-scoped providers is harmless: at worst, each cachedFileIOholds its own snapshot. A follow-up could dropreference_pathfrom the key for location-scoped providers, so tables in one catalog share a snapshot.Documentation to update
CometS3LocationScopedCredentialProviderJavadoc, line 43: "Locations apply to Comet's native Parquet reads. Iceberg reads do not use them."s3-credential-providers.md, line 300: "Locations apply to Comet's native Parquet reads only".s3-credential-provider-design.md: line 91 ("The Iceberg path does not use locations"), the "Path-specific behavior" table, and a new "Location-scoped credentials on the Iceberg path" section based on this document.CometS3CredentialBridgedoc comment's "per-table-location (Iceberg)" (credential_bridge.rs:69).Testing
Rust unit tests (
native/core), using a fake innerStoragethat records(bucket, credential_path, operation):Storagemethod routes by its path, including longest-match, root fallback, and percent-decoded paths.new_inputreturns a file bound to the wrapper, and its reads route and retry.for_location, with no secondensureInitialized.FileReadreopens through the new route on a 403.null) still gets today's single-loader factory.Integration tests (
CometS3CredentialBridgeSuite, MinIO,MinioLocationScopedCredentialProvider). MinIO does not enforce per-prefix policies here, so, as the Parquet test does, assert on the recorded credential paths rather than on 403s.credentialPaths()contains each location's credential path, not only the metadata file's.write.data.pathunder a different installed location, and assert its reads use that location's credential path.CometS3CredentialBridgeSuite.scala:108) is unchanged.Alternatives
s3_config_buildalready receives the path (s3.rs:129).iceberg-storage-opendalcould accept a loader factory keyed by bucket and path instead of one loader. That is cleaner for other embedders, but it needs an upstream release and a revision bump, and Comet would still own routing and refresh. It could be adopted later without changing the SPI.Open questions
reference_pathbe dropped from theFileIOcache key for location-scoped providers?delete_prefixfan out across the locations under the prefix? This matters only if a caller deletes a prefix that spans policies.