Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pr_build_linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,7 @@ jobs:
org.apache.comet.rules.CometScanRuleSuite
org.apache.comet.rules.CometScanContribSuite
org.apache.comet.rules.CometScanSchemeFallbackSuite
org.apache.comet.cloud.s3.CometS3CredentialReleaseSuite
org.apache.comet.rules.CometMultiStoreScanSuite
org.apache.comet.rules.CometExecRuleSuite
org.apache.spark.sql.comet.CometPlanEqualitySuite
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/pr_build_macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,7 @@ jobs:
org.apache.comet.rules.CometScanRuleSuite
org.apache.comet.rules.CometScanContribSuite
org.apache.comet.rules.CometScanSchemeFallbackSuite
org.apache.comet.cloud.s3.CometS3CredentialReleaseSuite
org.apache.comet.rules.CometMultiStoreScanSuite
org.apache.comet.rules.CometExecRuleSuite
org.apache.spark.sql.comet.CometPlanEqualitySuite
Expand Down
17 changes: 13 additions & 4 deletions docs/source/contributor-guide/s3-credential-provider-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,20 +158,29 @@ The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properti

## Built-in adapters

Comet ships two reference SPI implementations under `org.apache.comet.cloud.s3`, so standard provider classes that the native Rust list does not match work with a config change instead of bespoke code:
Comet ships three SPI implementations under `org.apache.comet.cloud.s3`, so standard provider classes that the native Rust list does not match work with a config change instead of bespoke code:

- `HadoopS3ACredentialProviderAdapter` delegates to Hadoop S3A's own provider construction (`S3AUtils.createAWSCredentialProviderSet` on Hadoop 3.3.x, `CredentialProviderListFactory.createAWSCredentialProviderList` on 3.4+).
- `AwsSdkCredentialProviderAdapter` wraps a raw AWS SDK provider named in `fs.s3a.comet.credential.adapter.class`.
- `IcebergVendedS3CredentialProvider` resolves REST-vended Iceberg credentials through Iceberg's `AwsClientProperties`, as `S3FileIO` does, and refreshes them through `client.refresh-credentials-endpoint`. It exists only in the spark-4.x source set. `CometIcebergNativeScan.withVendedCredentialProvider` selects it for the native Iceberg scan, not the writer, when the catalog returns a refresh endpoint and a `uri` and no provider is configured.

Each has a spark-3.x (SDK v1) and a spark-4.x (SDK v2) body under the same FQCN, selected by the `shims.majorVerSrc` source set, so each Comet build compiles against exactly the one AWS SDK its Hadoop line ships. The SDK and `hadoop-aws` are `provided` scope only (see the `hadoop-aws.version` property in the root `pom.xml`), so Comet does not bundle a second copy.
The two Hadoop and AWS SDK adapters each have a spark-3.x (SDK v1) and a spark-4.x (SDK v2) body under the same FQCN, selected by the `shims.majorVerSrc` source set, so each Comet build compiles against exactly the one AWS SDK its Hadoop line ships. `IcebergVendedS3CredentialProvider` has only the spark-4.x (SDK v2) body, so Spark 3.x builds don't ship it and never select it. The SDK and `hadoop-aws` are `provided` scope only (see the `hadoop-aws.version` property in the root `pom.xml`), so Comet does not bundle a second copy.

## Returns or throws, not a fall-through value

The SPI returns a `CometS3Credentials` or throws. There is no sentinel "I do not know" return. Vendors that are only authoritative for some paths resolve the default AWS chain themselves for the rest and return the result. This matches the contract on every other AWS credential SPI in the JVM ecosystem (AWS SDK v1/v2, Hadoop S3A, Iceberg `VendedCredentialsProvider`).

## Lifecycle: `AutoCloseable` plus a JVM shutdown hook
## Lifecycle: `AutoCloseable`, reference counts and a JVM shutdown hook

`CometS3CredentialProvider` extends `AutoCloseable` with a default no-op `close()`. The dispatcher installs a JVM shutdown hook that iterates every cached instance and calls `close()`, swallowing per-provider exceptions so a slow or buggy vendor cannot block other providers from cleaning up. Stateless providers ignore this entirely; vendors that hold long-lived resources (HTTP clients, scheduled-refresh executors, STS connection pools) override `close()` to release them. Shutdown hooks are best-effort, so a `SIGKILL` or abrupt JVM termination skips them; vendors must not rely on `close()` for correctness, only for resource hygiene.
`CometS3CredentialProvider` extends `AutoCloseable` with a default no-op `close()`.

Each `ensureInitialized` call takes a reference to the instance it returns. A native `CometS3CredentialBridge` holds that reference, shares it with the bridges it derives (`for_path`, `for_location`), and the last of them to be dropped gives it back with `release`. When an instance's last reference is released, the dispatcher closes it and forgets its key, so a later `ensureInitialized` creates a new instance. A reference count rather than "close when superseded" keeps an instance open while any reader still uses it.

The last bridge can be dropped on any thread: a Tokio worker running an iceberg-rust task, or a task thread inside `execute` that just missed the `FileIO` cache. So `release` hands the close to a single dispatcher-owned daemon thread, and a `close()` that blocks (Iceberg's `RefreshingAuthManager.close()` waits up to a minute for a refresh in flight) delays neither. The shutdown hook and a concurrent last release both claim an instance by setting its count to retired, so only one of them closes it.

On the Iceberg path, bridges live in the executor's `FileIO` cache. A REST catalog that vends new credentials for a table changes the table's FileIO properties, and so the `(FQCN, dispatchKey, catalogProperties)` key. Inserting the table's new `FileIO` evicts the ones it supersedes: same access mode, catalog and table, different properties with an earlier expiry. A read identifies the table by its metadata file's directory, since Iceberg writes every metadata file of a table into one directory (`write.metadata.path`, or `<location>/metadata`). A write identifies it by its data location. Only properties whose `s3.session-token-expires-at-ms` is later than the cached entry's supersede it, so both must carry one. Properties can also differ for reasons that don't say which is newer, such as two sessions or tenants that share a catalog name with different tokens, OAuth credentials or tenant ids. Those entries stay side by side, so the sessions don't evict each other on every task, and a provider re-vended without an expiry is released only when the LRU bound retires its entry. A commit that keeps the properties supersedes nothing. Evicted providers are closed once the tasks reading through them finish, so a long-lived executor holds one provider per table in use, not one per credential it was ever vended. The cache's 64-entry LRU bound limits the rest. On the Parquet path the object store cache never evicts, so its providers live until shutdown.

The dispatcher also installs a JVM shutdown hook that iterates every remaining instance and calls `close()`, swallowing per-provider exceptions so a slow or buggy vendor cannot block other providers from cleaning up. Stateless providers ignore this entirely; vendors that hold long-lived resources (HTTP clients, scheduled-refresh executors, STS connection pools) override `close()` to release them. Shutdown hooks are best-effort, so a `SIGKILL` or abrupt JVM termination skips them; vendors must not rely on `close()` for correctness, only for resource hygiene.

## Iceberg path: error message fidelity caveat

Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/latest/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ For a custom S3-compatible endpoint, configure the catalog with the endpoint, pa
--conf spark.sql.catalog.s3_cat.s3.secret-access-key=...
```

These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION` or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials.
These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION` or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials and returns `client.refresh-credentials-endpoint`, the native reader on Spark 4.x refreshes them through `IcebergVendedS3CredentialProvider`, which Comet selects automatically. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials.

### HDFS

Expand Down
62 changes: 30 additions & 32 deletions docs/source/user-guide/latest/s3-credential-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,29 @@ spark.hadoop.fs.s3a.bucket.<bucket>.comet.credential.adapter.class=<FQCN>

Use `HadoopS3ACredentialProviderAdapter` unless you have a plain SDK provider not wired through S3A. Both class names are the same on every Comet build; each build automatically uses the AWS SDK its Hadoop line ships (v1 on the Spark 3.4/3.5 builds, v2 on 4.0+), so you configure one name and get the right implementation.

### `IcebergVendedS3CredentialProvider` (native Iceberg reads, Spark 4.x)

Without a provider, the native Iceberg scan signs every request with the `s3.*` credentials the REST catalog vended when the driver loaded the table. Vended credentials are short-lived, so a scan that runs past their expiry fails with `ExpiredToken`. Iceberg's `S3FileIO`, which Spark's own Iceberg reads use, refreshes them through `client.refresh-credentials-endpoint` when the catalog returns one.

This provider resolves credentials the way `S3FileIO` does, through Iceberg's `AwsClientProperties`. It refreshes through the catalog's endpoint when one is returned, and otherwise uses the static vended keys and reports their vended expiry. Iceberg is loaded by reflection from the thread context class loader, so `iceberg-aws` can sit on the user-jar loader. The AWS SDK v2 must be visible from Comet's loader, as for the adapters above.

Comet selects this provider automatically for native Iceberg scans when all of these hold:

- The FileIO properties carry `client.refresh-credentials-endpoint`, and `client.refresh-credentials-enabled` is not `false`.
- The properties carry the catalog `uri`, whether the endpoint is relative or absolute.
- No provider is configured for the catalog.
- On the driver, Comet's class loader and the thread context class loader see the same AWS SDK v2. Comet logs at `INFO` when they don't.

An explicit setting takes precedence. An empty value turns the provider off for that catalog:

```
spark.sql.catalog.<catalog>.s3.comet.credential.provider.class=
```

Each instance of the provider holds its own HTTP client for the refresh endpoint. Comet closes an instance once the table's cached `FileIO` is replaced, which happens only when the catalog vends credentials whose `s3.session-token-expires-at-ms` is later than the cached ones (see [Lifecycle](#lifecycle)). Catalogs that vend S3 credentials, such as Apache Polaris and Unity Catalog, send that expiry. A catalog that rotates the `s3.*` keys without it gives each table reload another instance, and its HTTP client, until the executor's cache of 64 `FileIO`s evicts the older entries.

Spark 3.x builds do not include this provider. On Spark 3.x, a native Iceberg scan that outlives its vended credentials fails with `ExpiredToken` unless you configure a provider of your own. The native Iceberg writer does not select it automatically: a catalog that returns `client.refresh-credentials-endpoint` makes the write fall back to Spark.

## Enabling a bridge

A bridge is activated by naming the vendor's class in a Spark config. Putting a JAR on the classpath alone has no effect; the config key must be set.
Expand Down Expand Up @@ -210,7 +233,7 @@ public interface CometS3CredentialProvider extends AutoCloseable {

CometS3Credentials getCredentialsForPath(CometS3CredentialContext context) throws Exception;

/** Invoked from the dispatcher's JVM shutdown hook. Default is a no-op. */
/** Invoked when Comet retires the instance, or at JVM shutdown. Default is a no-op. */
@Override default void close() throws Exception {}
}
```
Expand All @@ -219,11 +242,13 @@ The class must have a public no-arg constructor. `getCredentialsForPath` may be

### Lifecycle

Comet keys provider instances by `(FQCN, dispatchKey, catalogProperties)`. The dispatch key is the Spark V2 catalog name on the Iceberg path and the S3 bucket name on the Parquet path. The first time a given key is seen on an executor, Comet reflects the class, calls `initialize(Map)` exactly once, and caches the instance for the JVM lifetime. Two catalogs sharing one provider FQCN therefore get isolated instances with their own `initialize` maps. Including `catalogProperties` in the key matters in multi-tenant JVMs (Spark Connect, Thrift Server, `SparkSession.newSession()`) where two sessions can otherwise collide on the same `(FQCN, dispatchKey)` and have the second session silently use the first session's credentials.
Comet keys provider instances by `(FQCN, dispatchKey, catalogProperties)`. The dispatch key is the Spark V2 catalog name on the Iceberg path and the S3 bucket name on the Parquet path. The first time a given key is seen on an executor, Comet reflects the class, calls `initialize(Map)` once, and keeps the instance while some reader uses it. Two catalogs sharing one provider FQCN therefore get isolated instances with their own `initialize` maps. Including `catalogProperties` in the key matters in multi-tenant JVMs (Spark Connect, Thrift Server, `SparkSession.newSession()`) where two sessions can otherwise collide on the same `(FQCN, dispatchKey)` and have the second session silently use the first session's credentials.

`initialize` should be cheap and non-blocking. Defer real credential fetches (REST round-trips, STS calls) to the first `getCredentialsForPath` invocation. On the Iceberg path the supplied `catalogProperties` carries the unfiltered FileIO bag, including REST-vended fields like `credentials.uri`, OAuth tokens, and any vendor-custom keys you set on the catalog config. On the Parquet path it carries the `fs.s3a.*` config subset (including static keys such as `fs.s3a.access.key`); in Comet 1.0 this map was empty on the Parquet path. The map may contain secrets, so do not log it.

`close()` is invoked from a JVM shutdown hook installed by the dispatcher. The default no-op is fine for stateless providers. Override it to release HTTP clients, scheduled-refresh executors, or STS connection pools. Shutdown hooks are best-effort: a `SIGKILL` or abrupt JVM termination skips them, so do not depend on `close()` for correctness.
Comet calls `close()` once no reader uses an instance any more, and at JVM shutdown for instances still in use. On the Iceberg path an instance is retired when the table's next `FileIO` replaces the one built on it, and when an executor's cache of 64 `FileIO`s evicts it. A new `FileIO` replaces the old one when the catalog vends new credentials for the table and both sets carry `s3.session-token-expires-at-ms`, the new one later. New credentials without a later expiry don't replace the old `FileIO`, so its instance stays open until the 64-entry bound evicts it. A later read with the same key then gets a new instance, with its own `initialize` call. On the Parquet path Comet keeps each object store, and so its provider, for the executor's lifetime.

So `close()` can run while the job is still running. Comet calls it at most once per instance, on a dispatcher thread rather than a reader's, and possibly while a new instance for the same key runs `initialize`. Release only the instance's own resources, never state that other instances share. A `close()` that blocks delays only the closes queued behind it. The default no-op is fine for stateless providers. Override it to release HTTP clients, scheduled-refresh executors, or STS connection pools. Shutdown hooks are best-effort: a `SIGKILL` or abrupt JVM termination skips them, so do not depend on `close()` for correctness.

### Caching, refresh, and distribution

Expand Down Expand Up @@ -358,36 +383,9 @@ public final class MyMultiTenantProvider implements CometS3CredentialProvider {
}
```

### Reference implementation: Iceberg REST vended credentials

For Iceberg REST catalogs that vend AWS credentials (`LoadTableResponse.credentials`), the canonical implementation wraps Iceberg's existing `VendedCredentialsProvider`:

```java
public final class IcebergRESTVendedS3Provider implements CometS3CredentialProvider {
private volatile VendedCredentialsProvider provider;

@Override
public void initialize(Map<String, String> catalogProperties) {
this.provider = VendedCredentialsProvider.create(catalogProperties);
}

@Override
public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) {
AwsCredentials c = provider.resolveCredentials();
String token = null;
long expiresAt = 0L;
if (c instanceof AwsSessionCredentials) {
AwsSessionCredentials session = (AwsSessionCredentials) c;
token = session.sessionToken();
// Report the vended expiry, so Comet reuses the credential until shortly before it.
expiresAt = session.expirationTime().map(Instant::toEpochMilli).orElse(0L);
}
return new CometS3Credentials(c.accessKeyId(), c.secretAccessKey(), token, expiresAt);
}
}
```
### Iceberg REST vended credentials

`VendedCredentialsProvider` reads `credentials.uri`, the catalog endpoint, and OAuth tokens from the supplied map (Comet forwards the unfiltered FileIO bag to `initialize`), and refreshes through its own `CachedSupplier`. Refresh-near-expiry and the REST round-trip live in Iceberg. Comet only reuses the credential this returns until five minutes before the expiry it reports, so report the expiry: returning `0` turns that reuse off. Comet ships a copy of this class under `spark/src/test` as a reference; copy it into your runtime jar alongside `iceberg-aws` and AWS SDK v2.
For Iceberg REST catalogs that vend AWS credentials, use the provider Comet ships, `IcebergVendedS3CredentialProvider` (see the section on it above). Comet selects it automatically when the catalog returns a refresh endpoint. To write your own, build Iceberg's credentials provider through `AwsClientProperties.credentialsProvider`, not `VendedCredentialsProvider.create`. A REST catalog returns its refresh endpoint as `client.refresh-credentials-endpoint`, often relative to `uri`, and `AwsClientProperties` resolves it into the `credentials.uri` that `VendedCredentialsProvider` reads. Report the credential's expiry: Comet reuses a credential until five minutes before the expiry you report, and returning `0` turns that reuse off.

### Access mode

Expand Down
Loading