Repository navigation
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of PR #70343 at 298f307: no actionable correctness findings identified. Review converged in 1 round: all three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS, all 20 changed files were covered, and every initial risk was dismissed with concrete evidence. There are no existing inline findings to retain as blocking.
Critical checkpoint conclusions:
- Goal and proof: REST encrypted metadata groundwork uses SDK crypto and preserves the staged SQL admission fence. Real HTTP connector unit tests exercise encrypted manifests/lists, descriptor reads, historical snapshots and refresh; the independent fixture reads actual encrypted Parquet. These tests were inspected, not executed during this review.
- Scope and reuse: the package-access bridge and read-only FileIO adapter keep the change focused and reuse SDK encryption, REST loading and configuration merging.
- Concurrency: parallel lazy initialization/manager replacement and close share the adapter monitor; metadata publication uses AtomicReference. Table and catalog leases retain resources for supported asynchronous reads. No reachable lock-order or concurrent refresh defect was established; writable refresh paths use fresh loads, while statement read views pin metadata. Potentially expensive initialization is confined to the table adapter rather than a global lock.
- Lifecycle: the SDK tracker uses weak operation keys without a backreference from the metadata holder. Cleanup closes the adapter idempotently, closes table-specific IO, and preserves shared catalog IO; KMS ownership survives manager refresh and closes on failure/retirement paths.
- Configuration: no new setting was added. The existing default-off read flag and backend-capability fence remain effective before KMS initialization; effective SDK FileIO properties retain catalog/server/table configuration.
- Compatibility: plugin API 31.0, its test pin and the added API baseline entry agree. Matching engine/plugins are required. No storage format, serialization or FE/BE wire contract changed.
- Thrift/protobuf IDs: not applicable; no gensrc definitions changed.
- Parallel paths: HMS/REST admission, plaintext REST, historical/system-table/COUNT/streaming paths, and wrapped/unwrapped vended credentials were inspected. The initial plaintext SDK path remains intact.
- Conditions and errors: encrypted-table detection includes retained keys; missing KMS/table-key states fail explicitly, typed descriptors select crypto, and unsupported mutation/transaction paths reject. The unused SDK create/replace-transaction overload did not establish a Doris production bug.
- Test coverage: positive, negative, ownership, concurrent initialization, refresh and admission cases cover the intended preparatory scope. Runtime validation is outside this static review; no builds, tests or benchmarks were run.
- Changed expected results: all 18 golden sections agree with the fixture rows and loop structure; the SQL regression contains 128 expected admission failures. Fixture locations are separate from HMS fixtures.
- Observability: explicit error classifications and existing cleanup logs are adequate for this scope; no new metric requirement identified.
- Transactions and persistence: no edit-log/replay or transaction-visibility protocol change. Loaded encrypted REST operations reject commit and temporary transaction operations.
- Data writes and crash behavior: no encrypted SQL execution or new data-write path is enabled; owned-resource cleanup was traced. Native visible-version/delete-bitmap invariants are unaffected.
- FE/BE variables: no new cross-boundary variable or sending path.
- Performance: KMS setup is lazy and reused per table; credential extraction remains outside per-file crypto work. No obvious new unbounded/redundant hot-path operation identified.
- Additional concerns: no extra user focus was supplied. SPI composition preserves the existing JDBC flavor policy and endpoint order/immutability; packaging includes the bridge with the pinned SDK. No remaining static coverage gaps or unresolved candidates.
Convergence and delivery are separate: this summary records converged=true after the complete first-round sweep. Static inspection does not establish runtime success.
Review converged after 1 round(s).
|
run buildall |
|
/review |
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70343
commit: f81331d40ed5dee61cec000d9025ae3f52d349d8
base: f7396d7229a26b421b07dde19f3be1dede8a7256
reviewed_at: 2026-11-06T16:33+08:00
reviewer: Gabriel39
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
There was a problem hiding this comment.
Static review of f81331d40ed5dee61cec000d9025ae3f52d349d8: no new actionable findings. All 20 changed files, related callers, and the pinned Iceberg SDK paths were inspected by the main reviewer and three scoped reviewers. No existing inline findings require carrying forward, and no additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: the change provides REST encrypted-metadata groundwork while retaining READ_DISABLED and BACKEND_CAPABILITY_UNAVAILABLE SQL admission. Connector HTTP tests exercise decryption; the independent fixture covers real encrypted Parquet and historical snapshots. Successful encrypted SQL execution is deliberately unavailable.
- Scope and reuse: the package bridge and FileIO decorator reuse SDK REST loading and standard crypto; the import suppression is narrowly scoped. No unrelated product behavior change identified.
- Concurrency and locks: manager construction and close share one monitor; refreshed metadata uses AtomicReference. Parallel readers reuse a client. Statement reads pin immutable metadata and mutable operations load separate tables; no supported new refresh/read race or lock-order cycle was substantiated.
- Lifecycle: the adapter avoids retaining the SDK tracker's weak operation key, distinguishes shared versus table-owned FileIO, and closes its KMS idempotently. Table/catalog leases retain resources through reader completion and authentication recovery. Failure cleanup was traced.
- Configuration: no new setting; the existing read-intent flag stays default-off. Effective SDK FileIO configuration supplies KMS properties, and refresh preserves the SDK's configuration contract. No new dynamic-config obligation.
- Compatibility: connector API 30.0 to 31.0, its version pin, and the public plugin baseline are consistent. The metadata-method surface is unchanged. Matching engine/plugins must be deployed together; no persisted format change.
- Thrift/protobuf: no gensrc definitions changed; no field-ID or enum compatibility work applies.
- Parallel paths: plaintext REST keeps SDK operations; HMS admission is retained and parameterized. Credential extraction traverses authentication and encryption decorators to preserve vended credentials.
- Conditional branches: declared encryption, retained keys, absent write key/KMS, and encrypted descriptor overloads have concrete failure or dispatch cases. No silent plaintext fallback identified in the new encrypted path.
- Test coverage: inspected lazy/concurrent initialization, ownership, refresh/history, descriptor types, missing/denied/wrong keys, mutation rejection, credentials, SPI baselines, and SQL cache/batch/COUNT/system/time-travel routes. Transition and refresh/close overlap tests are absent, but no supported current failing call chain was established.
- Expected results: the 18 golden checks and 128 READ_DISABLED cases match regression control flow. Fixture checks validate encrypted file magic, distinct keys, historical rows and authenticated wrong-key rejection. Output-generation provenance was not independently executed.
- Observability: stable admission and missing-key/KMS classifications provide actionable errors; no additional metric is necessary for currently fenced encrypted SQL execution.
- Transactions and persistence: loaded encrypted table transactions/commits and FileIO output/deletion reject changes. No EditLog, replay, failover, or persistence representation changed. Separate SDK catalog create/replace-transaction overloads have no Doris production caller.
- Data writes and crash behavior: no new admitted encrypted write path; ownership/exception cleanup was inspected. Catalog DROP/rename behavior is pre-existing and outside the table-operation decorator.
- FE/BE propagation: no new transmitted value or encrypted range is admitted; no additional send path requires modification.
- Performance: lazy, once-per-table KMS construction avoids load/admission work; manager reconstruction follows metadata identity changes. No new supported-path hot-loop regression found.
- Other/core invariants: no new data-visibility, native-memory, version/delete-bitmap, or static-initialization defect identified; most BE-specific checkpoints do not apply to these Java metadata changes.
Review converged in round 1 with all assigned reviewers returning NO_NEW_VALUABLE_FINDINGS and all initial risk focuses disposed with evidence. Coverage is static: no builds, tests, benchmarks, runtime interleaving checks, or source modifications were performed. The PR author's execution claims were inspected, not independently reproduced. No unresolved review candidate remains within the stated staged scope.
Review converged after 1 round(s).
TPC-H: Total hot run time: 28470 ms |
TPC-DS: Total hot run time: 154429 ms |
ClickBench: Total hot run time: 24.03 s |
TPC-H: Total hot run time: 28424 ms |
TPC-DS: Total hot run time: 154368 ms |
ClickBench: Total hot run time: 23.96 s |
|
run buildall |
2 similar comments
|
run buildall |
|
run buildall |
f81331d to
5822148
Compare
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of PR #70343 at 5822148c76bfba0e53834311be4c17a8f0566285: no actionable new findings. All 20 changed files and the related integration/lifecycle paths were inspected. No existing inline finding needs to be retained as blocking, and no additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: REST encrypted metadata groundwork uses SDK crypto while preserving READ_DISABLED and BACKEND_CAPABILITY_UNAVAILABLE SQL admission. HTTP connector tests cover descriptor decryption, historical snapshots and refresh; the independent fixture covers actual encrypted Parquet. Successful encrypted SQL execution remains outside this stage.
- Scope and reuse: the small package-access bridge and FileIO decorator reuse SDK REST loading, configuration merging, encryption and the existing admission contract. The Checkstyle exception is narrowly scoped.
- Concurrency and locks: lazy manager construction/replacement and close share a per-table monitor; metadata publication uses AtomicReference. Existing table/catalog leases retain resources through readers. No supported refresh/read failure or lock-order cycle was substantiated. Initial KMS setup holds only the table monitor; no catalog-wide lock was added.
- Lifecycle: the metadata holder avoids a reference back to the SDK tracker's weak operations key. Idempotent adapter cleanup closes its KMS and table-specific IO while preserving shared catalog IO; cache failure/unwind and catalog-generation retirement were traced.
- Configuration: no new setting. The existing read-intent flag remains default-off, and both disabled and enabled SQL admission reject before KMS initialization. Effective SDK FileIO properties retain catalog/server/table configuration; no new dynamic-configuration obligation.
- Compatibility: connector API 32.0, version pin and public plugin baseline agree; metadata-method API is unchanged. Matching engine/plugins are required. No persisted format or FE/BE wire change.
- Thrift/protobuf: no gensrc definitions changed, so field-ID and enum compatibility checkpoints do not apply.
- Parallel paths: plaintext REST retains SDK operations; HMS admission, ordinary/COUNT/streaming/system/historical reads and wrapped vended credentials were inspected. Existing server-side scan restrictions remain in force.
- Conditions and errors: retained keys still trigger admission; missing KMS/table-key states fail explicitly; typed descriptors preserve crypto metadata. Loaded encrypted operations reject commit, temp transactions, output and deletion rather than silently proceeding.
- Test coverage: positive, negative, concurrent lazy initialization, resource ownership, refresh/history, credentials, SPI composition and admission cases cover the intended preparatory scope. Direct concurrent SDK refresh and plaintext-to-encrypted transitions are not comprehensively tested; no failing supported Doris path was established under the current execution fence.
- Expected results: all 18 golden sections match fixture rows/counts and regression control flow; 128 encrypted statements expect READ_DISABLED. REST fixture namespaces/object paths are separate from HMS. Output generation provenance was not independently reproduced.
- Observability: stable admission/missing-KMS/key classifications and existing cleanup logs are adequate for the currently fenced scope; no additional metric requirement identified.
- Transactions and persistence: no EditLog, replay, failover or transaction-visibility representation changed. Existing-table encrypted mutations reject through loaded operations; separate SDK catalog-builder transaction overloads have no Doris caller in the inspected paths.
- Writes and crash behavior: no encrypted SQL scan or new product data-write capability is enabled. Resource ownership and failure cleanup were inspected; native visible-version and delete-bitmap invariants are unaffected.
- FE/BE variables: no new transmitted value or sending path was added.
- Performance: KMS creation is lazy and reused per table; manager reconstruction follows metadata identity changes. No obvious redundant/unbounded operation was identified on supported paths.
- Other concerns: no additional user focus. Packaging includes the bridge alongside the pinned SDK; SPI composition retains the existing JDBC flavor policy and immutable endpoint ordering. No unresolved candidate or uncovered changed path remains.
The SDK boundary was checked against the primary Iceberg 1.12.0 REST implementation. Review converged in 2 rounds after all assigned reviewers returned NO_NEW_VALUABLE_FINDINGS on the frozen payload and the final coverage sweep completed. This is static evidence only: no builds, tests, benchmarks or runtime interleaving checks were executed, and the author's execution claims were not independently reproduced.
Review converged after 2 round(s).
5822148 to
fc04c45
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### Background Iceberg encrypted reads use table metadata to locate retained table keys and decrypt manifest lists, manifests and file descriptors. Doris currently limits this groundwork to HMS catalogs. The pinned Iceberg REST operations retain the plaintext encryption manager, even when the REST response contains an encrypted table's key chain. ### What problem does this PR solve? Issue Number: close apache#70342 Related PR: apache#70192 (encrypted metadata groundwork) PR Type: Feature Problem Summary: Adding REST to the catalog allowlist alone leaves encrypted metadata unreadable. This change supplies lazy, read-only REST encryption operations while retaining the SDK REST table, authentication, credentials, refresh and scan capabilities. Most of the change is unit tests and an independent REST encryption fixture/regression suite. REST now follows the same staged scope as HMS; the common encrypted SQL execution fence remains in place. ### Details - **Change map:** 299 product/build/style lines, 895 unit/regression/fixture lines, 85 generated baseline/output lines and 38 documentation lines (insertions plus deletions). - **Scope:** Iceberg v3 REST tables using the standard SDK encryption manager and configured KMS. The initial file target remains Parquet modular encryption with AES_GCM_V1; encrypted writes and unrestricted encrypted SQL execution are outside this PR. - **Design:** A package-access bridge decorates the SDK's loaded REST table operations. Descriptor-aware FileIO decrypts encrypted metadata lazily using the effective catalog/server/table configuration. A standalone metadata holder avoids retaining weak FileIOTracker operation keys; refresh replaces the manager's key chain and reuses the table-owned KMS client. Idempotent close preserves shared catalog FileIO ownership and retires table-specific resources. Transactions, output files and deletion are rejected for encrypted REST tables. - **Usage:** Configure an Iceberg REST catalog with the existing KMS properties. There is no new setting, dependency, service or port. The newly consumed outbound destination is `kms.endpoint`, controlled by catalog configuration and CREATE_PRIV for CREATE CATALOG or ALTER_PRIV for ALTER CATALOG. The provider declares it for the existing SSRF policy while retaining its JDBC driver URL checks. REST server configuration/table overrides remain inside the configured catalog's trust boundary. - **Default behavior:** `enable_encryption_read` remains false. Disabled SQL reads fail with READ_DISABLED before KMS initialization. Enabling the existing intent flag still fails with BACKEND_CAPABILITY_UNAVAILABLE until protected key transport and backend admission are complete. Plaintext REST reads retain the SDK path. - **Compatibility:** No storage or FE/BE wire changes. The additive security-surface composition method requires connector plugin API 33.0 under the repository's SPI contract; deploy the matching engine and connector plugins together. The plugin API baseline is regenerated and the unchanged metadata-method baseline is verified. - **Behavior change:** REST joins HMS in the encrypted metadata target matrix. Missing KMS configuration and retained keys without the mandatory table-key property fail explicitly rather than selecting plaintext crypto. Encrypted REST mutation paths fail as read-only. ### Testing - Local `build.sh --be --fe -j6`, ASAN: passed, followed by an isolated current-worktree FE/BE cluster run. - Connector Iceberg tests with Maven build cache disabled: 1,892 tests, zero failures/errors, 10 existing skips. This includes real connector HTTP REST loading, actual encrypted SDK manifests and descriptor payloads, historical scans/refresh, shared/per-table ownership, concurrent lazy initialization, missing/denied/wrong KMS keys, read-only transactions and vended credential forwarding. The REST manager test fails against the original SDK plaintext operations. - Connector SPI own suite: 154 tests passed, including both API baselines and the plugin version pin. Endpoint policy composition preserves the JDBC policy and immutable endpoint lists. - FE Checkstyle: zero violations during product build and final connector test validation. - New `test_iceberg_encrypted_rest_metadata`: output generated by the regression framework, then compared successfully; 18 golden checks and 128 expected READ_DISABLED errors across cache, batch and COUNT settings. The independent fixture checks two historical row sets, encrypted Parquet/manifest/list magic and independent file keys, plus missing KMS, bad credentials, denied decrypt and wrong-key failures. - Existing `test_iceberg_encrypted_hms_metadata`: passed, including the SDK/Spark reference checks. - Restarted the local FE with the existing read-intent flag enabled: 16 REST SQL queries rejected with BACKEND_CAPABILITY_UNAVAILABLE, while plaintext rows remained correct. - Staged patch whitespace and secret scan: passed. ### Notes for Reviewers The SQL regression deliberately verifies the staged admission contract; successful decryption is exercised by the connector HTTP unit tests and an independent SDK reference reader over real encrypted files. It does not claim that this PR enables encrypted SQL scans. Start with DorisRESTSessionCatalog and IcebergRestEncryptionFileIO for the ownership and metadata-generation boundaries. ### Release note Extend Iceberg encrypted metadata read groundwork to REST Catalog with standard SDK encryption and configured KMS. The existing default-off setting and encrypted SQL execution fence remain in effect. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; connector documentation and a regression reproduction command are included. User-facing documentation can follow when encrypted SQL execution is admitted. - Pre-submission status: Ready; local validation passed, CI pending. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Implementation, unit/regression tests, connector documentation and this description. - Human review: Pending maintainer review; the agent performed a full self-review and local validation.
### Background Encrypted REST tables reject transactions during creation as part of their read-only contract. SDK temporary operations retain virtual FileIO dispatch. ### What problem does this PR solve? Issue Number: apache#70342 Related PR: None PR Type: Build/CI/Chore Problem Summary: Correct the transaction-guard comment to describe eager read-only enforcement before callers can stage mutable metadata or initialize KMS for a write. Remove the inaccurate claim that SDK temporary operations expose raw FileIO. Executable code is unchanged. ### Details - **Affected:** The explanatory comment beside the encrypted REST transaction guard. - **Developer and release impact:** No behavior change; the comment now matches the pinned SDK implementation. - **License and third-party:** No dependency or license changes. - **Rollback:** Restore the prior comment only. ### Testing Verified the SDK 1.12.0 temporary operations bytecode and inspected the complete follow-up diff. FE connector Checkstyle validation passed. Existing unit, regression and ASAN build evidence applies to the unchanged executable implementation. ### Release note None. ### Check List (For Author) - Behavior changed: No. - Does this need documentation: No. - Pre-submission status: Ready. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Comment and commit description. - Human review: Pending maintainer review; the agent reviewed the complete diff.
fc04c45 to
8186868
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
There was a problem hiding this comment.
Static review of PR #70343 at 8186868f80bd29be6e50923e9fadff518546ce13: no actionable new findings. All 20 changed files and related callers were inspected by the main reviewer, two complete-scope reviewers and a separate ownership/refresh reviewer. No existing inline findings require carrying forward, and no additional user focus was supplied.
Reviewed flow:
REST catalog load -> SDK table/operations -> lazy descriptor-aware FileIO/KMS
statement/cache lease -> frozen metadata -> common encrypted-read admission
lease release -> adapter cleanup -> catalog-generation retirement
Critical checkpoint conclusions:
- Goal and proof: REST encrypted-metadata groundwork reuses SDK crypto while retaining READ_DISABLED and BACKEND_CAPABILITY_UNAVAILABLE SQL admission. HTTP connector unit tests cover encrypted manifests/lists/descriptors, historical snapshots and refresh; an independent fixture covers actual encrypted Parquet. These test sources were inspected, not executed.
- Scope and reuse: the package-access bridge and FileIO decorator use existing REST loading, effective configuration and standard encryption. The Checkstyle exception is limited to the bridge's ImportControl rule.
- Concurrency and locks: lazy manager initialization/replacement and close share one per-table monitor; refreshed encryption metadata uses AtomicReference. Existing statement/cache/catalog owners retain resources through supported readers. No new lock-order cycle or supported concurrent-refresh failure was established. This does not certify arbitrary concurrent calls to the raw SDK refresh method.
- Lifecycle: the metadata holder avoids retaining the SDK tracker's weak operations key. Frozen snapshot operations retain the original operations; idempotent adapter cleanup closes its KMS and table-specific IO while preserving shared catalog IO. Catalog-generation leases defer retirement through borrower completion.
- Configuration: no new setting or default. The existing default-off read flag and enabled-mode backend fence reject encrypted SQL before KMS initialization. Effective SDK FileIO properties carry catalog/server/table configuration; refresh preserves the pinned SDK's configuration contract.
- Compatibility: the additive SPI method, plugin API 33.0, version test and public API baseline agree. The metadata-method surface is unchanged. Matching engine/plugins are required; no storage-format or FE/BE wire change was introduced.
- Thrift/protobuf IDs: not applicable; no gensrc definitions changed.
- Parallel paths: HMS admission remains; plaintext REST keeps SDK operations. Ordinary, COUNT, batch/streaming, historical and system-table paths were traced. Vended credentials remain accessible through both authentication and encryption decorators.
- Conditions and errors: retained keys trigger encrypted-table detection; missing KMS/current table key fail explicitly. Typed descriptors retain encryption metadata, while plaintext descriptors delegate. Normal fresh REST loads select operations from current metadata. A direct plaintext-to-encrypted raw SDK refresh retains plaintext operations, but the common Doris admission fence prevents an admitted encrypted read.
- Test coverage: inspected positive/negative crypto, lazy/concurrent initialization, ownership, history/refresh, mutation rejection, credential forwarding, SPI composition/version and SQL admission tests. Arbitrary refresh/read/close overlap and plaintext-to-encrypted transitions are not comprehensively tested; no failing supported current Doris path was found. Runtime validation remains outside this review.
- Expected results: all 18 golden checks match fixture rows/counts and cache/batch/COUNT loops; 128 encrypted statements expect READ_DISABLED. REST and HMS fixture locations are separate. Output-generation provenance was not independently reproduced.
- Observability: stable encryption classifications and surrounding table-load/cleanup errors provide context for this staged scope; no new critical metric requirement was identified.
- Transactions and persistence: loaded encrypted operations reject temporary transactions and commit; no EditLog, replay, failover or transaction-visibility protocol changed. Separate SDK create/replace-transaction overloads have no Doris production caller in the inspected paths.
- Data writes and crash behavior: no encrypted SQL execution or new production data-write capability is enabled. FileIO output/deletion reject on the encrypted adapter; resource cleanup and failure/unwind paths were traced. Native visible-version/delete-bitmap invariants are unaffected.
- FE/BE variables: no new transmitted value or sending path; encrypted task dispatch stays fenced.
- Performance: KMS setup is lazy and reused per table; manager reconstruction follows metadata identity changes. Credential extraction preserves the existing granularity. No obvious redundant/unbounded operation was identified on supported paths; no benchmark ran.
- Other concerns and user focus: no additional focus was supplied. SPI composition preserves the JDBC flavor policy and secret declarations along with endpoint ordering/immutability. No unresolved candidate or uncovered changed path remains within the current staged scope.
Review converged in round 1: all assigned reviewers returned NO_NEW_VALUABLE_FINDINGS, required coverage is complete, and every initial risk was dismissed with evidence after independent verification and the final sweep. Validation was static only: no builds, tests, benchmarks, runtime interleaving checks, source edits or submodule initialization. The PR author's execution claims were not independently reproduced.
Review converged after 1 round(s).
Background
Iceberg encrypted reads use table metadata to locate retained table keys and decrypt manifest lists, manifests and file descriptors. Doris currently limits this groundwork to HMS catalogs. The pinned Iceberg REST operations retain the plaintext encryption manager, even when the REST response contains an encrypted table's key chain.
What problem does this PR solve?
Issue Number: close #70342
Related PR: #70192 (encrypted metadata groundwork)
PR Type: Feature
Problem Summary:
Adding REST to the catalog allowlist alone leaves encrypted metadata unreadable. This change supplies lazy, read-only REST encryption operations while retaining the SDK REST table, authentication, credentials, refresh and scan capabilities. Most of the change is unit tests and an independent REST encryption fixture/regression suite. REST now follows the same staged scope as HMS; the common encrypted SQL execution fence remains in place.
Details
kms.endpoint, controlled by catalog configuration and CREATE_PRIV for CREATE CATALOG or ALTER_PRIV for ALTER CATALOG. The provider declares it for the existing SSRF policy while retaining its JDBC driver URL checks. REST server configuration/table overrides remain inside the configured catalog's trust boundary.enable_encryption_readremains false. Disabled SQL reads fail with READ_DISABLED before KMS initialization. Enabling the existing intent flag still fails with BACKEND_CAPABILITY_UNAVAILABLE until protected key transport and backend admission are complete. Plaintext REST reads retain the SDK path.Testing
Rebase validation (2026-11-17, master
3659256fa7d5): all 165 SPI tests passed, including both API baselines and the 33.0 version pin. The complete Iceberg connector suite ran 1,987 tests with zero failures/errors and eleven existing skips. FE Checkstyle passed in all 21 reactor modules. Both pinned-1.12.0 fixture Java sources compiled together, and the conflict resolution preserves the native-delete and REST routes with separate namespaces and warehouses. The original two commits, commit messages, generated regression output and SPI security-policy composition are retained. BE builds and cluster regression were not repeated for this rebase; CI is pending.Validation recorded before this rebase:
build.sh --be --fe -j6, ASAN: passed, followed by an isolated current-worktree FE/BE cluster run.test_iceberg_encrypted_rest_metadata: output generated by the regression framework, then compared successfully; 18 golden checks and 128 expected READ_DISABLED errors across cache, batch and COUNT settings. The independent fixture checks two historical row sets, encrypted Parquet/manifest/list magic and independent file keys, plus missing KMS, bad credentials, denied decrypt and wrong-key failures.test_iceberg_encrypted_hms_metadata: passed, including the SDK/Spark reference checks.Notes for Reviewers
The SQL regression deliberately verifies the staged admission contract; successful decryption is exercised by the connector HTTP unit tests and an independent SDK reference reader over real encrypted files. It does not claim that this PR enables encrypted SQL scans. Start with DorisRESTSessionCatalog and IcebergRestEncryptionFileIO for the ownership and metadata-generation boundaries.
Release note
Extend Iceberg encrypted metadata read groundwork to REST Catalog with standard SDK encryption and configured KMS. The existing default-off setting and encrypted SQL execution fence remain in effect.
Check List (For Author)
AI Disclosure