Skip to content

[feature](iceberg) Add REST encrypted metadata read support - #70343

Open
Gabriel39 wants to merge 2 commits into
apache:masterfrom
Gabriel39:dev/iceberg-rest-encryption
Open

Gabriel39 wants to merge 2 commits into
apache:masterfrom
Gabriel39:dev/iceberg-rest-encryption

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Background

Iceberg encrypted reads use table metadata to locate retained table keys and decrypt manifest lists, manifests and file descriptors. Doris currently limits this groundwork to HMS catalogs. The pinned Iceberg REST operations retain the plaintext encryption manager, even when the REST response contains an encrypted table's key chain.

What problem does this PR solve?

Issue Number: close #70342
Related PR: #70192 (encrypted metadata groundwork)
PR Type: Feature

Problem Summary:

Adding REST to the catalog allowlist alone leaves encrypted metadata unreadable. This change supplies lazy, read-only REST encryption operations while retaining the SDK REST table, authentication, credentials, refresh and scan capabilities. Most of the change is unit tests and an independent REST encryption fixture/regression suite. REST now follows the same staged scope as HMS; the common encrypted SQL execution fence remains in place.

Details

  • Change map: 299 product/build/style lines, 895 unit/regression/fixture lines, 85 generated baseline/output lines and 38 documentation lines (insertions plus deletions).
  • Scope: Iceberg v3 REST tables using the standard SDK encryption manager and configured KMS. The initial file target remains Parquet modular encryption with AES_GCM_V1; encrypted writes and unrestricted encrypted SQL execution are outside this PR.
  • Design: A package-access bridge decorates the SDK's loaded REST table operations. Descriptor-aware FileIO decrypts encrypted metadata lazily using the effective catalog/server/table configuration. A standalone metadata holder avoids retaining weak FileIOTracker operation keys; refresh replaces the manager's key chain and reuses the table-owned KMS client. Idempotent close preserves shared catalog FileIO ownership and retires table-specific resources. Transactions, output files and deletion are rejected for encrypted REST tables.
  • Usage: Configure an Iceberg REST catalog with the existing KMS properties. There is no new setting, dependency, service or port. The newly consumed outbound destination is kms.endpoint, controlled by catalog configuration and CREATE_PRIV for CREATE CATALOG or ALTER_PRIV for ALTER CATALOG. The provider declares it for the existing SSRF policy while retaining its JDBC driver URL checks. REST server configuration/table overrides remain inside the configured catalog's trust boundary.
  • Default behavior: enable_encryption_read remains false. Disabled SQL reads fail with READ_DISABLED before KMS initialization. Enabling the existing intent flag still fails with BACKEND_CAPABILITY_UNAVAILABLE until protected key transport and backend admission are complete. Plaintext REST reads retain the SDK path.
  • Compatibility: No storage or FE/BE wire changes. The additive security-surface composition method requires connector plugin API 33.0 under the repository's SPI contract; deploy the matching engine and connector plugins together. The plugin API baseline is regenerated and the unchanged metadata-method baseline is verified.
  • Behavior change: REST joins HMS in the encrypted metadata target matrix. Missing KMS configuration and retained keys without the mandatory table-key property fail explicitly rather than selecting plaintext crypto. Encrypted REST mutation paths fail as read-only.

Testing

Rebase validation (2026-11-17, master 3659256fa7d5): all 165 SPI tests passed, including both API baselines and the 33.0 version pin. The complete Iceberg connector suite ran 1,987 tests with zero failures/errors and eleven existing skips. FE Checkstyle passed in all 21 reactor modules. Both pinned-1.12.0 fixture Java sources compiled together, and the conflict resolution preserves the native-delete and REST routes with separate namespaces and warehouses. The original two commits, commit messages, generated regression output and SPI security-policy composition are retained. BE builds and cluster regression were not repeated for this rebase; CI is pending.

Validation recorded before this rebase:

  • Local build.sh --be --fe -j6, ASAN: passed, followed by an isolated current-worktree FE/BE cluster run.
  • Connector Iceberg tests with Maven build cache disabled: 1,892 tests, zero failures/errors, 10 existing skips. This includes real connector HTTP REST loading, actual encrypted SDK manifests and descriptor payloads, historical scans/refresh, shared/per-table ownership, concurrent lazy initialization, missing/denied/wrong KMS keys, read-only transactions and vended credential forwarding. The REST manager test fails against the original SDK plaintext operations.
  • Connector SPI own suite: 154 tests passed, including both API baselines and the plugin version pin. Endpoint policy composition preserves the JDBC policy and immutable endpoint lists.
  • FE Checkstyle: zero violations during product build and final connector test validation.
  • New test_iceberg_encrypted_rest_metadata: output generated by the regression framework, then compared successfully; 18 golden checks and 128 expected READ_DISABLED errors across cache, batch and COUNT settings. The independent fixture checks two historical row sets, encrypted Parquet/manifest/list magic and independent file keys, plus missing KMS, bad credentials, denied decrypt and wrong-key failures.
  • Existing test_iceberg_encrypted_hms_metadata: passed, including the SDK/Spark reference checks.
  • Restarted the local FE with the existing read-intent flag enabled: 16 REST SQL queries rejected with BACKEND_CAPABILITY_UNAVAILABLE, while plaintext rows remained correct.
  • Staged patch whitespace and secret scan: passed.

Notes for Reviewers

The SQL regression deliberately verifies the staged admission contract; successful decryption is exercised by the connector HTTP unit tests and an independent SDK reference reader over real encrypted files. It does not claim that this PR enables encrypted SQL scans. Start with DorisRESTSessionCatalog and IcebergRestEncryptionFileIO for the ownership and metadata-generation boundaries.

Release note

Extend Iceberg encrypted metadata read groundwork to REST Catalog with standard SDK encryption and configured KMS. The existing default-off setting and encrypted SQL execution fence remain in effect.

Check List (For Author)

  • Behavior changed: Yes (see Details).
  • Does this need documentation: Yes; connector documentation and a regression reproduction command are included. User-facing documentation can follow when encrypted SQL execution is admitted.
  • Pre-submission status: Ready; local validation passed, CI pending.

AI Disclosure

  • Tool: OpenAI Codex.
  • Generated: Implementation, unit/regression tests, connector documentation and this description.
  • Human review: Pending maintainer review; the agent performed a full self-review and local validation.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70343 at 298f307: no actionable correctness findings identified. Review converged in 1 round: all three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS, all 20 changed files were covered, and every initial risk was dismissed with concrete evidence. There are no existing inline findings to retain as blocking.

Critical checkpoint conclusions:

  • Goal and proof: REST encrypted metadata groundwork uses SDK crypto and preserves the staged SQL admission fence. Real HTTP connector unit tests exercise encrypted manifests/lists, descriptor reads, historical snapshots and refresh; the independent fixture reads actual encrypted Parquet. These tests were inspected, not executed during this review.
  • Scope and reuse: the package-access bridge and read-only FileIO adapter keep the change focused and reuse SDK encryption, REST loading and configuration merging.
  • Concurrency: parallel lazy initialization/manager replacement and close share the adapter monitor; metadata publication uses AtomicReference. Table and catalog leases retain resources for supported asynchronous reads. No reachable lock-order or concurrent refresh defect was established; writable refresh paths use fresh loads, while statement read views pin metadata. Potentially expensive initialization is confined to the table adapter rather than a global lock.
  • Lifecycle: the SDK tracker uses weak operation keys without a backreference from the metadata holder. Cleanup closes the adapter idempotently, closes table-specific IO, and preserves shared catalog IO; KMS ownership survives manager refresh and closes on failure/retirement paths.
  • Configuration: no new setting was added. The existing default-off read flag and backend-capability fence remain effective before KMS initialization; effective SDK FileIO properties retain catalog/server/table configuration.
  • Compatibility: plugin API 31.0, its test pin and the added API baseline entry agree. Matching engine/plugins are required. No storage format, serialization or FE/BE wire contract changed.
  • Thrift/protobuf IDs: not applicable; no gensrc definitions changed.
  • Parallel paths: HMS/REST admission, plaintext REST, historical/system-table/COUNT/streaming paths, and wrapped/unwrapped vended credentials were inspected. The initial plaintext SDK path remains intact.
  • Conditions and errors: encrypted-table detection includes retained keys; missing KMS/table-key states fail explicitly, typed descriptors select crypto, and unsupported mutation/transaction paths reject. The unused SDK create/replace-transaction overload did not establish a Doris production bug.
  • Test coverage: positive, negative, ownership, concurrent initialization, refresh and admission cases cover the intended preparatory scope. Runtime validation is outside this static review; no builds, tests or benchmarks were run.
  • Changed expected results: all 18 golden sections agree with the fixture rows and loop structure; the SQL regression contains 128 expected admission failures. Fixture locations are separate from HMS fixtures.
  • Observability: explicit error classifications and existing cleanup logs are adequate for this scope; no new metric requirement identified.
  • Transactions and persistence: no edit-log/replay or transaction-visibility protocol change. Loaded encrypted REST operations reject commit and temporary transaction operations.
  • Data writes and crash behavior: no encrypted SQL execution or new data-write path is enabled; owned-resource cleanup was traced. Native visible-version/delete-bitmap invariants are unaffected.
  • FE/BE variables: no new cross-boundary variable or sending path.
  • Performance: KMS setup is lazy and reused per table; credential extraction remains outside per-file crypto work. No obvious new unbounded/redundant hot-path operation identified.
  • Additional concerns: no extra user focus was supplied. SPI composition preserves the existing JDBC flavor policy and endpoint order/immutability; packaging includes the bridge with the pinned SDK. No remaining static coverage gaps or unresolved candidates.

Convergence and delivery are separate: this summary records converged=true after the complete first-round sweep. Static inspection does not establish runtime success.

Review converged after 1 round(s).

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70343
commit: f81331d40ed5dee61cec000d9025ae3f52d349d8
base: f7396d7229a26b421b07dde19f3be1dede8a7256
reviewed_at: 2026-11-06T16:33+08:00
reviewer: Gabriel39
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • No builds or tests were run during this read-only review; existing author validation evidence was inspected.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of f81331d40ed5dee61cec000d9025ae3f52d349d8: no new actionable findings. All 20 changed files, related callers, and the pinned Iceberg SDK paths were inspected by the main reviewer and three scoped reviewers. No existing inline findings require carrying forward, and no additional user focus was supplied.

Critical checkpoint conclusions:

  • Goal and proof: the change provides REST encrypted-metadata groundwork while retaining READ_DISABLED and BACKEND_CAPABILITY_UNAVAILABLE SQL admission. Connector HTTP tests exercise decryption; the independent fixture covers real encrypted Parquet and historical snapshots. Successful encrypted SQL execution is deliberately unavailable.
  • Scope and reuse: the package bridge and FileIO decorator reuse SDK REST loading and standard crypto; the import suppression is narrowly scoped. No unrelated product behavior change identified.
  • Concurrency and locks: manager construction and close share one monitor; refreshed metadata uses AtomicReference. Parallel readers reuse a client. Statement reads pin immutable metadata and mutable operations load separate tables; no supported new refresh/read race or lock-order cycle was substantiated.
  • Lifecycle: the adapter avoids retaining the SDK tracker's weak operation key, distinguishes shared versus table-owned FileIO, and closes its KMS idempotently. Table/catalog leases retain resources through reader completion and authentication recovery. Failure cleanup was traced.
  • Configuration: no new setting; the existing read-intent flag stays default-off. Effective SDK FileIO configuration supplies KMS properties, and refresh preserves the SDK's configuration contract. No new dynamic-config obligation.
  • Compatibility: connector API 30.0 to 31.0, its version pin, and the public plugin baseline are consistent. The metadata-method surface is unchanged. Matching engine/plugins must be deployed together; no persisted format change.
  • Thrift/protobuf: no gensrc definitions changed; no field-ID or enum compatibility work applies.
  • Parallel paths: plaintext REST keeps SDK operations; HMS admission is retained and parameterized. Credential extraction traverses authentication and encryption decorators to preserve vended credentials.
  • Conditional branches: declared encryption, retained keys, absent write key/KMS, and encrypted descriptor overloads have concrete failure or dispatch cases. No silent plaintext fallback identified in the new encrypted path.
  • Test coverage: inspected lazy/concurrent initialization, ownership, refresh/history, descriptor types, missing/denied/wrong keys, mutation rejection, credentials, SPI baselines, and SQL cache/batch/COUNT/system/time-travel routes. Transition and refresh/close overlap tests are absent, but no supported current failing call chain was established.
  • Expected results: the 18 golden checks and 128 READ_DISABLED cases match regression control flow. Fixture checks validate encrypted file magic, distinct keys, historical rows and authenticated wrong-key rejection. Output-generation provenance was not independently executed.
  • Observability: stable admission and missing-key/KMS classifications provide actionable errors; no additional metric is necessary for currently fenced encrypted SQL execution.
  • Transactions and persistence: loaded encrypted table transactions/commits and FileIO output/deletion reject changes. No EditLog, replay, failover, or persistence representation changed. Separate SDK catalog create/replace-transaction overloads have no Doris production caller.
  • Data writes and crash behavior: no new admitted encrypted write path; ownership/exception cleanup was inspected. Catalog DROP/rename behavior is pre-existing and outside the table-operation decorator.
  • FE/BE propagation: no new transmitted value or encrypted range is admitted; no additional send path requires modification.
  • Performance: lazy, once-per-table KMS construction avoids load/admission work; manager reconstruction follows metadata identity changes. No new supported-path hot-loop regression found.
  • Other/core invariants: no new data-visibility, native-memory, version/delete-bitmap, or static-initialization defect identified; most BE-specific checkpoints do not apply to these Java metadata changes.

Review converged in round 1 with all assigned reviewers returning NO_NEW_VALUABLE_FINDINGS and all initial risk focuses disposed with evidence. Coverage is static: no builds, tests, benchmarks, runtime interleaving checks, or source modifications were performed. The PR author's execution claims were inspected, not independently reproduced. No unresolved review candidate remains within the stated staged scope.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28470 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f81331d40ed5dee61cec000d9025ae3f52d349d8, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17593	5218	5108	5108
q2	2124	308	258	258
q3	10254	1386	822	822
q4	4683	467	336	336
q5	7446	794	534	534
q6	179	190	158	158
q7	756	826	613	613
q8	9260	1370	1071	1071
q9	5599	4221	4198	4198
q10	6817	1292	1036	1036
q11	433	246	232	232
q12	644	397	275	275
q13	18123	3943	3060	3060
q14	273	272	244	244
q15	q16	732	718	642	642
q17	1196	748	566	566
q18	6998	6188	6130	6130
q19	1130	954	598	598
q20	391	347	224	224
q21	5022	2113	2316	2113
q22	342	293	252	252
Total cold run time: 99995 ms
Total hot run time: 28470 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5617	5585	5606	5585
q2	673	561	530	530
q3	5192	5386	4950	4950
q4	2002	2160	1546	1546
q5	5267	5185	5818	5185
q6	258	202	144	144
q7	2416	2098	1905	1905
q8	3080	2852	2768	2768
q9	8075	8004	7949	7949
q10	3740	3658	3251	3251
q11	603	490	390	390
q12	686	689	526	526
q13	3481	4004	3289	3289
q14	306	327	292	292
q15	q16	715	732	648	648
q17	8517	7587	7302	7302
q18	12968	12207	12901	12207
q19	890	807	785	785
q20	2130	2142	1910	1910
q21	5588	4553	4730	4553
q22	497	439	424	424
Total cold run time: 72701 ms
Total hot run time: 66139 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154429 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f81331d40ed5dee61cec000d9025ae3f52d349d8, data reload: false

query5	4322	518	394	394
query6	422	210	194	194
query7	4794	425	230	230
query8	318	175	160	160
query9	8782	3175	3206	3175
query10	463	315	260	260
query11	5957	3872	3581	3581
query12	152	91	90	90
query13	1254	442	326	326
query14	6161	3958	3621	3621
query14_1	3407	3404	3408	3404
query15	210	206	182	182
query16	947	488	428	428
query17	863	659	539	539
query18	2425	419	305	305
query19	207	168	139	139
query20	86	85	84	84
query21	215	128	109	109
query22	13033	13060	12753	12753
query23	13564	12856	12375	12375
query23_1	12552	12654	12668	12654
query24	7402	830	484	484
query24_1	498	484	487	484
query25	563	430	363	363
query26	1256	261	142	142
query27	2759	439	297	297
query28	4597	2058	2045	2045
query29	1591	609	458	458
query30	301	223	182	182
query31	1033	937	825	825
query32	151	94	94	94
query33	546	309	249	249
query34	892	861	506	506
query35	752	764	639	639
query36	819	812	737	737
query37	133	101	87	87
query38	1782	1748	1669	1669
query39	798	798	744	744
query39_1	735	736	753	736
query40	237	120	99	99
query41	74	68	70	68
query42	88	95	91	91
query43	353	357	315	315
query44	1314	721	728	721
query45	195	184	177	177
query46	829	975	581	581
query47	2884	3003	2914	2914
query48	306	328	222	222
query49	578	401	293	293
query50	673	275	207	207
query51	10849	10760	10761	10760
query52	87	80	75	75
query53	192	206	156	156
query54	252	195	177	177
query55	77	73	63	63
query56	223	212	221	212
query57	1679	1615	1543	1543
query58	291	265	268	265
query59	2226	2240	2016	2016
query60	311	245	231	231
query61	146	149	145	145
query62	396	341	283	283
query63	190	160	160	160
query64	2764	922	774	774
query65	3314	3293	3265	3265
query66	1751	432	303	303
query67	22602	22700	22517	22517
query68	3001	1012	626	626
query69	386	299	251	251
query70	874	806	774	774
query71	280	234	218	218
query72	2773	2612	2274	2274
query73	508	524	293	293
query74	4657	4488	4343	4343
query75	2358	2251	1904	1904
query76	2345	1007	586	586
query77	353	357	278	278
query78	9812	9705	9235	9235
query79	1012	846	521	521
query80	806	423	346	346
query81	565	310	259	259
query82	579	143	113	113
query83	280	200	189	189
query84	316	127	109	109
query85	970	437	381	381
query86	384	243	234	234
query87	1977	1931	1794	1794
query88	3750	2777	2776	2776
query89	336	282	247	247
query90	1913	170	171	170
query91	169	157	127	127
query92	102	81	92	81
query93	1000	995	585	585
query94	565	290	269	269
query95	571	337	321	321
query96	681	481	233	233
query97	2459	2447	2358	2358
query98	163	153	149	149
query99	773	771	651	651
Total cold run time: 238144 ms
Total hot run time: 154429 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f81331d40ed5dee61cec000d9025ae3f52d349d8, data reload: false

query1	0.00	0.01	0.00
query2	0.09	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.09	0.10
query5	0.23	0.24	0.24
query6	1.16	0.68	0.67
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.39	0.35	0.33
query10	0.58	0.56	0.58
query11	0.20	0.15	0.14
query12	0.19	0.15	0.15
query13	0.49	0.50	0.50
query14	1.02	1.04	1.00
query15	0.63	0.62	0.60
query16	0.35	0.35	0.35
query17	1.15	1.17	1.17
query18	0.21	0.20	0.18
query19	2.05	2.01	2.00
query20	0.02	0.02	0.01
query21	15.42	0.21	0.13
query22	4.83	0.05	0.05
query23	16.25	0.32	0.13
query24	1.06	0.23	0.16
query25	0.09	0.04	0.04
query26	0.76	0.21	0.16
query27	0.03	0.03	0.03
query28	3.63	0.82	0.36
query29	12.50	4.20	3.25
query30	0.30	0.13	0.09
query31	2.77	0.56	0.33
query32	3.22	0.60	0.50
query33	3.17	3.15	3.22
query34	15.37	3.93	3.36
query35	3.31	3.33	3.35
query36	0.56	0.46	0.45
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 94.52 s
Total hot run time: 24.03 s

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28424 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f81331d40ed5dee61cec000d9025ae3f52d349d8, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17619	5171	5029	5029
q2	2103	295	250	250
q3	10284	1405	843	843
q4	4681	453	333	333
q5	7431	799	527	527
q6	186	185	155	155
q7	746	789	598	598
q8	9259	1332	1164	1164
q9	5554	4348	4212	4212
q10	6808	1295	1024	1024
q11	439	237	226	226
q12	629	396	280	280
q13	18091	3602	2963	2963
q14	268	268	237	237
q15	q16	706	701	663	663
q17	1194	756	588	588
q18	7018	6188	6120	6120
q19	1105	988	604	604
q20	393	332	227	227
q21	4631	2126	2298	2126
q22	343	289	255	255
Total cold run time: 99488 ms
Total hot run time: 28424 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5593	5582	5576	5576
q2	675	563	533	533
q3	5230	5361	4944	4944
q4	1992	2162	1500	1500
q5	5271	5124	5510	5124
q6	258	199	154	154
q7	2349	2108	1881	1881
q8	3052	2677	2674	2674
q9	8054	7983	7944	7944
q10	3889	3689	3216	3216
q11	631	435	415	415
q12	682	713	530	530
q13	3578	3856	3296	3296
q14	329	334	299	299
q15	q16	679	708	640	640
q17	8245	7059	6818	6818
q18	13026	12238	12841	12238
q19	945	855	837	837
q20	2153	2130	1925	1925
q21	5513	4535	4665	4535
q22	480	458	405	405
Total cold run time: 72624 ms
Total hot run time: 65484 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154368 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f81331d40ed5dee61cec000d9025ae3f52d349d8, data reload: false

query5	4311	528	398	398
query6	426	206	190	190
query7	4812	448	243	243
query8	320	163	150	150
query9	8802	3224	3248	3224
query10	484	301	249	249
query11	6010	3858	3575	3575
query12	140	91	87	87
query13	1251	444	321	321
query14	6135	3926	3649	3649
query14_1	3469	3420	3439	3420
query15	206	207	189	189
query16	943	410	402	402
query17	860	647	544	544
query18	2417	420	313	313
query19	205	166	134	134
query20	84	83	85	83
query21	214	127	105	105
query22	13022	12952	12768	12768
query23	13508	13000	12775	12775
query23_1	12850	12809	12602	12602
query24	7367	836	473	473
query24_1	492	485	490	485
query25	534	424	367	367
query26	1272	259	142	142
query27	2756	465	279	279
query28	4641	2057	2052	2052
query29	1571	602	459	459
query30	299	219	188	188
query31	997	926	814	814
query32	148	92	97	92
query33	533	305	243	243
query34	971	860	518	518
query35	733	748	643	643
query36	834	829	727	727
query37	129	99	85	85
query38	1794	1736	1671	1671
query39	791	791	752	752
query39_1	740	736	741	736
query40	228	114	99	99
query41	71	67	72	67
query42	89	85	88	85
query43	347	362	314	314
query44	1318	717	721	717
query45	181	177	168	168
query46	847	967	583	583
query47	2790	2856	2783	2783
query48	305	365	218	218
query49	567	396	288	288
query50	677	278	217	217
query51	10622	10744	10548	10548
query52	76	79	75	75
query53	188	214	151	151
query54	226	197	200	197
query55	72	65	61	61
query56	241	210	211	210
query57	1668	1648	1580	1580
query58	281	268	261	261
query59	2126	2132	1980	1980
query60	289	241	222	222
query61	142	139	149	139
query62	389	344	283	283
query63	190	165	170	165
query64	2762	937	799	799
query65	3383	3290	3274	3274
query66	1762	405	296	296
query67	22807	22658	22424	22424
query68	3013	1018	617	617
query69	387	291	252	252
query70	846	780	770	770
query71	295	246	212	212
query72	2688	2567	2270	2270
query73	515	527	297	297
query74	4593	4524	4364	4364
query75	2338	2289	1901	1901
query76	2312	1003	602	602
query77	354	366	271	271
query78	9946	9672	9200	9200
query79	2075	855	554	554
query80	1439	422	357	357
query81	584	316	267	267
query82	581	138	111	111
query83	317	203	182	182
query84	317	126	104	104
query85	851	431	396	396
query86	378	246	233	233
query87	2001	1941	1801	1801
query88	3847	2796	2740	2740
query89	325	288	257	257
query90	1909	174	170	170
query91	165	146	127	127
query92	96	77	91	77
query93	1112	1034	580	580
query94	571	310	271	271
query95	569	318	301	301
query96	676	494	241	241
query97	2435	2457	2312	2312
query98	158	151	148	148
query99	767	761	663	663
Total cold run time: 239877 ms
Total hot run time: 154368 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f81331d40ed5dee61cec000d9025ae3f52d349d8, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.25	0.09	0.09
query4	1.62	0.10	0.11
query5	0.24	0.24	0.24
query6	1.17	0.68	0.69
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.34	0.34
query10	0.58	0.59	0.60
query11	0.20	0.15	0.13
query12	0.19	0.15	0.15
query13	0.50	0.50	0.49
query14	1.01	1.01	1.03
query15	0.64	0.61	0.60
query16	0.35	0.36	0.33
query17	1.17	1.14	1.09
query18	0.21	0.18	0.17
query19	2.06	1.98	1.90
query20	0.02	0.01	0.01
query21	15.49	0.21	0.14
query22	4.81	0.05	0.06
query23	16.28	0.32	0.11
query24	1.06	0.24	0.16
query25	0.11	0.05	0.05
query26	0.74	0.21	0.16
query27	0.04	0.04	0.03
query28	3.53	0.85	0.37
query29	12.47	4.18	3.25
query30	0.27	0.12	0.10
query31	2.78	0.56	0.33
query32	3.23	0.59	0.51
query33	3.26	3.26	3.20
query34	15.38	3.96	3.40
query35	3.33	3.36	3.32
query36	0.55	0.44	0.43
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 94.65 s
Total hot run time: 23.96 s

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

2 similar comments
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39
Gabriel39 force-pushed the dev/iceberg-rest-encryption branch from f81331d to 5822148 Compare November 16, 2026 03:44
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70343 at 5822148c76bfba0e53834311be4c17a8f0566285: no actionable new findings. All 20 changed files and the related integration/lifecycle paths were inspected. No existing inline finding needs to be retained as blocking, and no additional user focus was supplied.

Critical checkpoint conclusions:

  • Goal and proof: REST encrypted metadata groundwork uses SDK crypto while preserving READ_DISABLED and BACKEND_CAPABILITY_UNAVAILABLE SQL admission. HTTP connector tests cover descriptor decryption, historical snapshots and refresh; the independent fixture covers actual encrypted Parquet. Successful encrypted SQL execution remains outside this stage.
  • Scope and reuse: the small package-access bridge and FileIO decorator reuse SDK REST loading, configuration merging, encryption and the existing admission contract. The Checkstyle exception is narrowly scoped.
  • Concurrency and locks: lazy manager construction/replacement and close share a per-table monitor; metadata publication uses AtomicReference. Existing table/catalog leases retain resources through readers. No supported refresh/read failure or lock-order cycle was substantiated. Initial KMS setup holds only the table monitor; no catalog-wide lock was added.
  • Lifecycle: the metadata holder avoids a reference back to the SDK tracker's weak operations key. Idempotent adapter cleanup closes its KMS and table-specific IO while preserving shared catalog IO; cache failure/unwind and catalog-generation retirement were traced.
  • Configuration: no new setting. The existing read-intent flag remains default-off, and both disabled and enabled SQL admission reject before KMS initialization. Effective SDK FileIO properties retain catalog/server/table configuration; no new dynamic-configuration obligation.
  • Compatibility: connector API 32.0, version pin and public plugin baseline agree; metadata-method API is unchanged. Matching engine/plugins are required. No persisted format or FE/BE wire change.
  • Thrift/protobuf: no gensrc definitions changed, so field-ID and enum compatibility checkpoints do not apply.
  • Parallel paths: plaintext REST retains SDK operations; HMS admission, ordinary/COUNT/streaming/system/historical reads and wrapped vended credentials were inspected. Existing server-side scan restrictions remain in force.
  • Conditions and errors: retained keys still trigger admission; missing KMS/table-key states fail explicitly; typed descriptors preserve crypto metadata. Loaded encrypted operations reject commit, temp transactions, output and deletion rather than silently proceeding.
  • Test coverage: positive, negative, concurrent lazy initialization, resource ownership, refresh/history, credentials, SPI composition and admission cases cover the intended preparatory scope. Direct concurrent SDK refresh and plaintext-to-encrypted transitions are not comprehensively tested; no failing supported Doris path was established under the current execution fence.
  • Expected results: all 18 golden sections match fixture rows/counts and regression control flow; 128 encrypted statements expect READ_DISABLED. REST fixture namespaces/object paths are separate from HMS. Output generation provenance was not independently reproduced.
  • Observability: stable admission/missing-KMS/key classifications and existing cleanup logs are adequate for the currently fenced scope; no additional metric requirement identified.
  • Transactions and persistence: no EditLog, replay, failover or transaction-visibility representation changed. Existing-table encrypted mutations reject through loaded operations; separate SDK catalog-builder transaction overloads have no Doris caller in the inspected paths.
  • Writes and crash behavior: no encrypted SQL scan or new product data-write capability is enabled. Resource ownership and failure cleanup were inspected; native visible-version and delete-bitmap invariants are unaffected.
  • FE/BE variables: no new transmitted value or sending path was added.
  • Performance: KMS creation is lazy and reused per table; manager reconstruction follows metadata identity changes. No obvious redundant/unbounded operation was identified on supported paths.
  • Other concerns: no additional user focus. Packaging includes the bridge alongside the pinned SDK; SPI composition retains the existing JDBC flavor policy and immutable endpoint ordering. No unresolved candidate or uncovered changed path remains.

The SDK boundary was checked against the primary Iceberg 1.12.0 REST implementation. Review converged in 2 rounds after all assigned reviewers returned NO_NEW_VALUABLE_FINDINGS on the frozen payload and the final coverage sweep completed. This is static evidence only: no builds, tests, benchmarks or runtime interleaving checks were executed, and the author's execution claims were not independently reproduced.

Review converged after 2 round(s).

@Gabriel39
Gabriel39 force-pushed the dev/iceberg-rest-encryption branch from 5822148 to fc04c45 Compare November 17, 2026 05:44
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44026273876

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

### Background

Iceberg encrypted reads use table metadata to locate retained table keys and decrypt manifest lists, manifests and file descriptors. Doris currently limits this groundwork to HMS catalogs. The pinned Iceberg REST operations retain the plaintext encryption manager, even when the REST response contains an encrypted table's key chain.

### What problem does this PR solve?

Issue Number: close apache#70342
Related PR: apache#70192 (encrypted metadata groundwork)
PR Type: Feature

Problem Summary:

Adding REST to the catalog allowlist alone leaves encrypted metadata unreadable. This change supplies lazy, read-only REST encryption operations while retaining the SDK REST table, authentication, credentials, refresh and scan capabilities. Most of the change is unit tests and an independent REST encryption fixture/regression suite. REST now follows the same staged scope as HMS; the common encrypted SQL execution fence remains in place.

### Details

- **Change map:** 299 product/build/style lines, 895 unit/regression/fixture lines, 85 generated baseline/output lines and 38 documentation lines (insertions plus deletions).
- **Scope:** Iceberg v3 REST tables using the standard SDK encryption manager and configured KMS. The initial file target remains Parquet modular encryption with AES_GCM_V1; encrypted writes and unrestricted encrypted SQL execution are outside this PR.
- **Design:** A package-access bridge decorates the SDK's loaded REST table operations. Descriptor-aware FileIO decrypts encrypted metadata lazily using the effective catalog/server/table configuration. A standalone metadata holder avoids retaining weak FileIOTracker operation keys; refresh replaces the manager's key chain and reuses the table-owned KMS client. Idempotent close preserves shared catalog FileIO ownership and retires table-specific resources. Transactions, output files and deletion are rejected for encrypted REST tables.
- **Usage:** Configure an Iceberg REST catalog with the existing KMS properties. There is no new setting, dependency, service or port. The newly consumed outbound destination is `kms.endpoint`, controlled by catalog configuration and CREATE_PRIV for CREATE CATALOG or ALTER_PRIV for ALTER CATALOG. The provider declares it for the existing SSRF policy while retaining its JDBC driver URL checks. REST server configuration/table overrides remain inside the configured catalog's trust boundary.
- **Default behavior:** `enable_encryption_read` remains false. Disabled SQL reads fail with READ_DISABLED before KMS initialization. Enabling the existing intent flag still fails with BACKEND_CAPABILITY_UNAVAILABLE until protected key transport and backend admission are complete. Plaintext REST reads retain the SDK path.
- **Compatibility:** No storage or FE/BE wire changes. The additive security-surface composition method requires connector plugin API 33.0 under the repository's SPI contract; deploy the matching engine and connector plugins together. The plugin API baseline is regenerated and the unchanged metadata-method baseline is verified.
- **Behavior change:** REST joins HMS in the encrypted metadata target matrix. Missing KMS configuration and retained keys without the mandatory table-key property fail explicitly rather than selecting plaintext crypto. Encrypted REST mutation paths fail as read-only.

### Testing

- Local `build.sh --be --fe -j6`, ASAN: passed, followed by an isolated current-worktree FE/BE cluster run.
- Connector Iceberg tests with Maven build cache disabled: 1,892 tests, zero failures/errors, 10 existing skips. This includes real connector HTTP REST loading, actual encrypted SDK manifests and descriptor payloads, historical scans/refresh, shared/per-table ownership, concurrent lazy initialization, missing/denied/wrong KMS keys, read-only transactions and vended credential forwarding. The REST manager test fails against the original SDK plaintext operations.
- Connector SPI own suite: 154 tests passed, including both API baselines and the plugin version pin. Endpoint policy composition preserves the JDBC policy and immutable endpoint lists.
- FE Checkstyle: zero violations during product build and final connector test validation.
- New `test_iceberg_encrypted_rest_metadata`: output generated by the regression framework, then compared successfully; 18 golden checks and 128 expected READ_DISABLED errors across cache, batch and COUNT settings. The independent fixture checks two historical row sets, encrypted Parquet/manifest/list magic and independent file keys, plus missing KMS, bad credentials, denied decrypt and wrong-key failures.
- Existing `test_iceberg_encrypted_hms_metadata`: passed, including the SDK/Spark reference checks.
- Restarted the local FE with the existing read-intent flag enabled: 16 REST SQL queries rejected with BACKEND_CAPABILITY_UNAVAILABLE, while plaintext rows remained correct.
- Staged patch whitespace and secret scan: passed.

### Notes for Reviewers

The SQL regression deliberately verifies the staged admission contract; successful decryption is exercised by the connector HTTP unit tests and an independent SDK reference reader over real encrypted files. It does not claim that this PR enables encrypted SQL scans. Start with DorisRESTSessionCatalog and IcebergRestEncryptionFileIO for the ownership and metadata-generation boundaries.

### Release note

Extend Iceberg encrypted metadata read groundwork to REST Catalog with standard SDK encryption and configured KMS. The existing default-off setting and encrypted SQL execution fence remain in effect.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; connector documentation and a regression reproduction command are included. User-facing documentation can follow when encrypted SQL execution is admitted.
- Pre-submission status: Ready; local validation passed, CI pending.

### AI Disclosure

- Tool: OpenAI Codex.
- Generated: Implementation, unit/regression tests, connector documentation and this description.
- Human review: Pending maintainer review; the agent performed a full self-review and local validation.
### Background

Encrypted REST tables reject transactions during creation as part of their read-only contract. SDK temporary operations retain virtual FileIO dispatch.

### What problem does this PR solve?

Issue Number: apache#70342
Related PR: None
PR Type: Build/CI/Chore

Problem Summary:

Correct the transaction-guard comment to describe eager read-only enforcement before callers can stage mutable metadata or initialize KMS for a write. Remove the inaccurate claim that SDK temporary operations expose raw FileIO. Executable code is unchanged.

### Details

- **Affected:** The explanatory comment beside the encrypted REST transaction guard.
- **Developer and release impact:** No behavior change; the comment now matches the pinned SDK implementation.
- **License and third-party:** No dependency or license changes.
- **Rollback:** Restore the prior comment only.

### Testing

Verified the SDK 1.12.0 temporary operations bytecode and inspected the complete follow-up diff. FE connector Checkstyle validation passed. Existing unit, regression and ASAN build evidence applies to the unchanged executable implementation.

### Release note

None.

### Check List (For Author)

- Behavior changed: No.
- Does this need documentation: No.
- Pre-submission status: Ready.

### AI Disclosure

- Tool: OpenAI Codex.
- Generated: Comment and commit description.
- Human review: Pending maintainer review; the agent reviewed the complete diff.
@Gabriel39
Gabriel39 force-pushed the dev/iceberg-rest-encryption branch from fc04c45 to 8186868 Compare November 17, 2026 06:54
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44034795317

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70343 at 8186868f80bd29be6e50923e9fadff518546ce13: no actionable new findings. All 20 changed files and related callers were inspected by the main reviewer, two complete-scope reviewers and a separate ownership/refresh reviewer. No existing inline findings require carrying forward, and no additional user focus was supplied.

Reviewed flow:

REST catalog load -> SDK table/operations -> lazy descriptor-aware FileIO/KMS
statement/cache lease -> frozen metadata -> common encrypted-read admission
lease release -> adapter cleanup -> catalog-generation retirement

Critical checkpoint conclusions:

  • Goal and proof: REST encrypted-metadata groundwork reuses SDK crypto while retaining READ_DISABLED and BACKEND_CAPABILITY_UNAVAILABLE SQL admission. HTTP connector unit tests cover encrypted manifests/lists/descriptors, historical snapshots and refresh; an independent fixture covers actual encrypted Parquet. These test sources were inspected, not executed.
  • Scope and reuse: the package-access bridge and FileIO decorator use existing REST loading, effective configuration and standard encryption. The Checkstyle exception is limited to the bridge's ImportControl rule.
  • Concurrency and locks: lazy manager initialization/replacement and close share one per-table monitor; refreshed encryption metadata uses AtomicReference. Existing statement/cache/catalog owners retain resources through supported readers. No new lock-order cycle or supported concurrent-refresh failure was established. This does not certify arbitrary concurrent calls to the raw SDK refresh method.
  • Lifecycle: the metadata holder avoids retaining the SDK tracker's weak operations key. Frozen snapshot operations retain the original operations; idempotent adapter cleanup closes its KMS and table-specific IO while preserving shared catalog IO. Catalog-generation leases defer retirement through borrower completion.
  • Configuration: no new setting or default. The existing default-off read flag and enabled-mode backend fence reject encrypted SQL before KMS initialization. Effective SDK FileIO properties carry catalog/server/table configuration; refresh preserves the pinned SDK's configuration contract.
  • Compatibility: the additive SPI method, plugin API 33.0, version test and public API baseline agree. The metadata-method surface is unchanged. Matching engine/plugins are required; no storage-format or FE/BE wire change was introduced.
  • Thrift/protobuf IDs: not applicable; no gensrc definitions changed.
  • Parallel paths: HMS admission remains; plaintext REST keeps SDK operations. Ordinary, COUNT, batch/streaming, historical and system-table paths were traced. Vended credentials remain accessible through both authentication and encryption decorators.
  • Conditions and errors: retained keys trigger encrypted-table detection; missing KMS/current table key fail explicitly. Typed descriptors retain encryption metadata, while plaintext descriptors delegate. Normal fresh REST loads select operations from current metadata. A direct plaintext-to-encrypted raw SDK refresh retains plaintext operations, but the common Doris admission fence prevents an admitted encrypted read.
  • Test coverage: inspected positive/negative crypto, lazy/concurrent initialization, ownership, history/refresh, mutation rejection, credential forwarding, SPI composition/version and SQL admission tests. Arbitrary refresh/read/close overlap and plaintext-to-encrypted transitions are not comprehensively tested; no failing supported current Doris path was found. Runtime validation remains outside this review.
  • Expected results: all 18 golden checks match fixture rows/counts and cache/batch/COUNT loops; 128 encrypted statements expect READ_DISABLED. REST and HMS fixture locations are separate. Output-generation provenance was not independently reproduced.
  • Observability: stable encryption classifications and surrounding table-load/cleanup errors provide context for this staged scope; no new critical metric requirement was identified.
  • Transactions and persistence: loaded encrypted operations reject temporary transactions and commit; no EditLog, replay, failover or transaction-visibility protocol changed. Separate SDK create/replace-transaction overloads have no Doris production caller in the inspected paths.
  • Data writes and crash behavior: no encrypted SQL execution or new production data-write capability is enabled. FileIO output/deletion reject on the encrypted adapter; resource cleanup and failure/unwind paths were traced. Native visible-version/delete-bitmap invariants are unaffected.
  • FE/BE variables: no new transmitted value or sending path; encrypted task dispatch stays fenced.
  • Performance: KMS setup is lazy and reused per table; manager reconstruction follows metadata identity changes. Credential extraction preserves the existing granularity. No obvious redundant/unbounded operation was identified on supported paths; no benchmark ran.
  • Other concerns and user focus: no additional focus was supplied. SPI composition preserves the JDBC flavor policy and secret declarations along with endpoint ordering/immutability. No unresolved candidate or uncovered changed path remains within the current staged scope.

Review converged in round 1: all assigned reviewers returned NO_NEW_VALUABLE_FINDINGS, required coverage is complete, and every initial risk was dismissed with evidence after independent verification and the final sweep. Validation was static only: no builds, tests, benchmarks, runtime interleaving checks, source edits or submodule initialization. The PR author's execution claims were not independently reproduced.

Review converged after 1 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support encrypted Iceberg metadata through REST Catalog

2 participants