Skip to content

[feature](iceberg) Read Parquet deletes with independent file keys - #70321

Merged
Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:dev/iceberg-encrypted-delete-reads
Nov 17, 2026
Merged

Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:dev/iceberg-encrypted-delete-reads

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Background

Iceberg data, position-delete and equality-delete files can each have a different native Parquet encryption key and AAD. The existing implementation rejects encrypted deletes during FE planning, and the native table/delete readers do not yet receive their independent descriptors.

What problem does this PR solve?

Issue Number: close #70316
Related PR: #70246 (file crypto), #70248 (native encrypted pages), #70314 (independent index/Bloom work)
PR Type: Feature

Problem Summary:

Implement D8: preserve each Parquet delete file's own key/AAD from FE planning to the physical reader, including the native position_deletes metadata reader. Keep decrypted position/equality objects and converted equality keys out of shared split caches so a warm path cannot authorize a later split with another key. Reject encrypted ORC/Puffin descriptors and keep metadata-only aggregates from bypassing delete authentication. This branch is based directly on master and does not depend on #70314.

Details

  • Change map: About 200 changed production lines connect immutable FE descriptors, a neutral TableReader encryption hook, delete-reader construction and cache bypass. Most changes are coverage: six BE scenarios with multiple key/file variants, FE planning tests, a Java reference fixture, SQL checks and generated golden results. Fixture documentation explains reproduction and the admission boundary.
  • Scope: Independent Parquet data/position/equality keys; mixed plaintext/encrypted files; native position_deletes; unsupported encrypted ORC/Puffin rejection; metadata COUNT protection.
  • Design: The generic TableReader receives a split-local descriptor through a protected hook. Iceberg supplies the data descriptor, while each delete reader uses only its own descriptor. Fresh Thrift buffers preserve carrier immutability. Encrypted decoded positions, equality filters and type-converted filters bypass shared caches; plaintext caching remains intact. No key bytes enter cache identities. The metadata-only COUNT capability probe returns false for per-file encryption instead of advertising support or throwing an execution error. It projects key metadata from live manifest entries without opening payloads or retaining file-task lists.
  • Usage: Existing physical-reader encryption descriptors support controlled native integrations. No configuration, dependency or wire-schema field is added. Reference fixtures reuse the existing encryption fixture service and its pinned Iceberg 1.12.0 jars.
  • Default behavior: Encrypted SQL admission remains closed pending protected key transport and cache integration. Encrypted files are reauthenticated per split, including after a successful read of the same path. Metadata aggregate rejection remains explicit.
  • Compatibility: Plaintext reads and delete semantics retain their existing behavior. The common plaintext FE carrier avoids an extra allocation. V3 coverage uses V2 snapshots upgraded to V3, preserving their existing Parquet position deletes.

Testing

Local verification of this branch:

  • build.sh --be --fe -j 6 with ASAN: passed, including the GLIBC 2.17 gate and FE Checkstyle.
  • run-be-ut.sh --run --filter='*Iceberg*:*FileEncryption*:*FileScanRange*:*Parquet*' -j 6: 519 tests, 517 passed and 2 existing opt-in microbenchmarks skipped. A focused source selection was used; the full BE UT suite was not run.
  • BE coverage includes all eight data/position/equality encryption combinations with distinct keys, repeated splits, warmed paths followed by missing/wrong keys and wrong AAD, recovery through abort_split, independent converted equality filters, metadata-table row projection, unsupported encrypted formats and metadata COUNT rejection before I/O. Four initial scenarios failed on master before the implementation.
  • Iceberg connector module tests, with Maven build cache disabled and Checkstyle enabled: 1,886 tests, zero failures/errors, 10 existing skips. Nine capability regressions cover independently encrypted data/deletes, both delete types, dangling-delete settings, empty tables, pinned snapshots and removed encrypted entries; seven failed before the capability fix and all nine pass afterward.
  • Independent Java reference reads verify 16 V2/upgraded-V3 tables, actual encrypted/plaintext payload magic, distinct descriptor keys/AAD and exact rows after applying both delete types.
  • test_iceberg_encryption_delete_reader: passed generation and comparison modes through run-regression-test.sh, against this branch's FE and ASAN BE. The matrix covers eager/lazy planning and COUNT pushdown on/off: 80 golden SQL checks and 176 explicit encrypted-read rejections. Plaintext position-delete metadata remains readable when only data or equality files are encrypted. Equality deletes make these SQL COUNT cases follow normal scan/dispatch; dedicated FE/BE UTs cover early metadata-only COUNT rejection.
  • Existing test_iceberg_encryption_native_reader: passed against the same deployment.
  • The new SQL suite reproduced master's earlier DELETE_READ_UNAVAILABLE rejection before FE descriptor propagation. Expected results were generated by the regression framework; only trailing EOF whitespace was normalized.
  • Clang-format 16, changed-line clang-tidy, build hygiene, whitespace checks and secret scanning: passed.

Notes for Reviewers

The fixture service compiles Java at startup. Restart encryption-fixture from the updated checkout before running the new SQL suite; it reuses the existing services, ports and reference jars. These public synthetic file-key fixtures keep table metadata plaintext to isolate native-file planning from KMS/catalog admission. Successful encrypted SQL execution remains outside this PR. The COUNT capability probe now performs a bounded-memory scan of live data-manifest key metadata, rather than reading only manifest-list counters, because those counters cannot establish whether individual files are encrypted.

Release note

None. This completes native encrypted-delete integration while retaining the SQL admission fence.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No; internal staged integration and fixture documentation are included.
  • Pre-submission status: Ready. Full CI remains to run; encrypted SQL admission and shared decrypted cache support remain separate work.

AI Disclosure

  • Tool: OpenAI Codex (GPT-6)
  • Generated: Implementation, tests, reference fixture and this description.
  • Human review: Pending maintainer review; the agent self-reviewed the complete diff and ran the verification above.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70321
commit: 4a1fb77e72f27fc6d7027ba0db06412e3514eba4
base: dcf82859fd0f9d660f62d04168df2945c7bfd9cf
reviewed_at: 2026-11-06T12:06+08:00
reviewer: Gabriel39
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Static review only; no build, test, cluster or benchmark was run during this review.
  • docker/thirdparties/docker-compose/iceberg/encryption/README.md:31 — Successful encrypted SQL execution remains intentionally fenced; the SQL matrix checks plaintext results and encrypted-read rejection.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28728 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a1fb77e72f27fc6d7027ba0db06412e3514eba4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17607	5178	5079	5079
q2	2147	302	253	253
q3	10237	1373	831	831
q4	4678	457	340	340
q5	7423	799	531	531
q6	177	182	152	152
q7	742	781	589	589
q8	9246	1344	1091	1091
q9	5585	4200	4171	4171
q10	6822	1298	1023	1023
q11	430	244	228	228
q12	635	392	277	277
q13	18107	3629	3033	3033
q14	273	277	251	251
q15	q16	729	713	672	672
q17	1183	738	572	572
q18	7065	6205	6126	6126
q19	1171	971	579	579
q20	383	331	234	234
q21	5552	2483	2420	2420
q22	383	362	276	276
Total cold run time: 100575 ms
Total hot run time: 28728 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5950	5935	6087	5935
q2	745	590	585	585
q3	5663	5822	5344	5344
q4	1973	2168	1541	1541
q5	5790	5968	5574	5574
q6	239	186	140	140
q7	2316	2034	1797	1797
q8	2997	2663	2664	2663
q9	7989	7479	7472	7472
q10	3666	3578	3129	3129
q11	577	408	375	375
q12	670	692	501	501
q13	3299	3622	3033	3033
q14	289	290	264	264
q15	q16	661	691	621	621
q17	7853	7026	6886	6886
q18	12990	12208	12856	12208
q19	909	779	781	779
q20	2136	2139	1957	1957
q21	5594	4581	4690	4581
q22	518	433	415	415
Total cold run time: 72824 ms
Total hot run time: 65800 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154257 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a1fb77e72f27fc6d7027ba0db06412e3514eba4, data reload: false

query5	4309	514	407	407
query6	433	218	190	190
query7	4813	435	244	244
query8	337	173	157	157
query9	8823	3214	3223	3214
query10	477	312	261	261
query11	5972	3854	3562	3562
query12	148	91	89	89
query13	1263	433	341	341
query14	6125	3939	3650	3650
query14_1	3375	3450	3472	3450
query15	216	204	184	184
query16	948	475	443	443
query17	864	659	571	571
query18	2420	430	312	312
query19	212	179	138	138
query20	90	87	85	85
query21	210	126	106	106
query22	12992	12924	12827	12827
query23	13614	12859	12360	12360
query23_1	12654	12480	12623	12480
query24	7388	825	469	469
query24_1	502	480	481	480
query25	536	426	375	375
query26	1275	253	141	141
query27	2778	449	280	280
query28	4604	2055	2047	2047
query29	1564	616	469	469
query30	299	224	181	181
query31	1023	951	825	825
query32	152	89	98	89
query33	553	315	247	247
query34	890	854	500	500
query35	734	752	656	656
query36	827	825	703	703
query37	131	99	122	99
query38	1774	1730	1666	1666
query39	799	785	744	744
query39_1	726	715	741	715
query40	216	112	94	94
query41	66	61	61	61
query42	88	90	84	84
query43	338	360	310	310
query44	1336	706	713	706
query45	178	182	176	176
query46	816	958	563	563
query47	2704	2892	2953	2892
query48	291	300	220	220
query49	576	402	297	297
query50	680	274	226	226
query51	10525	10523	10547	10523
query52	77	77	68	68
query53	186	209	151	151
query54	238	204	181	181
query55	74	65	61	61
query56	240	218	216	216
query57	1652	1603	1631	1603
query58	291	262	260	260
query59	2145	2192	1967	1967
query60	283	247	221	221
query61	153	139	150	139
query62	386	330	284	284
query63	192	162	167	162
query64	2750	952	808	808
query65	3339	3269	3283	3269
query66	1773	428	301	301
query67	22723	22624	22728	22624
query68	2985	995	630	630
query69	387	285	258	258
query70	872	781	759	759
query71	304	240	210	210
query72	2738	2607	2277	2277
query73	510	528	293	293
query74	4650	4533	4348	4348
query75	2363	2283	1912	1912
query76	2331	1024	594	594
query77	363	367	259	259
query78	9798	9773	9149	9149
query79	994	919	545	545
query80	565	427	348	348
query81	541	324	266	266
query82	582	143	113	113
query83	315	197	189	189
query84	316	131	104	104
query85	814	433	378	378
query86	304	239	235	235
query87	1959	1959	1802	1802
query88	3777	2782	2783	2782
query89	325	286	238	238
query90	2006	173	167	167
query91	165	153	126	126
query92	100	92	93	92
query93	1029	1011	598	598
query94	560	332	273	273
query95	566	415	309	309
query96	679	486	240	240
query97	2474	2438	2371	2371
query98	165	157	145	145
query99	768	768	654	654
Total cold run time: 237273 ms
Total hot run time: 154257 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a1fb77e72f27fc6d7027ba0db06412e3514eba4, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.60	0.10	0.10
query5	0.23	0.24	0.25
query6	1.17	0.67	0.68
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.36	0.34
query10	0.61	0.59	0.57
query11	0.21	0.14	0.13
query12	0.18	0.15	0.14
query13	0.49	0.48	0.49
query14	1.04	1.01	1.02
query15	0.61	0.60	0.61
query16	0.31	0.34	0.34
query17	1.14	1.13	1.14
query18	0.21	0.19	0.17
query19	2.12	2.00	2.01
query20	0.02	0.01	0.01
query21	15.49	0.20	0.15
query22	4.83	0.06	0.05
query23	16.27	0.31	0.12
query24	1.07	0.22	0.17
query25	0.08	0.05	0.04
query26	0.76	0.21	0.16
query27	0.03	0.03	0.04
query28	3.73	0.84	0.33
query29	12.47	4.10	3.23
query30	0.27	0.12	0.09
query31	2.77	0.58	0.32
query32	3.22	0.61	0.50
query33	3.18	3.19	3.30
query34	15.33	4.02	3.40
query35	3.35	3.37	3.34
query36	0.56	0.46	0.44
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 94.7 s
Total hot run time: 24.02 s

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR 70321 at head 4a1fb77 found one actionable P2 issue.

The FE now rejects encrypted files during COUNT range construction, but canServeMetadataOnlyCount() delegates to a manifest-only proof that does not account for per-file encrypted data. A valid encrypted DataFile with manifest row counters can therefore advertise metadata-only COUNT support and then fail when the collapsed planner reaches its encryption check. The new encrypted-delete guard also throws from the same boolean capability path instead of returning an unsupported result. The inline comment identifies the changed guard and the required consistency fix.

I traced all changed FE carriers (data files, independent position/equality deletes, native $position_deletes, deletion vectors, malformed metadata, mixed snapshots, streaming and both COUNT planners) and the BE TableReader/ParquetReader lifecycle. Data and delete descriptors remain independent; encrypted delete results stay scan-local; unsupported encrypted ORC/Puffin and aggregate paths are rejected before I/O; split reuse and nested-reader cleanup were verified. The new Java reference fixture, Docker classpath/endpoints, FE/BE tests, and regression control flow were also reviewed with no additional actionable defect.

The required FileScannerV2 architecture, cache, predicate, lifecycle, and static review checkpoints were applied. No user-provided review focus was supplied. Validation was static inspection only; no builds or tests were executed under the review contract. Existing inline review context was empty, and no existing P0/P1 finding applies.

The review converged after two bounded rounds: the accepted finding was independently rechecked, and all focused reviewers returned NO_NEW_VALUABLE_FINDINGS on unchanged bytes.

Review converged after 2 round(s).

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Nov 6, 2026
### Background

Iceberg advertises metadata-only COUNT support through a boolean capability probe. Data manifest counters prove row totals but do not indicate whether individual data files carry encryption key metadata.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70321 (review follow-up)
PR Type: Bugfix

Problem Summary:

The capability probe could return true for encrypted data files and throw COUNT_READ_UNAVAILABLE for encrypted delete files. Return false for unsupported file-level encryption while preserving execution-time rejection and snapshot-scoped plaintext counts.

### Details

- **Frequency:** Snapshots containing per-file encrypted data or encrypted deletes inspected for metadata-only COUNT.
- **Root cause:** The probe trusted aggregate data counters and reused a delete helper that throws an execution-specific encryption exception.
- **Why existing tests missed it:** Encryption tests covered descriptor propagation and COUNT execution rejection but not boolean capability probing on mixed encrypted/plaintext snapshots.
- **Fix:** Translate only the deterministic encryption exception to false in the probe, then stream projected key metadata from live data manifest entries before advertising support. Manifest readers close on early exit; no payload I/O or retained task list is added.
- **Other paths and branches:** Actual COUNT execution remains fail-closed. Existing empty, plaintext, delete-netting and historical snapshot behavior is preserved. File-level inspection is O(live files) with bounded memory.
- **Behavior change:** Unsupported file encryption returns false from the capability probe rather than true or an execution exception.

### Testing

- New focused tests: nine cases, seven expected failures before the fix and zero failures/errors after it.
- Iceberg connector tests with Maven build cache disabled: 1,886 tests, zero failures/errors, 10 existing skips.
- FE Checkstyle: zero violations. Whitespace verification and secret scan passed.
- SQL suites were not rerun for this capability-only follow-up.

### Release note

Iceberg metadata-only COUNT capability checks consistently report unsupported file-level encryption.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready; local checks passed and CI will be retriggered.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6)
- Generated: Code, tests and commit description.
- Human review: Pending maintainer review; full diff self-reviewed by the agent.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of #70321 at d781ffb: no new actionable findings. Review converged in round 1 after FE, BE and dedicated risk reviewers all returned NO_NEW_VALUABLE_FINDINGS, followed by independent verification and a final sweep of all 16 changed paths.

Existing review context: P2 comment 4489140690 is fixed on this head. The boolean COUNT capability probe catches only EncryptedCountException, streams live data-file key metadata and returns false for encryption. Execution still rejects independent encrypted data/delete descriptors. No independently confirmed existing P0/P1 finding remains, and no existing concern is reposted. No additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and correctness: code preserves each data/position/equality file's own key/AAD through FE carriers and native reader construction, including native position_deletes. Inspected tests cover exact surviving rows and descriptor independence; encrypted SQL admission remains deliberately closed.
  2. Scope and reuse: the focused integration reuses the immutable encryption carrier, existing optional wire fields and native Parquet constructor through a small neutral TableReader hook.
  3. Concurrency and locking: no new threads or locks; split/reader state owns keys and decoded filters, immutable carriers clone mutable Thrift buffers, and plaintext cache publication retains its existing contract. No new lock-order issue identified.
  4. Lifecycle and memory: manifest readers close on early returns; encrypted position maps use RAII and copy selected rows before destruction; equality predicates retain Block/index ownership. Nested metadata readers close before their owned range changes, and physical readers copy descriptors. No new static-initialization dependency or ownership cycle identified.
  5. Configuration: none added; no dynamic-update requirement introduced.
  6. Compatibility: no new dependency, stored format or external API change; plaintext cache paths remain. The fixture upgrades V2 snapshots before testing retained position deletes in V3, consistent with the Iceberg specification.
  7. Thrift/protobuf IDs: neither schema tree changes; existing optional encryption fields are reused.
  8. Parallel paths: eager/streaming data planning, position/equality auxiliary readers, native position_deletes and v3 rewrite-supply serialization were inspected; encrypted unsupported formats reject explicitly.
  9. Conditions and errors: encryption-presence, unsupported-format, aggregate and cache-bypass checks have concrete documented triggers. Missing/wrong keys or AAD cannot reuse decoded encrypted cache entries; unrelated metadata errors still propagate.
  10. Test coverage: inspected FE independent/malformed descriptors and COUNT capability cases; BE eight-combination, repeated-split, warm failure, abort/retry, equality-conversion, metadata-row and unsupported-format cases; Java reference fixtures and SQL eager/lazy plus COUNT settings. Successful encrypted SQL execution is outside the stated scope.
  11. Changed results: all 80 ordered golden blocks match fixture delete arithmetic and suite order; the matrix contains 176 expected encrypted-read rejections. Output-generation provenance is author-reported and was not independently rerun.
  12. Observability: existing classifications and reader/cache profiles remain; encrypted equality cache misses and hash-index memory are accounted. No new per-row logging or missing critical diagnostic identified.
  13. Transactions and persistence: no production EditLog, transaction, visibility/version or failover protocol changes.
  14. Writes and crash behavior: production changes are read planning/consumption; no new data-write atomicity requirement. Partial reader cleanup and abort/retry paths were inspected. Fixture writes are isolated test data.
  15. FE/BE variables: no new wire variable; both normal delete serialization and the distinct native metadata-table branch carry their own existing encryption_info. Existing eager/lazy dispatch guards remain in place.
  16. Performance: COUNT capability now performs O(live manifest entries) work in bounded memory; it projects key metadata without opening payload files or retaining tasks. Encrypted per-split delete rereads intentionally avoid shared decrypted caches; plaintext caching remains. No runtime performance claim is made.
  17. Other risks and focus: initial COUNT, cache and descriptor-lifetime risks were dismissed with concrete caller/consumer evidence. Final rechecks found no unresolved candidate or static coverage gap. No extra user focus was supplied.

Validation limits: static inspection only. No builds, tests, benchmarks, sanitizers, build-hygiene commands or final ELF compatibility checks were executed in this review. Author-reported runs were not independently verified. No repository source was modified. These execution limits do not represent an unfinished assigned static-review scope.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28315 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d781ffb4353186e5c3b9d9695a9922c1d9dc8cbc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17609	5202	5066	5066
q2	2132	315	245	245
q3	10266	1360	830	830
q4	4677	453	333	333
q5	7410	778	541	541
q6	187	187	157	157
q7	765	785	603	603
q8	9245	1366	1081	1081
q9	5587	4180	4165	4165
q10	6820	1280	1030	1030
q11	430	240	221	221
q12	641	398	276	276
q13	18103	3718	3048	3048
q14	284	270	247	247
q15	q16	736	711	652	652
q17	1191	756	563	563
q18	6960	6168	6107	6107
q19	1236	969	595	595
q20	374	326	221	221
q21	5583	2080	2301	2080
q22	348	292	254	254
Total cold run time: 100584 ms
Total hot run time: 28315 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5573	5564	5605	5564
q2	671	550	529	529
q3	5252	5376	4981	4981
q4	1972	2145	1496	1496
q5	5270	5183	5723	5183
q6	251	204	153	153
q7	2349	2079	1990	1990
q8	3034	2815	2716	2716
q9	8074	8056	8080	8056
q10	3721	3760	3231	3231
q11	632	452	410	410
q12	679	672	519	519
q13	3495	3919	3245	3245
q14	322	303	281	281
q15	q16	701	734	643	643
q17	8376	7674	6943	6943
q18	12942	12198	12843	12198
q19	894	763	758	758
q20	2131	2138	1900	1900
q21	5499	4631	4643	4631
q22	491	431	411	411
Total cold run time: 72329 ms
Total hot run time: 65838 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153625 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d781ffb4353186e5c3b9d9695a9922c1d9dc8cbc, data reload: false

query5	4305	521	369	369
query6	421	211	204	204
query7	4796	426	226	226
query8	332	170	151	151
query9	8787	3163	3173	3163
query10	461	313	267	267
query11	6076	3844	3557	3557
query12	152	92	89	89
query13	1280	441	330	330
query14	6085	3885	3593	3593
query14_1	3402	3403	3423	3403
query15	209	199	182	182
query16	937	431	415	415
query17	850	650	538	538
query18	2412	412	310	310
query19	207	160	147	147
query20	84	86	83	83
query21	213	127	106	106
query22	12927	12880	12750	12750
query23	13549	12902	12429	12429
query23_1	12655	12653	12446	12446
query24	7378	822	474	474
query24_1	482	496	482	482
query25	538	423	366	366
query26	1261	263	143	143
query27	2766	438	271	271
query28	4621	2051	2007	2007
query29	1575	605	469	469
query30	300	221	184	184
query31	1028	930	811	811
query32	144	97	96	96
query33	554	305	250	250
query34	902	842	508	508
query35	736	753	645	645
query36	816	814	737	737
query37	121	98	99	98
query38	1787	1712	1668	1668
query39	788	796	744	744
query39_1	740	737	736	736
query40	219	116	100	100
query41	70	71	71	71
query42	92	86	85	85
query43	339	364	313	313
query44	1301	719	710	710
query45	174	176	167	167
query46	831	971	564	564
query47	3000	2857	2837	2837
query48	297	311	263	263
query49	570	392	297	297
query50	664	278	203	203
query51	10633	10866	10507	10507
query52	86	86	75	75
query53	182	209	149	149
query54	249	200	196	196
query55	71	72	63	63
query56	224	222	208	208
query57	1648	1573	1546	1546
query58	295	261	260	260
query59	2168	2232	1958	1958
query60	290	242	226	226
query61	151	149	168	149
query62	404	333	290	290
query63	188	158	159	158
query64	2730	967	784	784
query65	3322	3255	3264	3255
query66	1786	408	307	307
query67	22660	22618	22436	22436
query68	3074	990	621	621
query69	385	287	252	252
query70	852	792	782	782
query71	284	242	214	214
query72	2837	2698	2309	2309
query73	508	526	292	292
query74	4607	4526	4362	4362
query75	2368	2280	1945	1945
query76	2325	993	592	592
query77	343	381	277	277
query78	9769	9670	9152	9152
query79	974	892	542	542
query80	678	420	354	354
query81	548	317	266	266
query82	575	137	110	110
query83	274	193	180	180
query84	306	124	98	98
query85	912	424	386	386
query86	302	238	236	236
query87	1968	1941	1813	1813
query88	3758	2761	2742	2742
query89	335	285	249	249
query90	1948	166	170	166
query91	166	152	134	134
query92	102	87	91	87
query93	990	995	579	579
query94	567	307	262	262
query95	571	387	294	294
query96	685	479	245	245
query97	2442	2480	2337	2337
query98	180	153	152	152
query99	763	761	645	645
Total cold run time: 237436 ms
Total hot run time: 153625 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d781ffb4353186e5c3b9d9695a9922c1d9dc8cbc, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.26	0.10	0.08
query4	1.60	0.10	0.11
query5	0.23	0.24	0.24
query6	1.17	0.68	0.67
query7	0.03	0.01	0.00
query8	0.05	0.04	0.04
query9	0.41	0.35	0.36
query10	0.57	0.57	0.62
query11	0.20	0.14	0.14
query12	0.18	0.14	0.14
query13	0.49	0.49	0.49
query14	1.04	1.00	0.99
query15	0.61	0.61	0.60
query16	0.36	0.35	0.35
query17	1.11	1.14	1.13
query18	0.21	0.19	0.18
query19	2.12	2.03	1.99
query20	0.02	0.01	0.01
query21	15.42	0.18	0.13
query22	4.94	0.05	0.06
query23	16.27	0.33	0.12
query24	1.13	0.27	0.17
query25	0.10	0.04	0.04
query26	0.74	0.22	0.16
query27	0.04	0.03	0.03
query28	3.80	0.78	0.34
query29	12.51	4.18	3.22
query30	0.28	0.12	0.09
query31	2.77	0.55	0.33
query32	3.22	0.60	0.52
query33	3.10	3.21	3.19
query34	15.28	4.02	3.36
query35	3.33	3.36	3.33
query36	0.56	0.46	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 94.77 s
Total hot run time: 23.99 s

924060929
924060929 previously approved these changes Nov 6, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.68% (32417/47898)
Line Coverage 52.56% (342599/651886)
Region Coverage 48.09% (278874/579943)
Branch Coverage 50.12% (132355/264091)

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

2 similar comments
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

### Background

Iceberg data, position-delete and equality-delete files can each have a different native Parquet encryption key and AAD. The existing implementation rejects encrypted deletes during FE planning, and the native table/delete readers do not yet receive their independent descriptors.

### What problem does this PR solve?

Issue Number: close apache#70316
Related PR: apache#70246 (file crypto), apache#70248 (native encrypted pages), apache#70314 (independent index/Bloom work)
PR Type: Feature

Problem Summary:

Implement D8: preserve each Parquet delete file's own key/AAD from FE planning to the physical reader, including the native `position_deletes` metadata reader. Keep decrypted position/equality objects and converted equality keys out of shared split caches so a warm path cannot authorize a later split with another key. Reject encrypted ORC/Puffin descriptors and keep metadata-only aggregates from bypassing delete authentication. This branch is based directly on master and does not depend on apache#70314.

### Details

- **Change map:** About 200 changed production lines connect immutable FE descriptors, a neutral TableReader encryption hook, delete-reader construction and cache bypass. Most changes are coverage: six BE scenarios with multiple key/file variants, FE planning tests, a Java reference fixture, SQL checks and generated golden results. Fixture documentation explains reproduction and the admission boundary.
- **Scope:** Independent Parquet data/position/equality keys; mixed plaintext/encrypted files; native `position_deletes`; unsupported encrypted ORC/Puffin rejection; metadata COUNT protection.
- **Design:** The generic TableReader receives a split-local descriptor through a protected hook. Iceberg supplies the data descriptor, while each delete reader uses only its own descriptor. Fresh Thrift buffers preserve carrier immutability. Encrypted decoded positions, equality filters and type-converted filters bypass shared caches; plaintext caching remains intact. No key bytes enter cache identities.
- **Usage:** Existing physical-reader encryption descriptors support controlled native integrations. No configuration, dependency or wire-schema field is added. Reference fixtures reuse the existing encryption fixture service and its pinned Iceberg 1.12.0 jars.
- **Default behavior:** Encrypted SQL admission remains closed pending protected key transport and cache integration. Encrypted files are reauthenticated per split, including after a successful read of the same path. Metadata aggregate rejection remains explicit.
- **Compatibility:** Plaintext reads and delete semantics retain their existing behavior. The common plaintext FE carrier avoids an extra allocation. V3 coverage uses V2 snapshots upgraded to V3, preserving their existing Parquet position deletes.

### Testing

Local verification of this branch:

- `build.sh --be --fe -j 6` with ASAN: passed, including the GLIBC 2.17 gate and FE Checkstyle.
- `run-be-ut.sh --run --filter='*Iceberg*:*FileEncryption*:*FileScanRange*:*Parquet*' -j 6`: 519 tests, 517 passed and 2 existing opt-in microbenchmarks skipped. A focused source selection was used; the full BE UT suite was not run.
- BE coverage includes all eight data/position/equality encryption combinations with distinct keys, repeated splits, warmed paths followed by missing/wrong keys and wrong AAD, recovery through `abort_split`, independent converted equality filters, metadata-table row projection, unsupported encrypted formats and metadata COUNT rejection before I/O. Four initial scenarios failed on master before the implementation.
- Iceberg connector module tests, with Maven build cache disabled and Checkstyle enabled: 1,877 tests, zero failures/errors, 10 existing skips.
- Independent Java reference reads verify 16 V2/upgraded-V3 tables, actual encrypted/plaintext payload magic, distinct descriptor keys/AAD and exact rows after applying both delete types.
- `test_iceberg_encryption_delete_reader`: passed generation and comparison modes through `run-regression-test.sh`, against this branch's FE and ASAN BE. The matrix covers eager/lazy planning and COUNT pushdown on/off: 80 golden SQL checks and 176 explicit encrypted-read rejections. Plaintext position-delete metadata remains readable when only data or equality files are encrypted. Equality deletes make these SQL COUNT cases follow normal scan/dispatch; dedicated FE/BE UTs cover early metadata-only COUNT rejection.
- Existing `test_iceberg_encryption_native_reader`: passed against the same deployment.
- The new SQL suite reproduced master's earlier `DELETE_READ_UNAVAILABLE` rejection before FE descriptor propagation. Expected results were generated by the regression framework; only trailing EOF whitespace was normalized.
- Clang-format 16, changed-line clang-tidy, build hygiene, whitespace checks and secret scanning: passed.

### Notes for Reviewers

The fixture service compiles Java at startup. Restart `encryption-fixture` from the updated checkout before running the new SQL suite; it reuses the existing services, ports and reference jars. These public synthetic file-key fixtures keep table metadata plaintext to isolate native-file planning from KMS/catalog admission. Successful encrypted SQL execution remains outside this PR.

### Release note

None. This completes native encrypted-delete integration while retaining the SQL admission fence.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No; internal staged integration and fixture documentation are included.
- Pre-submission status: Ready. Full CI remains to run; encrypted SQL admission and shared decrypted cache support remain separate work.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6)
- Generated: Implementation, tests, reference fixture and this description.
- Human review: Pending maintainer review; the agent self-reviewed the complete diff and ran the verification above.
### Background

Iceberg advertises metadata-only COUNT support through a boolean capability probe. Data manifest counters prove row totals but do not indicate whether individual data files carry encryption key metadata.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70321 (review follow-up)
PR Type: Bugfix

Problem Summary:

The capability probe could return true for encrypted data files and throw COUNT_READ_UNAVAILABLE for encrypted delete files. Return false for unsupported file-level encryption while preserving execution-time rejection and snapshot-scoped plaintext counts.

### Details

- **Frequency:** Snapshots containing per-file encrypted data or encrypted deletes inspected for metadata-only COUNT.
- **Root cause:** The probe trusted aggregate data counters and reused a delete helper that throws an execution-specific encryption exception.
- **Why existing tests missed it:** Encryption tests covered descriptor propagation and COUNT execution rejection but not boolean capability probing on mixed encrypted/plaintext snapshots.
- **Fix:** Translate only the deterministic encryption exception to false in the probe, then stream projected key metadata from live data manifest entries before advertising support. Manifest readers close on early exit; no payload I/O or retained task list is added.
- **Other paths and branches:** Actual COUNT execution remains fail-closed. Existing empty, plaintext, delete-netting and historical snapshot behavior is preserved. File-level inspection is O(live files) with bounded memory.
- **Behavior change:** Unsupported file encryption returns false from the capability probe rather than true or an execution exception.

### Testing

- New focused tests: nine cases, seven expected failures before the fix and zero failures/errors after it.
- Iceberg connector tests with Maven build cache disabled: 1,886 tests, zero failures/errors, 10 existing skips.
- FE Checkstyle: zero violations. Whitespace verification and secret scan passed.
- SQL suites were not rerun for this capability-only follow-up.

### Release note

Iceberg metadata-only COUNT capability checks consistently report unsupported file-level encryption.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready; local checks passed and CI will be retriggered.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6)
- Generated: Code, tests and commit description.
- Human review: Pending maintainer review; full diff self-reviewed by the agent.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44002401362

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44017695545

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44024742455

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44030119667

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39
Gabriel39 merged commit 3659256 into apache:master Nov 17, 2026
34 of 36 checks passed
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Nov 17, 2026
### Background

The spatial branch passes its Parquet spatial-annotation policy through data and delete readers. Upstream master added independent file-encryption descriptors to those same reader constructors and a generic table-reader hook.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032, apache#70321 (independent delete keys), apache#70314 (encrypted indexes)
PR Type: Build/CI/Chore

Problem Summary:

Merge master 43963ce into the spatial branch. Preserve each data/delete file's own encryption descriptor together with the existing spatial-annotation option, and retain both protected table-reader hooks. Keep upstream encrypted-cache isolation and the spatial read behavior intact.

### Details

- **Change map:** Manually reconcile three files: iceberg_reader.cpp, table_reader.cpp and table_reader.h. Other changes retain the upstream merge result. The earlier merge b6c42b5 already reconciles the connector API as major 33.
- **Affected:** Two Parquet reader construction sites and the table-reader hook declarations.
- **Developer and release impact:** Spatial readers retain their annotation policy while data and delete readers keep independent encryption keys. Existing encrypted-cache bypass, metadata-aggregate rejection and plaintext caching remain in place.
- **License and third-party:** No additional dependency or license change beyond upstream.
- **Rollback:** Revert this merge while retaining its first parent if necessary.

### Testing

- Final merged candidate: 291 FE tests in 39 classes passed with zero failures/errors/skips, including the complete connector SPI, IcebergWritePlanProviderTest and PluginApiVersionWiringTest.
- Final merged candidate: BE ASAN compiled and ran 895 tests in 55 suites: 893 passed, zero failed, two existing opt-in Parquet performance microbenchmarks skipped. Includes 17 executed encryption-related cases and the spatial/Iceberg/Parquet reader coverage.
- BE used the standard personal clang21 wrapper, GLIBC_COMPATIBILITY=OFF and a private TMPDIR. The task-owned dependency adapter was restored.
- Clang-format16, build hygiene, explicitly enabled full FE reactor CheckStyle and code whitespace checks passed. Changed-line clang-tidy validates the two implementations against the merged source; the compiler resource include path is provided by a task-private adapter.
- Complete product packaging, SQL regression, full unit-test matrix and a new full PR review were not run for this merge.

### Release note

None.

### Check List (For Author)

- Behavior changed: No additional behavior beyond the existing spatial branch and upstream changes.
- Does this need documentation: No
- Pre-submission status: Partially verified: targeted merge validation passed; complete community CI remains separate.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: The three conflict resolutions and this message.
- Human review: Pending author review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Read Iceberg Parquet deletes with independent file encryption keys

4 participants