Repository navigation
[feature](iceberg) Read Parquet deletes with independent file keys - #70321
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70321
commit: 4a1fb77e72f27fc6d7027ba0db06412e3514eba4
base: dcf82859fd0f9d660f62d04168df2945c7bfd9cf
reviewed_at: 2026-11-06T12:06+08:00
reviewer: Gabriel39
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
TPC-H: Total hot run time: 28728 ms |
TPC-DS: Total hot run time: 154257 ms |
ClickBench: Total hot run time: 24.02 s |
There was a problem hiding this comment.
Static review of PR 70321 at head 4a1fb77 found one actionable P2 issue.
The FE now rejects encrypted files during COUNT range construction, but canServeMetadataOnlyCount() delegates to a manifest-only proof that does not account for per-file encrypted data. A valid encrypted DataFile with manifest row counters can therefore advertise metadata-only COUNT support and then fail when the collapsed planner reaches its encryption check. The new encrypted-delete guard also throws from the same boolean capability path instead of returning an unsupported result. The inline comment identifies the changed guard and the required consistency fix.
I traced all changed FE carriers (data files, independent position/equality deletes, native $position_deletes, deletion vectors, malformed metadata, mixed snapshots, streaming and both COUNT planners) and the BE TableReader/ParquetReader lifecycle. Data and delete descriptors remain independent; encrypted delete results stay scan-local; unsupported encrypted ORC/Puffin and aggregate paths are rejected before I/O; split reuse and nested-reader cleanup were verified. The new Java reference fixture, Docker classpath/endpoints, FE/BE tests, and regression control flow were also reviewed with no additional actionable defect.
The required FileScannerV2 architecture, cache, predicate, lifecycle, and static review checkpoints were applied. No user-provided review focus was supplied. Validation was static inspection only; no builds or tests were executed under the review contract. Existing inline review context was empty, and no existing P0/P1 finding applies.
The review converged after two bounded rounds: the accepted finding was independently rechecked, and all focused reviewers returned NO_NEW_VALUABLE_FINDINGS on unchanged bytes.
Review converged after 2 round(s).
### Background Iceberg advertises metadata-only COUNT support through a boolean capability probe. Data manifest counters prove row totals but do not indicate whether individual data files carry encryption key metadata. ### What problem does this PR solve? Issue Number: None Related PR: apache#70321 (review follow-up) PR Type: Bugfix Problem Summary: The capability probe could return true for encrypted data files and throw COUNT_READ_UNAVAILABLE for encrypted delete files. Return false for unsupported file-level encryption while preserving execution-time rejection and snapshot-scoped plaintext counts. ### Details - **Frequency:** Snapshots containing per-file encrypted data or encrypted deletes inspected for metadata-only COUNT. - **Root cause:** The probe trusted aggregate data counters and reused a delete helper that throws an execution-specific encryption exception. - **Why existing tests missed it:** Encryption tests covered descriptor propagation and COUNT execution rejection but not boolean capability probing on mixed encrypted/plaintext snapshots. - **Fix:** Translate only the deterministic encryption exception to false in the probe, then stream projected key metadata from live data manifest entries before advertising support. Manifest readers close on early exit; no payload I/O or retained task list is added. - **Other paths and branches:** Actual COUNT execution remains fail-closed. Existing empty, plaintext, delete-netting and historical snapshot behavior is preserved. File-level inspection is O(live files) with bounded memory. - **Behavior change:** Unsupported file encryption returns false from the capability probe rather than true or an execution exception. ### Testing - New focused tests: nine cases, seven expected failures before the fix and zero failures/errors after it. - Iceberg connector tests with Maven build cache disabled: 1,886 tests, zero failures/errors, 10 existing skips. - FE Checkstyle: zero violations. Whitespace verification and secret scan passed. - SQL suites were not rerun for this capability-only follow-up. ### Release note Iceberg metadata-only COUNT capability checks consistently report unsupported file-level encryption. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready; local checks passed and CI will be retriggered. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Code, tests and commit description. - Human review: Pending maintainer review; full diff self-reviewed by the agent.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of #70321 at d781ffb: no new actionable findings. Review converged in round 1 after FE, BE and dedicated risk reviewers all returned NO_NEW_VALUABLE_FINDINGS, followed by independent verification and a final sweep of all 16 changed paths.
Existing review context: P2 comment 4489140690 is fixed on this head. The boolean COUNT capability probe catches only EncryptedCountException, streams live data-file key metadata and returns false for encryption. Execution still rejects independent encrypted data/delete descriptors. No independently confirmed existing P0/P1 finding remains, and no existing concern is reposted. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and correctness: code preserves each data/position/equality file's own key/AAD through FE carriers and native reader construction, including native position_deletes. Inspected tests cover exact surviving rows and descriptor independence; encrypted SQL admission remains deliberately closed.
- Scope and reuse: the focused integration reuses the immutable encryption carrier, existing optional wire fields and native Parquet constructor through a small neutral TableReader hook.
- Concurrency and locking: no new threads or locks; split/reader state owns keys and decoded filters, immutable carriers clone mutable Thrift buffers, and plaintext cache publication retains its existing contract. No new lock-order issue identified.
- Lifecycle and memory: manifest readers close on early returns; encrypted position maps use RAII and copy selected rows before destruction; equality predicates retain Block/index ownership. Nested metadata readers close before their owned range changes, and physical readers copy descriptors. No new static-initialization dependency or ownership cycle identified.
- Configuration: none added; no dynamic-update requirement introduced.
- Compatibility: no new dependency, stored format or external API change; plaintext cache paths remain. The fixture upgrades V2 snapshots before testing retained position deletes in V3, consistent with the Iceberg specification.
- Thrift/protobuf IDs: neither schema tree changes; existing optional encryption fields are reused.
- Parallel paths: eager/streaming data planning, position/equality auxiliary readers, native position_deletes and v3 rewrite-supply serialization were inspected; encrypted unsupported formats reject explicitly.
- Conditions and errors: encryption-presence, unsupported-format, aggregate and cache-bypass checks have concrete documented triggers. Missing/wrong keys or AAD cannot reuse decoded encrypted cache entries; unrelated metadata errors still propagate.
- Test coverage: inspected FE independent/malformed descriptors and COUNT capability cases; BE eight-combination, repeated-split, warm failure, abort/retry, equality-conversion, metadata-row and unsupported-format cases; Java reference fixtures and SQL eager/lazy plus COUNT settings. Successful encrypted SQL execution is outside the stated scope.
- Changed results: all 80 ordered golden blocks match fixture delete arithmetic and suite order; the matrix contains 176 expected encrypted-read rejections. Output-generation provenance is author-reported and was not independently rerun.
- Observability: existing classifications and reader/cache profiles remain; encrypted equality cache misses and hash-index memory are accounted. No new per-row logging or missing critical diagnostic identified.
- Transactions and persistence: no production EditLog, transaction, visibility/version or failover protocol changes.
- Writes and crash behavior: production changes are read planning/consumption; no new data-write atomicity requirement. Partial reader cleanup and abort/retry paths were inspected. Fixture writes are isolated test data.
- FE/BE variables: no new wire variable; both normal delete serialization and the distinct native metadata-table branch carry their own existing encryption_info. Existing eager/lazy dispatch guards remain in place.
- Performance: COUNT capability now performs O(live manifest entries) work in bounded memory; it projects key metadata without opening payload files or retaining tasks. Encrypted per-split delete rereads intentionally avoid shared decrypted caches; plaintext caching remains. No runtime performance claim is made.
- Other risks and focus: initial COUNT, cache and descriptor-lifetime risks were dismissed with concrete caller/consumer evidence. Final rechecks found no unresolved candidate or static coverage gap. No extra user focus was supplied.
Validation limits: static inspection only. No builds, tests, benchmarks, sanitizers, build-hygiene commands or final ELF compatibility checks were executed in this review. Author-reported runs were not independently verified. No repository source was modified. These execution limits do not represent an unfinished assigned static-review scope.
Review converged after 1 round(s).
TPC-H: Total hot run time: 28315 ms |
TPC-DS: Total hot run time: 153625 ms |
ClickBench: Total hot run time: 23.99 s |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
2 similar comments
|
run buildall |
|
run buildall |
### Background Iceberg data, position-delete and equality-delete files can each have a different native Parquet encryption key and AAD. The existing implementation rejects encrypted deletes during FE planning, and the native table/delete readers do not yet receive their independent descriptors. ### What problem does this PR solve? Issue Number: close apache#70316 Related PR: apache#70246 (file crypto), apache#70248 (native encrypted pages), apache#70314 (independent index/Bloom work) PR Type: Feature Problem Summary: Implement D8: preserve each Parquet delete file's own key/AAD from FE planning to the physical reader, including the native `position_deletes` metadata reader. Keep decrypted position/equality objects and converted equality keys out of shared split caches so a warm path cannot authorize a later split with another key. Reject encrypted ORC/Puffin descriptors and keep metadata-only aggregates from bypassing delete authentication. This branch is based directly on master and does not depend on apache#70314. ### Details - **Change map:** About 200 changed production lines connect immutable FE descriptors, a neutral TableReader encryption hook, delete-reader construction and cache bypass. Most changes are coverage: six BE scenarios with multiple key/file variants, FE planning tests, a Java reference fixture, SQL checks and generated golden results. Fixture documentation explains reproduction and the admission boundary. - **Scope:** Independent Parquet data/position/equality keys; mixed plaintext/encrypted files; native `position_deletes`; unsupported encrypted ORC/Puffin rejection; metadata COUNT protection. - **Design:** The generic TableReader receives a split-local descriptor through a protected hook. Iceberg supplies the data descriptor, while each delete reader uses only its own descriptor. Fresh Thrift buffers preserve carrier immutability. Encrypted decoded positions, equality filters and type-converted filters bypass shared caches; plaintext caching remains intact. No key bytes enter cache identities. - **Usage:** Existing physical-reader encryption descriptors support controlled native integrations. No configuration, dependency or wire-schema field is added. Reference fixtures reuse the existing encryption fixture service and its pinned Iceberg 1.12.0 jars. - **Default behavior:** Encrypted SQL admission remains closed pending protected key transport and cache integration. Encrypted files are reauthenticated per split, including after a successful read of the same path. Metadata aggregate rejection remains explicit. - **Compatibility:** Plaintext reads and delete semantics retain their existing behavior. The common plaintext FE carrier avoids an extra allocation. V3 coverage uses V2 snapshots upgraded to V3, preserving their existing Parquet position deletes. ### Testing Local verification of this branch: - `build.sh --be --fe -j 6` with ASAN: passed, including the GLIBC 2.17 gate and FE Checkstyle. - `run-be-ut.sh --run --filter='*Iceberg*:*FileEncryption*:*FileScanRange*:*Parquet*' -j 6`: 519 tests, 517 passed and 2 existing opt-in microbenchmarks skipped. A focused source selection was used; the full BE UT suite was not run. - BE coverage includes all eight data/position/equality encryption combinations with distinct keys, repeated splits, warmed paths followed by missing/wrong keys and wrong AAD, recovery through `abort_split`, independent converted equality filters, metadata-table row projection, unsupported encrypted formats and metadata COUNT rejection before I/O. Four initial scenarios failed on master before the implementation. - Iceberg connector module tests, with Maven build cache disabled and Checkstyle enabled: 1,877 tests, zero failures/errors, 10 existing skips. - Independent Java reference reads verify 16 V2/upgraded-V3 tables, actual encrypted/plaintext payload magic, distinct descriptor keys/AAD and exact rows after applying both delete types. - `test_iceberg_encryption_delete_reader`: passed generation and comparison modes through `run-regression-test.sh`, against this branch's FE and ASAN BE. The matrix covers eager/lazy planning and COUNT pushdown on/off: 80 golden SQL checks and 176 explicit encrypted-read rejections. Plaintext position-delete metadata remains readable when only data or equality files are encrypted. Equality deletes make these SQL COUNT cases follow normal scan/dispatch; dedicated FE/BE UTs cover early metadata-only COUNT rejection. - Existing `test_iceberg_encryption_native_reader`: passed against the same deployment. - The new SQL suite reproduced master's earlier `DELETE_READ_UNAVAILABLE` rejection before FE descriptor propagation. Expected results were generated by the regression framework; only trailing EOF whitespace was normalized. - Clang-format 16, changed-line clang-tidy, build hygiene, whitespace checks and secret scanning: passed. ### Notes for Reviewers The fixture service compiles Java at startup. Restart `encryption-fixture` from the updated checkout before running the new SQL suite; it reuses the existing services, ports and reference jars. These public synthetic file-key fixtures keep table metadata plaintext to isolate native-file planning from KMS/catalog admission. Successful encrypted SQL execution remains outside this PR. ### Release note None. This completes native encrypted-delete integration while retaining the SQL admission fence. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; internal staged integration and fixture documentation are included. - Pre-submission status: Ready. Full CI remains to run; encrypted SQL admission and shared decrypted cache support remain separate work. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, tests, reference fixture and this description. - Human review: Pending maintainer review; the agent self-reviewed the complete diff and ran the verification above.
### Background Iceberg advertises metadata-only COUNT support through a boolean capability probe. Data manifest counters prove row totals but do not indicate whether individual data files carry encryption key metadata. ### What problem does this PR solve? Issue Number: None Related PR: apache#70321 (review follow-up) PR Type: Bugfix Problem Summary: The capability probe could return true for encrypted data files and throw COUNT_READ_UNAVAILABLE for encrypted delete files. Return false for unsupported file-level encryption while preserving execution-time rejection and snapshot-scoped plaintext counts. ### Details - **Frequency:** Snapshots containing per-file encrypted data or encrypted deletes inspected for metadata-only COUNT. - **Root cause:** The probe trusted aggregate data counters and reused a delete helper that throws an execution-specific encryption exception. - **Why existing tests missed it:** Encryption tests covered descriptor propagation and COUNT execution rejection but not boolean capability probing on mixed encrypted/plaintext snapshots. - **Fix:** Translate only the deterministic encryption exception to false in the probe, then stream projected key metadata from live data manifest entries before advertising support. Manifest readers close on early exit; no payload I/O or retained task list is added. - **Other paths and branches:** Actual COUNT execution remains fail-closed. Existing empty, plaintext, delete-netting and historical snapshot behavior is preserved. File-level inspection is O(live files) with bounded memory. - **Behavior change:** Unsupported file encryption returns false from the capability probe rather than true or an execution exception. ### Testing - New focused tests: nine cases, seven expected failures before the fix and zero failures/errors after it. - Iceberg connector tests with Maven build cache disabled: 1,886 tests, zero failures/errors, 10 existing skips. - FE Checkstyle: zero violations. Whitespace verification and secret scan passed. - SQL suites were not rerun for this capability-only follow-up. ### Release note Iceberg metadata-only COUNT capability checks consistently report unsupported file-level encryption. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready; local checks passed and CI will be retriggered. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Code, tests and commit description. - Human review: Pending maintainer review; full diff self-reviewed by the agent.
d781ffb to
c03626f
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### Background The spatial branch passes its Parquet spatial-annotation policy through data and delete readers. Upstream master added independent file-encryption descriptors to those same reader constructors and a generic table-reader hook. ### What problem does this PR solve? Issue Number: None Related PR: apache#69032, apache#70321 (independent delete keys), apache#70314 (encrypted indexes) PR Type: Build/CI/Chore Problem Summary: Merge master 43963ce into the spatial branch. Preserve each data/delete file's own encryption descriptor together with the existing spatial-annotation option, and retain both protected table-reader hooks. Keep upstream encrypted-cache isolation and the spatial read behavior intact. ### Details - **Change map:** Manually reconcile three files: iceberg_reader.cpp, table_reader.cpp and table_reader.h. Other changes retain the upstream merge result. The earlier merge b6c42b5 already reconciles the connector API as major 33. - **Affected:** Two Parquet reader construction sites and the table-reader hook declarations. - **Developer and release impact:** Spatial readers retain their annotation policy while data and delete readers keep independent encryption keys. Existing encrypted-cache bypass, metadata-aggregate rejection and plaintext caching remain in place. - **License and third-party:** No additional dependency or license change beyond upstream. - **Rollback:** Revert this merge while retaining its first parent if necessary. ### Testing - Final merged candidate: 291 FE tests in 39 classes passed with zero failures/errors/skips, including the complete connector SPI, IcebergWritePlanProviderTest and PluginApiVersionWiringTest. - Final merged candidate: BE ASAN compiled and ran 895 tests in 55 suites: 893 passed, zero failed, two existing opt-in Parquet performance microbenchmarks skipped. Includes 17 executed encryption-related cases and the spatial/Iceberg/Parquet reader coverage. - BE used the standard personal clang21 wrapper, GLIBC_COMPATIBILITY=OFF and a private TMPDIR. The task-owned dependency adapter was restored. - Clang-format16, build hygiene, explicitly enabled full FE reactor CheckStyle and code whitespace checks passed. Changed-line clang-tidy validates the two implementations against the merged source; the compiler resource include path is provided by a task-private adapter. - Complete product packaging, SQL regression, full unit-test matrix and a new full PR review were not run for this merge. ### Release note None. ### Check List (For Author) - Behavior changed: No additional behavior beyond the existing spatial branch and upstream changes. - Does this need documentation: No - Pre-submission status: Partially verified: targeted merge validation passed; complete community CI remains separate. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: The three conflict resolutions and this message. - Human review: Pending author review.
Background
Iceberg data, position-delete and equality-delete files can each have a different native Parquet encryption key and AAD. The existing implementation rejects encrypted deletes during FE planning, and the native table/delete readers do not yet receive their independent descriptors.
What problem does this PR solve?
Issue Number: close #70316
Related PR: #70246 (file crypto), #70248 (native encrypted pages), #70314 (independent index/Bloom work)
PR Type: Feature
Problem Summary:
Implement D8: preserve each Parquet delete file's own key/AAD from FE planning to the physical reader, including the native
position_deletesmetadata reader. Keep decrypted position/equality objects and converted equality keys out of shared split caches so a warm path cannot authorize a later split with another key. Reject encrypted ORC/Puffin descriptors and keep metadata-only aggregates from bypassing delete authentication. This branch is based directly on master and does not depend on #70314.Details
position_deletes; unsupported encrypted ORC/Puffin rejection; metadata COUNT protection.Testing
Local verification of this branch:
build.sh --be --fe -j 6with ASAN: passed, including the GLIBC 2.17 gate and FE Checkstyle.run-be-ut.sh --run --filter='*Iceberg*:*FileEncryption*:*FileScanRange*:*Parquet*' -j 6: 519 tests, 517 passed and 2 existing opt-in microbenchmarks skipped. A focused source selection was used; the full BE UT suite was not run.abort_split, independent converted equality filters, metadata-table row projection, unsupported encrypted formats and metadata COUNT rejection before I/O. Four initial scenarios failed on master before the implementation.test_iceberg_encryption_delete_reader: passed generation and comparison modes throughrun-regression-test.sh, against this branch's FE and ASAN BE. The matrix covers eager/lazy planning and COUNT pushdown on/off: 80 golden SQL checks and 176 explicit encrypted-read rejections. Plaintext position-delete metadata remains readable when only data or equality files are encrypted. Equality deletes make these SQL COUNT cases follow normal scan/dispatch; dedicated FE/BE UTs cover early metadata-only COUNT rejection.test_iceberg_encryption_native_reader: passed against the same deployment.DELETE_READ_UNAVAILABLErejection before FE descriptor propagation. Expected results were generated by the regression framework; only trailing EOF whitespace was normalized.Notes for Reviewers
The fixture service compiles Java at startup. Restart
encryption-fixturefrom the updated checkout before running the new SQL suite; it reuses the existing services, ports and reference jars. These public synthetic file-key fixtures keep table metadata plaintext to isolate native-file planning from KMS/catalog admission. Successful encrypted SQL execution remains outside this PR. The COUNT capability probe now performs a bounded-memory scan of live data-manifest key metadata, rather than reading only manifest-list counters, because those counters cannot establish whether individual files are encrypted.Release note
None. This completes native encrypted-delete integration while retaining the SQL admission fence.
Check List (For Author)
AI Disclosure