Skip to content

[fix](iceberg) Bound encrypted metadata ownership to statements - #70675

Open
Gabriel39 wants to merge 2 commits into
apache:masterfrom
Gabriel39:fix/iceberg-encrypted-cache-lifetime
Open

Gabriel39 wants to merge 2 commits into
apache:masterfrom
Gabriel39:fix/iceberg-encrypted-cache-lifetime

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Background

Iceberg uses table encryption managers to unwrap metadata keys, and manifests can contain plaintext data-file or delete-file keys. Doris caches table owners and parsed manifests across statements; the SDK can also cache decrypted manifest bytes.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

A cached encryption manager can reuse an unwrapped key after the next statement loses KMS access. Parsed manifests, SDK byte-cache hits and shared unclassified loads can likewise reuse key-bearing metadata without validating the current read. Bound encrypted table ownership to statement leases, isolate unclassified loads and retain only plaintext manifest values. Encrypted SQL admission remains closed.

Details

  • Change map: 212 changed production lines in cache retention/scoped loading and encrypted IO views, 881 test lines, and 37 documentation lines.
  • Frequency: When encrypted metadata is warmed and then read by another statement or concurrently loaded before classification.
  • Root cause: Catalog-scoped cache ownership outlives statement-scoped key authorization. The SDK manifest byte cache is keyed by path and cannot validate the current descriptor from cached decrypted bytes.
  • Why existing tests missed it: Earlier cache tests primarily used plaintext metadata, and the encrypted fixture did not enable the SDK byte cache. Existing cancellation tests assumed unclassified results were safe to share.
  • Fix: Add a value-retention predicate covering count/weight limits and publication variants; use independent load scopes; bypass decoded encrypted results and SDK manifest byte caching; preserve active table leases until their final release. Query-local partition projections inherit retention classification so warm schema-evolved scans keep lazy file adaptation.
  • Other paths and branches: Covers data, position-delete and equality-delete descriptors, synchronous/future reads, direct/bulk publication, replacement, refresh, concurrent cancellation and catalog retirement. No protocol fields or SQL enablement change.
  • Behavior change: Cold unclassified table/manifest misses may perform independent IO before a plaintext result is published. Completed plaintext entries remain reusable. Key-bearing metadata is not retained across statements; deterministic JVM memory zeroization is not promised.

Testing

  • Full connector cache, SPI and Iceberg suites: 2,342 tests, zero failures/errors, 11 existing skips; FE Checkstyle passed. Four new evaluation-count regressions cover lazy data/delete projections with and without key metadata. Cancellation fixtures preserve manifest cardinality when SDK worker counts exceed automatic-merge thresholds.
  • ASAN native encryption/delete/scan tests: 147 passed. New coverage includes eight concurrent readers across AES key sizes, independent decryptor state, wrong-key/AAD retries and cleanup after success, cancellation and failed initialization.
  • Observed failing baselines for encrypted table retention, unclassified concurrent manifest loads, and a missing manifest descriptor bypassing the SDK byte cache; all corresponding tests now pass.
  • clang-format 16, BE header hygiene, clang-tidy and staged-diff checks passed.

Release note

Encrypted Iceberg metadata no longer remains in shared table or manifest caches across statements. Encrypted SQL read admission is unchanged.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: Yes; the connector encryption-read design note is updated in this PR.
  • Pre-submission status: Ready for cache and lifetime hardening; encrypted SQL transport and end-to-end enablement are separate work.

AI Disclosure

  • Tool: Codex (GPT-6)
  • Generated: Code, tests, documentation and this description.
  • Human review: Pending.

### Background

Iceberg uses table encryption managers to unwrap metadata keys, and manifests can contain plaintext data-file or delete-file keys. Doris caches table owners and parsed manifests across statements; the SDK can also cache decrypted manifest bytes.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

A cached encryption manager can reuse an unwrapped key after the next statement loses KMS access. Parsed manifests, SDK byte-cache hits and shared unclassified loads can likewise reuse key-bearing metadata without validating the current read. Bound encrypted table ownership to statement leases, isolate unclassified loads and retain only plaintext manifest values. Encrypted SQL admission remains closed.

### Details

- **Change map:** About 200 lines of production changes in cache retention/scoped loading and encrypted IO views, 774 lines of tests, and 37 lines of documentation.
- **Frequency:** When encrypted metadata is warmed and then read by another statement or concurrently loaded before classification.
- **Root cause:** Catalog-scoped cache ownership outlives statement-scoped key authorization. The SDK manifest byte cache is keyed by path and cannot validate the current descriptor from cached decrypted bytes.
- **Why existing tests missed it:** Earlier cache tests primarily used plaintext metadata, and the encrypted fixture did not enable the SDK byte cache. Existing cancellation tests assumed unclassified results were safe to share.
- **Fix:** Add a value-retention predicate covering count/weight limits and publication variants; use independent load scopes; bypass decoded encrypted results and SDK manifest byte caching; preserve active table leases until their final release.
- **Other paths and branches:** Covers data, position-delete and equality-delete descriptors, synchronous/future reads, direct/bulk publication, replacement, refresh, concurrent cancellation and catalog retirement. No protocol fields or SQL enablement change.
- **Behavior change:** Cold unclassified table/manifest misses may perform independent IO before a plaintext result is published. Completed plaintext entries remain reusable. Key-bearing metadata is not retained across statements; deterministic JVM memory zeroization is not promised.

### Testing

- Full connector cache, SPI and Iceberg suites: 2,178 tests, zero failures/errors, five existing skips; FE Checkstyle passed.
- ASAN native encryption/delete/scan tests: 147 passed. New coverage includes eight concurrent readers across AES key sizes, independent decryptor state, wrong-key/AAD retries and cleanup after success, cancellation and failed initialization.
- Observed failing baselines for encrypted table retention, unclassified concurrent manifest loads, and a missing manifest descriptor bypassing the SDK byte cache; all corresponding tests now pass.
- clang-format 16, BE header hygiene, clang-tidy and staged-diff checks passed.

### Release note

Encrypted Iceberg metadata no longer remains in shared table or manifest caches across statements. Encrypted SQL read admission is unchanged.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; the connector encryption-read design note is updated in this PR.
- Pre-submission status: Ready for cache and lifetime hardening; encrypted SQL transport and end-to-end enablement are separate work.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, tests, documentation and this description.
- Human review: Pending.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of all 18 changed files at 3b7847c1d3a543558f041158e66ebdb90be28b45 found one P2 performance regression: retention classification eagerly evaluates lazy partition-adaptation views on warm manifest scans. The inline finding describes the trigger and fix. No new P0/P1 findings or existing blocking comments were identified.

Critical checkpoint conclusions:

  1. Goal and proof: The inspected retention predicates, isolated load scopes and IO views support statement ownership for encrypted tables and adapter-loaded key-bearing manifests. New fixtures cover these mechanisms; the warm-scan regression remains to fix.
  2. Scope and clarity: Changes stay focused on cache ownership, IO views, tests and documentation; preserving the already-computed retention flag would avoid coupling classification to scan projection.
  3. Concurrency: Independent caller/metadata-worker loads use identity scopes; existing scope/key publication locks and atomic owner references protect publication and retirement. Loaders remain outside cache locks; no new lock-order defect found.
  4. Lifecycle: Cache, loader, statement and active-callable leases balance across discard, invalidation and close. The supported HMS path creates fresh table operations per load, with the manager owned by those operations (Iceberg 1.12 HiveTableOperations). No new static-initialization dependency found.
  5. Configuration: No configuration keys or dynamic-update semantics change.
  6. Compatibility: No persisted storage format or FE/BE protocol change; existing admission remains closed. Cache API changes are additive and internal to the connector cache module.
  7. Thrift/protobuf: No gensrc definitions or field/enum IDs change.
  8. Parallel paths: Inspected synchronous/future and direct/bulk/replacement/refresh publication, data/position/equality-delete manifests, streaming/synchronous scans, COUNT and system-table admission. Undeclared external file-key cases on raw SDK paths predate this diff and remain outside the documented table-level contract; this review does not claim universal SDK cache cleanup.
  9. Conditions: New encryption and retention branches explain concrete descriptor/key-lifetime failures; no new silent-success error path found.
  10. Tests: Inspected count/weight admission, independent loads, descriptor failure/retry, KMS revocation, cancellation, catalog retirement and native reader close coverage. Negative and concurrency cases use explicit assertions/barriers. No end-to-end encrypted SQL execution is established.
  11. Expected results: No regression result files change; new unit assertions match the inspected ownership behavior, but do not detect the eager partition-projection work.
  12. Observability: Existing cache counters, load errors and encryption classifications cover affected operations; no additional logging need identified.
  13. Transactions/persistence: No EditLog, replay, failover or transaction logic changes.
  14. Writes/atomicity: No data-write path changes; cache publication retains its existing guarded commit semantics.
  15. FE/BE variables: No new transport variables or send paths. BE changes add tests only.
  16. Performance: Cold independent IO is an explicit classification-isolation tradeoff. The extra whole-manifest adaptation/allocation pass on warm schema-evolved scans is the P2 finding.
  17. Other invariants: No additional data-correctness, version-visibility, delete-consistency, memory-ownership or exception-propagation defect attributable to this diff was established.

User focus: none supplied. Coverage limits: static inspection only; no builds, tests, benchmarks or runtime/KMS validation were executed, and author-reported test results were not independently reproduced. Encrypted SQL admission remains closed. No unresolved in-scope candidate or changed-path coverage gap remains.

Review converged after 2 round(s).

### Background

Cached manifests expose query-local partition views after schema evolution.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70675
PR Type: Bugfix

Problem Summary:

Retention classification eagerly consumed lazy partition views before streaming scans could read their first file. Preserve the source manifest's retention decision when projecting partitions.

### Details

- **Frequency:** Warm scans whose partition types require adaptation.
- **Root cause:** The projected value constructor repeated key classification through transformed lists.
- **Why existing tests missed it:** Functional adaptation checks did not measure evaluation before consumption.
- **Fix:** Carry the unchanged key-retention classification into the query-local view.
- **Other paths and branches:** Cover data and delete files with and without key metadata. Disable automatic manifest merging in cancellation fixtures so their queue cardinality is independent of processor count.

### Testing

Four regression cases failed before the fix and passed afterward. Connector cache, SPI and Iceberg suites ran 2,342 tests with zero failures/errors and 11 existing skips. FE Checkstyle passed.

### Release note

Warm Iceberg scans preserve lazy partition adaptation after schema evolution.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, tests and this description.
- Human review: Pending.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of all 19 changed files at 7c04a372b900997dd71864031d25ece1bb65d982 found no new substantiated defect. The existing P2 comment 4597416579 is fixed: projected manifest values carry the source retention decision without traversing lazy partition views. No existing P0/P1 inline blocker remains.

Critical checkpoint conclusions:

  1. Goal and proof: Retention predicates, isolated load scopes and uncached IO views support statement ownership of encrypted table owners and adapter-loaded key-bearing manifests. Focused unit fixtures cover the changed mechanisms; this review inspected them statically.
  2. Scope and abstraction: Changes stay focused on shared cache policy, Iceberg ownership, tests and documentation, reusing existing cache and IO abstractions.
  3. Concurrency: Caller/metadata-worker loads use identity scopes. Existing scope/key publication guards and atomic references protect publication and retirement; loaders stay outside publication/prefetch locks. No new lock-order defect found.
  4. Lifecycle: Loader bridge, cache, statement and active-callable leases preserve table owners across declined admission, replacement, cancellation, invalidation and catalog retirement. Construction failures clean up. No new production static-initialization dependency found.
  5. Configuration: No configuration keys or dynamic-update semantics change.
  6. Compatibility: No persisted storage format, production native ABI or public connector SPI change. Cache APIs are additive; encrypted SQL admission remains closed.
  7. Thrift/protobuf: No definitions, field IDs or enum values change.
  8. Parallel paths: Checked synchronous/future, direct/bulk/CAS/refresh, count/weight, data/position/equality-delete, snapshot/SDK and COUNT/system-table paths. Unchanged raw SDK probes for undeclared file keys remain a pre-existing boundary, described below.
  9. Conditions and errors: Comments explain key-lifetime, descriptor-validation and lazy-projection conditions. No new silent-success error path established; read failures continue to propagate.
  10. Tests: Inspected negative descriptor/key/AAD/KMS, failure/retry, concurrent load, cancellation, retirement, weighted retention and lazy-projection coverage. Native additions exercise independent readers and close after success/cancellation/failed initialization. Encrypted SQL end-to-end execution is not established.
  11. Expected results: No regression output files change; new unit assertions match the inspected control flow. The four lazy data/delete projection cases address the prior P2.
  12. Observability: Existing cache hit/miss/failure counters, path-bearing read failures and encryption classifications remain available. No additional logging requirement identified.
  13. Transactions/persistence: No EditLog, replay, failover or transaction implementation changes.
  14. Writes/atomicity: No production data-write path changes; guarded cache publication semantics remain intact.
  15. FE/BE variables: No new transport variables or send paths. C++ changes add tests only.
  16. Performance: Independent simultaneous cold IO is an explicit isolation tradeoff until classification. Completed plaintext cache hits remain reusable, and schema-evolved partition projections remain lazy. No additional performance defect substantiated.
  17. Other invariants: No new data-correctness, visibility, delete-consistency, memory-ownership or exception-propagation defect established. Native tests add no hub/PCH/template/unity change.

User focus: no additional focus supplied. Existing review context was refreshed and deduplicated against this head.

Coverage limits: static inspection only; no builds, tests, benchmarks or runtime/KMS validation ran, and author-reported test results were not independently reproduced. The unchanged canProveCountFromManifests raw SDK COUNT probe can still populate SDK bytes for otherwise plaintext tables with undeclared per-file key metadata; it is present at the merge base and has no new trigger in this diff. This review does not claim universal SDK cache cleanup, deterministic JVM key zeroization, or encrypted SQL enablement. No unresolved in-scope candidate or changed-file coverage gap remains.

Convergence: all three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS in round 1; main verification and the final sweep are complete.

Review converged after 1 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant