Skip to content

[feature](iceberg) Negotiate and validate native encrypted reads - #70697

Draft
Gabriel39 wants to merge 9 commits into
apache:masterfrom
Gabriel39:dev/iceberg-pme-native-read-integration
Draft

Gabriel39 wants to merge 9 commits into
apache:masterfrom
Gabriel39:dev/iceberg-pme-native-read-integration

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

Background

Native Parquet data/delete readers and binary per-file key descriptors exist, but encrypted SQL must stay closed until both private RPC channels and the actual selected readers can prove support.

What problem does this PR solve?

Issue Number: None
Related PR: #70675, #70693
PR Type: Enhancement

Add default-off HMS v3 native encrypted reads with fresh backend capability leases, dispatch/fetch checks and end-to-end SQL coverage. The existing cache/lifetime and private transport changes remain independently reviewed in the related PRs. Until they merge, this draft also contains those prerequisite commits; the integration change starts at 55be6efea33e.

Details

  • Optional heartbeat and split-request capabilities default to unavailable for old nodes. Capability proofs expire, clear on failed/missing heartbeats and never survive FE image replay. Both selected private TLS providers and the live split listener must prove support.
  • Key-bearing scans use a distinct native format, including shared parameters and plaintext files with encrypted deletes. Assigned readers are checked before serialization and again before send; lazy responses check the actual caller and keep refusals terminal. A capability refusal does not blacklist a healthy backend.
  • Declared encrypted COUNT uses physical readers. Undeclared per-file keys in raw COUNT probes fail explicitly, and SDK byte-cache hits cannot bypass current descriptor/KMS validation. Plaintext metadata COUNT remains available.
  • Independent data/position/equality-delete keys, binary AAD, snapshots, native position-delete metadata, pruning and batch settings have SQL coverage. Encrypted JNI metadata, unsupported formats, custom transports without proof and encrypted writes remain outside admission. V3 planning remains synchronous.
  • Connector plugin API advances to 34.0; 33.0 is reserved by [feature](iceberg) Add REST encrypted metadata read support #70343. Protocol IDs were checked against master and branches 4.2, 4.1, 4.0 and 3.1. FE and its connector plugins must be deployed together.

Testing

  • Complete RELEASE FE/BE build; FE Checkstyle reports zero violations. Final ASAN native subset: 181 tests in 13 suites. FE capability/heartbeat/dispatch/split/provider coverage: 125 tests. Connector cache/SPI/Iceberg suites: 2,349 tests, zero failures/errors and 11 existing skips.
  • Two enabled SQL matrix runs cover all eight key modes, table/page cache settings, COUNT/pruning/batch settings, SDK/Spark reference snapshots, joins and deletes. Both default-off admission suites pass.
  • Same-endpoint mock KMS revocation fails warmed SELECT and COUNT, then restoration recovers. Physical object-read failures, 40 concurrent reads, 10 cancellations, five timeouts and five query-memory refusals preserve subsequent reads. Mock failures validate protocol/error handling, not production AWS IAM.
  • Real authenticated fragment and split RPCs, binary descriptor round trips, trust/identity rejection and cache reconnect/recovery pass. Old/new FE/BE combinations preserve plaintext and reject unsupported encrypted execution.
  • Live-object/resource checks and exact fixture-key diagnostic audits are recorded with their limits in the validation note. No deterministic JVM zeroization, arbitrary transformed-key absence or production rolling-upgrade certification is claimed.
  • Matching RELEASE cost measurements use the same synthetic million-row fixture, cache definitions, fixed affinity and a predeclared 5% plaintext median limit. All six fixed-preconditioning conditions satisfy the unchanged numerical gate; earlier failing protocols, CPU/RSS, object access, pruning and KMS cost remain visible in fe/fe-connector/docs/iceberg-encryption-validation.md. Shared-host results require performance review.

Release note

Opt-in native Iceberg encrypted reads require HMS v3, the SDK standard encryption manager, supported Parquet encryption, authenticated private RPC and fresh capable nodes. The deployment switch remains false by default.

Check List (For Author)

  • Behavior changed: Yes; see Details.
  • Documentation: Read/wire contracts and reproducible fixture guidance are included.
  • Readiness: Draft pending prerequisite merges, performance evidence review and maintainer/security review.

AI Disclosure

  • Tool: Codex (GPT-6).
  • Generated: Implementation, tests, documentation and this description.
  • Human review: Pending.

### Background

Iceberg uses table encryption managers to unwrap metadata keys, and manifests can contain plaintext data-file or delete-file keys. Doris caches table owners and parsed manifests across statements; the SDK can also cache decrypted manifest bytes.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

A cached encryption manager can reuse an unwrapped key after the next statement loses KMS access. Parsed manifests, SDK byte-cache hits and shared unclassified loads can likewise reuse key-bearing metadata without validating the current read. Bound encrypted table ownership to statement leases, isolate unclassified loads and retain only plaintext manifest values. Encrypted SQL admission remains closed.

### Details

- **Change map:** About 200 lines of production changes in cache retention/scoped loading and encrypted IO views, 774 lines of tests, and 37 lines of documentation.
- **Frequency:** When encrypted metadata is warmed and then read by another statement or concurrently loaded before classification.
- **Root cause:** Catalog-scoped cache ownership outlives statement-scoped key authorization. The SDK manifest byte cache is keyed by path and cannot validate the current descriptor from cached decrypted bytes.
- **Why existing tests missed it:** Earlier cache tests primarily used plaintext metadata, and the encrypted fixture did not enable the SDK byte cache. Existing cancellation tests assumed unclassified results were safe to share.
- **Fix:** Add a value-retention predicate covering count/weight limits and publication variants; use independent load scopes; bypass decoded encrypted results and SDK manifest byte caching; preserve active table leases until their final release.
- **Other paths and branches:** Covers data, position-delete and equality-delete descriptors, synchronous/future reads, direct/bulk publication, replacement, refresh, concurrent cancellation and catalog retirement. No protocol fields or SQL enablement change.
- **Behavior change:** Cold unclassified table/manifest misses may perform independent IO before a plaintext result is published. Completed plaintext entries remain reusable. Key-bearing metadata is not retained across statements; deterministic JVM memory zeroization is not promised.

### Testing

- Full connector cache, SPI and Iceberg suites: 2,178 tests, zero failures/errors, five existing skips; FE Checkstyle passed.
- ASAN native encryption/delete/scan tests: 147 passed. New coverage includes eight concurrent readers across AES key sizes, independent decryptor state, wrong-key/AAD retries and cleanup after success, cancellation and failed initialization.
- Observed failing baselines for encrypted table retention, unclassified concurrent manifest loads, and a missing manifest descriptor bypassing the SDK byte cache; all corresponding tests now pass.
- clang-format 16, BE header hygiene, clang-tidy and staged-diff checks passed.

### Release note

Encrypted Iceberg metadata no longer remains in shared table or manifest caches across statements. Encrypted SQL read admission is unchanged.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; the connector encryption-read design note is updated in this PR.
- Pre-submission status: Ready for cache and lifetime hardening; encrypted SQL transport and end-to-end enablement are separate work.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, tests, documentation and this description.
- Human review: Pending.
### Background

FE sends query fragments to BE using gRPC/BRPC. BE retrieves lazy external scan splits from FE over Thrift. OSS implementations expose TLS provider interfaces, while their private clients use plaintext and their servers reject TLS activation.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70675 (separate encrypted table/cache ownership hardening)
PR Type: Enhancement

Problem Summary:

OSS deployments need authenticated, encrypted internal RPC before transporting private file-key descriptors. Implement mutual TLS for fragment channels and blocking Thrift clients/listeners, using the existing TLS configuration and explicit protocol exclusions. Preserve the original certified hostname after DNS resolution, keep reconnects encrypted, and bound authentication IO independently of normal RPC read timeouts. Connection and server lifecycle tests also cover failed initialization and repeated cleanup.

### Details

- **Change map:** 930 changed production lines, 802 test/fixture lines, 97 documentation lines, and no generated files. Native TLS setup and bounded sockets live in `private_rpc_tls`; Java contexts and Thrift transports live in `tls`; OSS providers select them, and the shared Thrift server accepts a provider listener. Tests use ephemeral OpenSSL identities and real listeners.
- **Before and after:** Included private protocols now require a trusted peer certificate and encrypt their traffic. Authentication failures propagate to callers. Java and native Thrift reconnects retain TLS, and a peer that stalls before authentication cannot inherit an unlimited RPC idle timeout. Cleanup handles servers whose TLS validation failed before start, as well as a server already joined during explicit shutdown.
- **Configs and defaults:** `enable_tls=false` stays the default. Private TLS requires `verify_fail_if_no_peer_cert`, certificate/key/CA files, static refresh interval zero, and a positive Thrift connect timeout. Unsupported SAN allowlists and BE private-key passwords are rejected. See `docs/private-rpc-tls.md` for public-protocol exclusions, certificate rotation and dedicated-cluster-CA requirements. Native BRPC clients authenticate the configured CA trust domain; FE gRPC and Thrift clients additionally verify the target hostname/IP.
- **Impact on existing users:** Default plaintext deployments keep their transport behavior. Operators can opt into OSS private RPC mutual TLS with coordinated node configuration. Existing encrypted Iceberg SQL capability/admission checks remain in place.

### Testing

- Local repository `run-fe-ut.sh`: 11 tests passed, including real gRPC/Thrift authentication, hostname and CA rejection, plaintext control, reconnects in both transport modes, and stalled-handshake cleanup.
- Local repository `run-be-ut.sh`: 158 ASAN tests in 11 suites passed, including 11 TLS tests and native encrypted Parquet/data/delete/cancellation cases. New lifecycle tests reproduced null cleanup and repeated-join failures before the fixes; stalled-peer tests failed before bounded authentication was added.
- Full ASAN `build.sh --be --fe` passed with FE Checkstyle; clang-format 16, header hygiene and clang-tidy passed on the affected C++ files. The shipped BE ELF has no GLIBC requirement newer than 2.17.
- An isolated local FE/BE cluster completed authenticated heartbeats, a BE fragment query returning count 100 and sum 4950, an HMS data scan returning IDs 1–4, and a lazy V2 delete scan returning 1,3,4,6,8. The FE `fetchSplitBatch` RPC counter confirmed BE-to-FE split fetches.
- All four live private RPC endpoints accepted trusted identities and rejected missing/untrusted identities, wrong CA, wrong hostname and plaintext. Trusted connections succeeded again after rejection checks.
- Independent SDK 1.12.0 and Spark reference reads agreed on historical snapshots with real encrypted manifests, manifest lists and file keys. Default encrypted SQL still rejected admission with `READ_DISABLED`.

### Release note

OSS deployments can enable mutual TLS for internal gRPC/BRPC and blocking Thrift using the existing TLS settings. Private RPC mutual authentication, static certificate configuration and explicit exclusions for unsupported public protocols are required. The default remains disabled.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; operator and trust-boundary notes are included in `docs/private-rpc-tls.md`.
- Pre-submission status: Ready for transport review; encrypted SQL capability negotiation and broader encrypted-read acceptance are separate work.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Implementation, tests, fixture script, documentation and this description.
- Human review: Pending maintainer review; agent self-review and local verification completed.
### Background

Cached manifests expose query-local partition views after schema evolution.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70675
PR Type: Bugfix

Problem Summary:

Retention classification eagerly consumed lazy partition views before streaming scans could read their first file. Preserve the source manifest's retention decision when projecting partitions.

### Details

- **Frequency:** Warm scans whose partition types require adaptation.
- **Root cause:** The projected value constructor repeated key classification through transformed lists.
- **Why existing tests missed it:** Functional adaptation checks did not measure evaluation before consumption.
- **Fix:** Carry the unchanged key-retention classification into the query-local view.
- **Other paths and branches:** Cover data and delete files with and without key metadata. Disable automatic manifest merging in cancellation fixtures so their queue cardinality is independent of processor count.

### Testing

Four regression cases failed before the fix and passed afterward. Connector cache, SPI and Iceberg suites ran 2,342 tests with zero failures/errors and 11 existing skips. FE Checkstyle passed.

### Release note

Warm Iceberg scans preserve lazy partition adaptation after schema evolution.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, tests and this description.
- Human review: Pending.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two P1 findings and one P2 finding are reported inline. They cover Java heartbeat certificate identity, unintended plaintext writes into encrypted tables, and KMS authorization bypass through explicitly enabled external SQL result caching.

Reviewed the complete authoritative 95-path diff at e9bdafb, including the cache and transport prerequisite commits. Review converged after two rounds: both normal reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS in round 2; every candidate and risk focus has an accepted, duplicate or evidence-based dismissed disposition. No existing inline findings were present in the final live refresh. No additional user focus was supplied.

Critical checkpoint conclusions:

  • Goal and proof: opt-in HMS/v3 native encrypted reads have admission, independent file/delete keys and protected dispatch. The intended read-only boundary fails at the write target (P1). Tests were inspected, not executed.
  • Scope and reuse: this is a broad feature plus disclosed prerequisite cache/TLS work; shared SPI, cache retention predicates and provider boundaries are reused. Write eligibility needs a separate invariant.
  • Concurrency: heartbeat updates race planning/dispatch through a volatile immutable capability lease; assigned targets are checked before serialization and sending. Load scopes isolate unclassified concurrent cache misses. No additional lock inversion or heavy new IO under metadata locks was established.
  • Lifecycle and initialization: cache/loader/statement references bridge declined publication, eviction and catalog retirement; frozen read views are separate from writable owners. TLS socket/factory ownership, failed initialization, close and reopen were traced. No new cross-translation-unit initialization dependency was identified.
  • Configuration: existing deployment read/TLS settings remain default off. Static TLS identities require coordinated restart; positive mutable handshake timeouts are checked when used. No new dynamic configuration mechanism was added.
  • Compatibility: absent capabilities default unavailable; distinct PME markers reach shared parameters and ranges. Custom providers default to no capability. Connector API 34.0 and its recorded surface/test pin agree; FE and plugins require coordinated deployment.
  • Thrift definitions: all touched definitions were read on HEAD and live-matched master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. New fields are optional; no new ID/enum collision was found. Split field 3 remains reserved for branch-4.2.
  • Parallel paths: initial/lazy and typed/serialized dispatch, simplified fragments, data/position/equality deletes, native position metadata, COUNT and row fetch were traced. Unsupported encrypted ORC/Puffin/JNI alternatives reject explicitly. FQDN heartbeat and SQL result-cache counterparts have the inline findings.
  • Conditions and failures: descriptor validation, capability refusal, terminal lazy errors and transport initialization fail explicitly. No additional ignored new Status or silent fallback was identified.
  • Test coverage: native, connector, FE transport/capability and SQL matrices cover many positive and negative cases. Resolved Java heartbeat identity, encrypted writes and opted-in result-cache revocation are material missing cases corresponding to the findings.
  • Expected results: changed default-off output removals match the new READ_DISABLED queries; delete rows, counts, sums and position offsets match fixture construction. Output-generation provenance was not independently executed.
  • Observability: stable classified admission/transport errors and existing RPC/profile counters remain available; provider diagnostics avoid formatting private inputs. No separate observability defect was established.
  • Persistence and failover: capability is transient in images; failed/missing/stale heartbeats revoke it, and positive heartbeat journal updates reach followers. Historical replay cannot grant a fresh process lease. No committed-data/version or MoW change is introduced.
  • Writes and atomicity: existing write transaction/commit machinery is unchanged but becomes newly reachable for encrypted targets; its ordinary plaintext output is the P1 write finding. No claim of encrypted-write support is justified by read capability.
  • FE/BE propagation: optional heartbeat and actual split-caller versions, independent data/delete descriptors and PME markers were checked across producer and consumer paths.
  • Performance and memory: repeated per-file backend inventory is avoided; warm plaintext cache reuse remains, while unknown cold loads are intentionally isolated. No new large untracked native query allocation was established. The author's benchmarks use a prerequisite-containing baseline and shared host and show higher FE RSS; they do not independently prove full-PR performance here.
  • Other boundaries: complete OSS cloud-control-plane TLS activation and plaintext/TLS rolling activation are documented limits. This review does not certify runtime interoperability, production IAM, deterministic zeroization or final ELF glibc compatibility.

Validation was static only: no builds, tests, benchmarks, cluster execution, source edits or submodule initialization. No unresolved static coverage gaps remain.

Review converged after 2 round(s).

return nonBlocking
? new TFramedTransport(new TSocket(address.hostname, address.port, timeoutMs))
TSocket socket = PrivateRpcTls.enabled(Protocol.THRIFT)
? new TlsThriftSocket(address.hostname, address.port, timeoutMs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve the registered DNS identity for TLS heartbeats

With enable_fqdn_mode=true, HeartbeatMgr.pingOnce resolves backend.getHost() through DnsCache and passes the numeric IP to backendHeartbeatPool. GenericPool forwards that address here, so the new TlsThriftSocket uses the IP for HTTPS certificate verification in PrivateRpcTls.openThriftSocket. A trusted backend certificate containing only its correctly registered DNS SAN therefore fails every heartbeat, eventually marking a healthy backend dead. The native client cache and gRPC authority handling preserve the original hostname, but this Java heartbeat path does not. Carry the certified hostname separately from the resolved connect/pool address, including on reopen, while retaining DNS-address turnover.

if (manager == null) {
manager = table.encryption();
}
if (!(manager instanceof StandardEncryptionManager)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep an independent fence on encrypted write targets

This check also runs from IcebergConnectorMetadata.getTableSchema, so allowing it to return removes the old schema-binding barrier for writes. IcebergWritePlanProvider.planWrite and IcebergConnectorTransaction.beginWrite have no encryption rejection, including after transaction refresh. Their native sink still creates ordinary Parquet files, and IcebergWriterHelper.genDataFile commits them without keyMetadata. Consequently, with enable_encryption_read=true and capable nodes, a simple INSERT INTO an encryption.key-id HMS v3 table can commit plaintext data files; SDK manifest encryption does not encrypt those physical objects. Reject encrypted write targets independently of read admission, before sink dispatch and against refreshed metadata. The read switch must not open this unsupported write path.

this.cleanup = cleanup;
// SDK managers cache unwrapped keys. Even an unweighted cache must release this table
// with its statement, so a later statement performs its own catalog/KMS authorization.
this.cacheable = !IcebergEncryptionRead.isEncrypted(table);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fence encrypted SQL result-cache replay as well

Declining this owner does not ensure current KMS authorization when enable_hive_sql_cache=true and enable_sql_cache=true. Once an encrypted snapshot passes the SQL-cache quiet window, warm SELECT COUNT(*) (or SUM), revoke KMS Decrypt at the same endpoint without changing metadata, and repeat the SQL. IcebergConnectorMetadata.getSqlCacheVersion returns the same UUID/snapshot/schema/metadata-location token, so NereidsSqlCacheManager returns cached rows without opening a manifest or invoking KMS. Reconstructing the SDK manager does not help because key unwrap is lazy. Exclude encrypted tables from result-cache admission and replay, or reauthorize hits; an empty connector token alone can fall back to legacy freshness caching. Add this explicitly enabled cache mode to the revocation regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant