Repository navigation
Conversation
### Background Iceberg uses table encryption managers to unwrap metadata keys, and manifests can contain plaintext data-file or delete-file keys. Doris caches table owners and parsed manifests across statements; the SDK can also cache decrypted manifest bytes. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: A cached encryption manager can reuse an unwrapped key after the next statement loses KMS access. Parsed manifests, SDK byte-cache hits and shared unclassified loads can likewise reuse key-bearing metadata without validating the current read. Bound encrypted table ownership to statement leases, isolate unclassified loads and retain only plaintext manifest values. Encrypted SQL admission remains closed. ### Details - **Change map:** About 200 lines of production changes in cache retention/scoped loading and encrypted IO views, 774 lines of tests, and 37 lines of documentation. - **Frequency:** When encrypted metadata is warmed and then read by another statement or concurrently loaded before classification. - **Root cause:** Catalog-scoped cache ownership outlives statement-scoped key authorization. The SDK manifest byte cache is keyed by path and cannot validate the current descriptor from cached decrypted bytes. - **Why existing tests missed it:** Earlier cache tests primarily used plaintext metadata, and the encrypted fixture did not enable the SDK byte cache. Existing cancellation tests assumed unclassified results were safe to share. - **Fix:** Add a value-retention predicate covering count/weight limits and publication variants; use independent load scopes; bypass decoded encrypted results and SDK manifest byte caching; preserve active table leases until their final release. - **Other paths and branches:** Covers data, position-delete and equality-delete descriptors, synchronous/future reads, direct/bulk publication, replacement, refresh, concurrent cancellation and catalog retirement. No protocol fields or SQL enablement change. - **Behavior change:** Cold unclassified table/manifest misses may perform independent IO before a plaintext result is published. Completed plaintext entries remain reusable. Key-bearing metadata is not retained across statements; deterministic JVM memory zeroization is not promised. ### Testing - Full connector cache, SPI and Iceberg suites: 2,178 tests, zero failures/errors, five existing skips; FE Checkstyle passed. - ASAN native encryption/delete/scan tests: 147 passed. New coverage includes eight concurrent readers across AES key sizes, independent decryptor state, wrong-key/AAD retries and cleanup after success, cancellation and failed initialization. - Observed failing baselines for encrypted table retention, unclassified concurrent manifest loads, and a missing manifest descriptor bypassing the SDK byte cache; all corresponding tests now pass. - clang-format 16, BE header hygiene, clang-tidy and staged-diff checks passed. ### Release note Encrypted Iceberg metadata no longer remains in shared table or manifest caches across statements. Encrypted SQL read admission is unchanged. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the connector encryption-read design note is updated in this PR. - Pre-submission status: Ready for cache and lifetime hardening; encrypted SQL transport and end-to-end enablement are separate work. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests, documentation and this description. - Human review: Pending.
### Background FE sends query fragments to BE using gRPC/BRPC. BE retrieves lazy external scan splits from FE over Thrift. OSS implementations expose TLS provider interfaces, while their private clients use plaintext and their servers reject TLS activation. ### What problem does this PR solve? Issue Number: None Related PR: apache#70675 (separate encrypted table/cache ownership hardening) PR Type: Enhancement Problem Summary: OSS deployments need authenticated, encrypted internal RPC before transporting private file-key descriptors. Implement mutual TLS for fragment channels and blocking Thrift clients/listeners, using the existing TLS configuration and explicit protocol exclusions. Preserve the original certified hostname after DNS resolution, keep reconnects encrypted, and bound authentication IO independently of normal RPC read timeouts. Connection and server lifecycle tests also cover failed initialization and repeated cleanup. ### Details - **Change map:** 930 changed production lines, 802 test/fixture lines, 97 documentation lines, and no generated files. Native TLS setup and bounded sockets live in `private_rpc_tls`; Java contexts and Thrift transports live in `tls`; OSS providers select them, and the shared Thrift server accepts a provider listener. Tests use ephemeral OpenSSL identities and real listeners. - **Before and after:** Included private protocols now require a trusted peer certificate and encrypt their traffic. Authentication failures propagate to callers. Java and native Thrift reconnects retain TLS, and a peer that stalls before authentication cannot inherit an unlimited RPC idle timeout. Cleanup handles servers whose TLS validation failed before start, as well as a server already joined during explicit shutdown. - **Configs and defaults:** `enable_tls=false` stays the default. Private TLS requires `verify_fail_if_no_peer_cert`, certificate/key/CA files, static refresh interval zero, and a positive Thrift connect timeout. Unsupported SAN allowlists and BE private-key passwords are rejected. See `docs/private-rpc-tls.md` for public-protocol exclusions, certificate rotation and dedicated-cluster-CA requirements. Native BRPC clients authenticate the configured CA trust domain; FE gRPC and Thrift clients additionally verify the target hostname/IP. - **Impact on existing users:** Default plaintext deployments keep their transport behavior. Operators can opt into OSS private RPC mutual TLS with coordinated node configuration. Existing encrypted Iceberg SQL capability/admission checks remain in place. ### Testing - Local repository `run-fe-ut.sh`: 11 tests passed, including real gRPC/Thrift authentication, hostname and CA rejection, plaintext control, reconnects in both transport modes, and stalled-handshake cleanup. - Local repository `run-be-ut.sh`: 158 ASAN tests in 11 suites passed, including 11 TLS tests and native encrypted Parquet/data/delete/cancellation cases. New lifecycle tests reproduced null cleanup and repeated-join failures before the fixes; stalled-peer tests failed before bounded authentication was added. - Full ASAN `build.sh --be --fe` passed with FE Checkstyle; clang-format 16, header hygiene and clang-tidy passed on the affected C++ files. The shipped BE ELF has no GLIBC requirement newer than 2.17. - An isolated local FE/BE cluster completed authenticated heartbeats, a BE fragment query returning count 100 and sum 4950, an HMS data scan returning IDs 1–4, and a lazy V2 delete scan returning 1,3,4,6,8. The FE `fetchSplitBatch` RPC counter confirmed BE-to-FE split fetches. - All four live private RPC endpoints accepted trusted identities and rejected missing/untrusted identities, wrong CA, wrong hostname and plaintext. Trusted connections succeeded again after rejection checks. - Independent SDK 1.12.0 and Spark reference reads agreed on historical snapshots with real encrypted manifests, manifest lists and file keys. Default encrypted SQL still rejected admission with `READ_DISABLED`. ### Release note OSS deployments can enable mutual TLS for internal gRPC/BRPC and blocking Thrift using the existing TLS settings. Private RPC mutual authentication, static certificate configuration and explicit exclusions for unsupported public protocols are required. The default remains disabled. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; operator and trust-boundary notes are included in `docs/private-rpc-tls.md`. - Pre-submission status: Ready for transport review; encrypted SQL capability negotiation and broader encrypted-read acceptance are separate work. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, fixture script, documentation and this description. - Human review: Pending maintainer review; agent self-review and local verification completed.
### Background Cached manifests expose query-local partition views after schema evolution. ### What problem does this PR solve? Issue Number: None Related PR: apache#70675 PR Type: Bugfix Problem Summary: Retention classification eagerly consumed lazy partition views before streaming scans could read their first file. Preserve the source manifest's retention decision when projecting partitions. ### Details - **Frequency:** Warm scans whose partition types require adaptation. - **Root cause:** The projected value constructor repeated key classification through transformed lists. - **Why existing tests missed it:** Functional adaptation checks did not measure evaluation before consumption. - **Fix:** Carry the unchanged key-retention classification into the query-local view. - **Other paths and branches:** Cover data and delete files with and without key metadata. Disable automatic manifest merging in cancellation fixtures so their queue cardinality is independent of processor count. ### Testing Four regression cases failed before the fix and passed afterward. Connector cache, SPI and Iceberg suites ran 2,342 tests with zero failures/errors and 11 existing skips. FE Checkstyle passed. ### Release note Warm Iceberg scans preserve lazy partition adaptation after schema evolution. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this description. - Human review: Pending.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes: two P1 findings and one P2 finding are reported inline. They cover Java heartbeat certificate identity, unintended plaintext writes into encrypted tables, and KMS authorization bypass through explicitly enabled external SQL result caching.
Reviewed the complete authoritative 95-path diff at e9bdafb, including the cache and transport prerequisite commits. Review converged after two rounds: both normal reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS in round 2; every candidate and risk focus has an accepted, duplicate or evidence-based dismissed disposition. No existing inline findings were present in the final live refresh. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: opt-in HMS/v3 native encrypted reads have admission, independent file/delete keys and protected dispatch. The intended read-only boundary fails at the write target (P1). Tests were inspected, not executed.
- Scope and reuse: this is a broad feature plus disclosed prerequisite cache/TLS work; shared SPI, cache retention predicates and provider boundaries are reused. Write eligibility needs a separate invariant.
- Concurrency: heartbeat updates race planning/dispatch through a volatile immutable capability lease; assigned targets are checked before serialization and sending. Load scopes isolate unclassified concurrent cache misses. No additional lock inversion or heavy new IO under metadata locks was established.
- Lifecycle and initialization: cache/loader/statement references bridge declined publication, eviction and catalog retirement; frozen read views are separate from writable owners. TLS socket/factory ownership, failed initialization, close and reopen were traced. No new cross-translation-unit initialization dependency was identified.
- Configuration: existing deployment read/TLS settings remain default off. Static TLS identities require coordinated restart; positive mutable handshake timeouts are checked when used. No new dynamic configuration mechanism was added.
- Compatibility: absent capabilities default unavailable; distinct PME markers reach shared parameters and ranges. Custom providers default to no capability. Connector API 34.0 and its recorded surface/test pin agree; FE and plugins require coordinated deployment.
- Thrift definitions: all touched definitions were read on HEAD and live-matched master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. New fields are optional; no new ID/enum collision was found. Split field 3 remains reserved for branch-4.2.
- Parallel paths: initial/lazy and typed/serialized dispatch, simplified fragments, data/position/equality deletes, native position metadata, COUNT and row fetch were traced. Unsupported encrypted ORC/Puffin/JNI alternatives reject explicitly. FQDN heartbeat and SQL result-cache counterparts have the inline findings.
- Conditions and failures: descriptor validation, capability refusal, terminal lazy errors and transport initialization fail explicitly. No additional ignored new Status or silent fallback was identified.
- Test coverage: native, connector, FE transport/capability and SQL matrices cover many positive and negative cases. Resolved Java heartbeat identity, encrypted writes and opted-in result-cache revocation are material missing cases corresponding to the findings.
- Expected results: changed default-off output removals match the new READ_DISABLED queries; delete rows, counts, sums and position offsets match fixture construction. Output-generation provenance was not independently executed.
- Observability: stable classified admission/transport errors and existing RPC/profile counters remain available; provider diagnostics avoid formatting private inputs. No separate observability defect was established.
- Persistence and failover: capability is transient in images; failed/missing/stale heartbeats revoke it, and positive heartbeat journal updates reach followers. Historical replay cannot grant a fresh process lease. No committed-data/version or MoW change is introduced.
- Writes and atomicity: existing write transaction/commit machinery is unchanged but becomes newly reachable for encrypted targets; its ordinary plaintext output is the P1 write finding. No claim of encrypted-write support is justified by read capability.
- FE/BE propagation: optional heartbeat and actual split-caller versions, independent data/delete descriptors and PME markers were checked across producer and consumer paths.
- Performance and memory: repeated per-file backend inventory is avoided; warm plaintext cache reuse remains, while unknown cold loads are intentionally isolated. No new large untracked native query allocation was established. The author's benchmarks use a prerequisite-containing baseline and shared host and show higher FE RSS; they do not independently prove full-PR performance here.
- Other boundaries: complete OSS cloud-control-plane TLS activation and plaintext/TLS rolling activation are documented limits. This review does not certify runtime interoperability, production IAM, deterministic zeroization or final ELF glibc compatibility.
Validation was static only: no builds, tests, benchmarks, cluster execution, source edits or submodule initialization. No unresolved static coverage gaps remain.
Review converged after 2 round(s).
| return nonBlocking | ||
| ? new TFramedTransport(new TSocket(address.hostname, address.port, timeoutMs)) | ||
| TSocket socket = PrivateRpcTls.enabled(Protocol.THRIFT) | ||
| ? new TlsThriftSocket(address.hostname, address.port, timeoutMs) |
There was a problem hiding this comment.
[P1] Preserve the registered DNS identity for TLS heartbeats
With enable_fqdn_mode=true, HeartbeatMgr.pingOnce resolves backend.getHost() through DnsCache and passes the numeric IP to backendHeartbeatPool. GenericPool forwards that address here, so the new TlsThriftSocket uses the IP for HTTPS certificate verification in PrivateRpcTls.openThriftSocket. A trusted backend certificate containing only its correctly registered DNS SAN therefore fails every heartbeat, eventually marking a healthy backend dead. The native client cache and gRPC authority handling preserve the original hostname, but this Java heartbeat path does not. Carry the certified hostname separately from the resolved connect/pool address, including on reopen, while retaining DNS-address turnover.
| if (manager == null) { | ||
| manager = table.encryption(); | ||
| } | ||
| if (!(manager instanceof StandardEncryptionManager)) { |
There was a problem hiding this comment.
[P1] Keep an independent fence on encrypted write targets
This check also runs from IcebergConnectorMetadata.getTableSchema, so allowing it to return removes the old schema-binding barrier for writes. IcebergWritePlanProvider.planWrite and IcebergConnectorTransaction.beginWrite have no encryption rejection, including after transaction refresh. Their native sink still creates ordinary Parquet files, and IcebergWriterHelper.genDataFile commits them without keyMetadata. Consequently, with enable_encryption_read=true and capable nodes, a simple INSERT INTO an encryption.key-id HMS v3 table can commit plaintext data files; SDK manifest encryption does not encrypt those physical objects. Reject encrypted write targets independently of read admission, before sink dispatch and against refreshed metadata. The read switch must not open this unsupported write path.
| this.cleanup = cleanup; | ||
| // SDK managers cache unwrapped keys. Even an unweighted cache must release this table | ||
| // with its statement, so a later statement performs its own catalog/KMS authorization. | ||
| this.cacheable = !IcebergEncryptionRead.isEncrypted(table); |
There was a problem hiding this comment.
[P2] Fence encrypted SQL result-cache replay as well
Declining this owner does not ensure current KMS authorization when enable_hive_sql_cache=true and enable_sql_cache=true. Once an encrypted snapshot passes the SQL-cache quiet window, warm SELECT COUNT(*) (or SUM), revoke KMS Decrypt at the same endpoint without changing metadata, and repeat the SQL. IcebergConnectorMetadata.getSqlCacheVersion returns the same UUID/snapshot/schema/metadata-location token, so NereidsSqlCacheManager returns cached rows without opening a manifest or invoking KMS. Reconstructing the SDK manager does not help because key unwrap is lazy. Exclude encrypted tables from result-cache admission and replay, or reauthorize hits; an empty connector token alone can fall back to legacy freshness caching. Add this explicitly enabled cache mode to the revocation regression.
Background
Native Parquet data/delete readers and binary per-file key descriptors exist, but encrypted SQL must stay closed until both private RPC channels and the actual selected readers can prove support.
What problem does this PR solve?
Issue Number: None
Related PR: #70675, #70693
PR Type: Enhancement
Add default-off HMS v3 native encrypted reads with fresh backend capability leases, dispatch/fetch checks and end-to-end SQL coverage. The existing cache/lifetime and private transport changes remain independently reviewed in the related PRs. Until they merge, this draft also contains those prerequisite commits; the integration change starts at
55be6efea33e.Details
Testing
fe/fe-connector/docs/iceberg-encryption-validation.md. Shared-host results require performance review.Release note
Opt-in native Iceberg encrypted reads require HMS v3, the SDK standard encryption manager, supported Parquet encryption, authenticated private RPC and fresh capable nodes. The deployment switch remains false by default.
Check List (For Author)
AI Disclosure