Repository navigation
[fix](be) Remove heartbeat tokens from logs and errors - #69532
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: When a BE receives its cluster token or a new authentication token, heartbeat INFO logs include the credential values. A cluster-token mismatch also includes both tokens in the error response, which can propagate to FE logs. Keep the heartbeat event messages and invalid-token error while removing credential values. Add a unit test covering initialization, repeated heartbeats, successive authentication-token updates, and token mismatch, including checks that logs and error responses do not contain the credentials. ### Release note Prevent cluster and authentication tokens from appearing in heartbeat logs and token-mismatch errors. ### Check List (For Author) - Test: Unit Test - 9 selected BE ASAN tests passed via run-be-ut.sh, covering HeartbeatServerTest.*, HttpAuthTest.*, and ChangelogTtlTest.HeartbeatFencesClusterMasterAndEpoch. Set GLIBC_COMPATIBILITY=OFF for the local test run to avoid duplicate symbols from the LDB toolchain. clang-format v16, build hygiene, clang-tidy, and git diff --check passed. - Behavior changed: Yes, heartbeat diagnostics no longer contain token values. - Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
The heartbeat diagnostic changes remove the credential values while preserving token initialization, rotation and mismatch rejection. One P2 finding concerns a concurrent read of the new test's captured-log buffer.
Static coverage includes both changed files, FE heartbeat sending and response logging, Status serialization, the common cloud/non-cloud heartbeat implementation, test registration/cleanup, and the pinned glog/BRPC threading contracts. No builds or tests were run, as required. There were no existing review threads and no additional user-provided focus points.
Critical checkpoint conclusions:
- Goal and evidence: all four affected diagnostics are token-free; the unit test reaches initial tokens, two auth rotations, a repeated heartbeat and a mismatched cluster token. Runtime validation was not performed.
- Scope and reuse: four focused production diagnostic edits and one unit test; the engine fixture follows the existing heartbeat-test convention.
- Concurrency and locking: production state updates retain the existing heartbeat mutex and lock order. The new test reads its log buffer while background callbacks can append; see the inline finding. Production lock cost decreases with the removed formatting.
- Lifecycle and static initialization: no production lifetime or global-initialization change. Test RAII removes the sink before destruction and resets the engine after the server is destroyed; the outstanding issue is synchronization before buffer inspection.
- Configuration: no configuration item added or changed; dynamic-update behavior is unaffected.
- Symbol and storage compatibility: no symbol, storage-format or wire change. The error code remains INTERNAL_ERROR; only credential-bearing error text changes.
- Thrift/protobuf IDs: no definitions changed; cross-branch ID comparison is inapplicable.
- Parallel paths: cloud and shared-nothing modes use the same changed heartbeat implementation. No separate affected heartbeat path was omitted.
- Conditional checks: no production condition added or changed; existing token mismatch failure behavior is retained.
- Test coverage: the new test checks successful state transitions, repeated-token behavior, rejection, retained cluster token, and absence of all supplied secrets in logs and the error. CMake includes the test through agent/*.cpp. Capture synchronization needs the inline correction; a separate end-to-end test is not essential for these text edits.
- Expected results: no generated result files changed. Assertions match the inspected Status serialization and token transitions; no execution result is claimed.
- Observability: initialization/update events and invalid-token diagnosis remain, with host/backend context in surrounding logs and serialized status. New metrics are unnecessary for these edits.
- Transactions and persistence: no transaction, EditLog or durable-state change; failover and data-visibility behavior are unaffected by the diagnostic edits.
- Data writes and atomicity: existing in-memory token assignments are unchanged; no new data-write or crash-recovery path.
- FE/BE variables: none added. The normal FE sender and returned sanitized error propagation were traced.
- Performance and memory: less production string formatting; no new production allocation or algorithm. The test's concurrent string access is the identified memory-safety concern.
- Other checks: no new header layering, template instantiation, source-.cpp test inclusion or glibc API requirement. No further actionable issue was found in the reviewed scope.
Coverage limit: static review only; build, sanitizer and runtime behavior remain unverified.
Review converged after 2 round(s).
TPC-H: Total hot run time: 28363 ms |
TPC-DS: Total hot run time: 150215 ms |
ClickBench: Total hot run time: 24.4 s |
Problem Summary: When a BE receives its cluster token or a new authentication token, heartbeat INFO logs include the credential values. A cluster-token mismatch also includes both tokens in the error response, which can propagate to FE logs. Keep the heartbeat event messages and invalid-token error while removing credential values. Add a unit test covering initialization, repeated heartbeats, successive authentication-token updates, and token mismatch, including checks that logs and error responses do not contain the credentials.
Release note
Prevent cluster and authentication tokens from appearing in heartbeat logs and token-mismatch errors.
Check List (For Author)
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)