Skip to content

[fix](be) Remove heartbeat tokens from logs and errors - #69532

Merged
deardeng merged 1 commit into
apache:masterfrom
deardeng:fix-heartbeat-token
Oct 26, 2026
Merged

deardeng merged 1 commit into
apache:masterfrom
deardeng:fix-heartbeat-token

Conversation

@deardeng

Copy link
Copy Markdown
Contributor

Problem Summary: When a BE receives its cluster token or a new authentication token, heartbeat INFO logs include the credential values. A cluster-token mismatch also includes both tokens in the error response, which can propagate to FE logs. Keep the heartbeat event messages and invalid-token error while removing credential values. Add a unit test covering initialization, repeated heartbeats, successive authentication-token updates, and token mismatch, including checks that logs and error responses do not contain the credentials.

Release note

Prevent cluster and authentication tokens from appearing in heartbeat logs and token-mismatch errors.

Check List (For Author)

  • Test: Unit Test - 9 selected BE ASAN tests passed via run-be-ut.sh, covering HeartbeatServerTest., HttpAuthTest., and ChangelogTtlTest.HeartbeatFencesClusterMasterAndEpoch. Set GLIBC_COMPATIBILITY=OFF for the local test run to avoid duplicate symbols from the LDB toolchain. clang-format v16, build hygiene, clang-tidy, and git diff --check passed.
  • Behavior changed: Yes, heartbeat diagnostics no longer contain token values.
  • Does this need documentation: No

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: When a BE receives its cluster token or a new authentication
token, heartbeat INFO logs include the credential values. A cluster-token
mismatch also includes both tokens in the error response, which can propagate
to FE logs. Keep the heartbeat event messages and invalid-token error while
removing credential values. Add a unit test covering initialization, repeated
heartbeats, successive authentication-token updates, and token mismatch,
including checks that logs and error responses do not contain the credentials.

### Release note

Prevent cluster and authentication tokens from appearing in heartbeat logs
and token-mismatch errors.

### Check List (For Author)

- Test: Unit Test - 9 selected BE ASAN tests passed via run-be-ut.sh,
  covering HeartbeatServerTest.*, HttpAuthTest.*, and
  ChangelogTtlTest.HeartbeatFencesClusterMasterAndEpoch. Set
  GLIBC_COMPATIBILITY=OFF for the local test run to avoid duplicate symbols
  from the LDB toolchain. clang-format v16, build hygiene, clang-tidy,
  and git diff --check passed.
- Behavior changed: Yes, heartbeat diagnostics no longer contain token values.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

@deardeng

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The heartbeat diagnostic changes remove the credential values while preserving token initialization, rotation and mismatch rejection. One P2 finding concerns a concurrent read of the new test's captured-log buffer.

Static coverage includes both changed files, FE heartbeat sending and response logging, Status serialization, the common cloud/non-cloud heartbeat implementation, test registration/cleanup, and the pinned glog/BRPC threading contracts. No builds or tests were run, as required. There were no existing review threads and no additional user-provided focus points.

Critical checkpoint conclusions:

  • Goal and evidence: all four affected diagnostics are token-free; the unit test reaches initial tokens, two auth rotations, a repeated heartbeat and a mismatched cluster token. Runtime validation was not performed.
  • Scope and reuse: four focused production diagnostic edits and one unit test; the engine fixture follows the existing heartbeat-test convention.
  • Concurrency and locking: production state updates retain the existing heartbeat mutex and lock order. The new test reads its log buffer while background callbacks can append; see the inline finding. Production lock cost decreases with the removed formatting.
  • Lifecycle and static initialization: no production lifetime or global-initialization change. Test RAII removes the sink before destruction and resets the engine after the server is destroyed; the outstanding issue is synchronization before buffer inspection.
  • Configuration: no configuration item added or changed; dynamic-update behavior is unaffected.
  • Symbol and storage compatibility: no symbol, storage-format or wire change. The error code remains INTERNAL_ERROR; only credential-bearing error text changes.
  • Thrift/protobuf IDs: no definitions changed; cross-branch ID comparison is inapplicable.
  • Parallel paths: cloud and shared-nothing modes use the same changed heartbeat implementation. No separate affected heartbeat path was omitted.
  • Conditional checks: no production condition added or changed; existing token mismatch failure behavior is retained.
  • Test coverage: the new test checks successful state transitions, repeated-token behavior, rejection, retained cluster token, and absence of all supplied secrets in logs and the error. CMake includes the test through agent/*.cpp. Capture synchronization needs the inline correction; a separate end-to-end test is not essential for these text edits.
  • Expected results: no generated result files changed. Assertions match the inspected Status serialization and token transitions; no execution result is claimed.
  • Observability: initialization/update events and invalid-token diagnosis remain, with host/backend context in surrounding logs and serialized status. New metrics are unnecessary for these edits.
  • Transactions and persistence: no transaction, EditLog or durable-state change; failover and data-visibility behavior are unaffected by the diagnostic edits.
  • Data writes and atomicity: existing in-memory token assignments are unchanged; no new data-write or crash-recovery path.
  • FE/BE variables: none added. The normal FE sender and returned sanitized error propagation were traced.
  • Performance and memory: less production string formatting; no new production allocation or algorithm. The test's concurrent string access is the identified memory-safety concern.
  • Other checks: no new header layering, template instantiation, source-.cpp test inclusion or glibc API requirement. No further actionable issue was found in the reviewed scope.

Coverage limit: static review only; build, sanitizer and runtime behavior remain unverified.

Review converged after 2 round(s).

Comment thread be/test/agent/heartbeat_server_test.cpp Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28363 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f31b3981178d58c48e0fc2efb054071bc57781d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17604	5156	5054	5054
q2	2213	309	255	255
q3	10189	1428	819	819
q4	4679	462	335	335
q5	7424	790	539	539
q6	182	184	152	152
q7	750	792	601	601
q8	9262	1323	1076	1076
q9	5390	4190	4197	4190
q10	6827	1281	1025	1025
q11	432	245	225	225
q12	632	396	279	279
q13	18102	3711	3027	3027
q14	272	267	249	249
q15	q16	761	725	676	676
q17	1217	750	559	559
q18	6985	6188	6122	6122
q19	1113	948	600	600
q20	407	349	227	227
q21	5389	2104	2268	2104
q22	349	293	249	249
Total cold run time: 100179 ms
Total hot run time: 28363 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5604	5572	5586	5572
q2	676	571	538	538
q3	5179	5349	4927	4927
q4	1973	2126	1520	1520
q5	5321	5184	5534	5184
q6	275	226	164	164
q7	2329	2077	1922	1922
q8	3056	2896	2703	2703
q9	8054	8073	7852	7852
q10	3732	3681	3233	3233
q11	606	429	416	416
q12	668	682	510	510
q13	3556	3805	3304	3304
q14	318	295	279	279
q15	q16	695	724	639	639
q17	8257	7478	7246	7246
q18	12958	12281	12975	12281
q19	921	800	790	790
q20	2141	2152	1929	1929
q21	5725	4575	4745	4575
q22	487	466	429	429
Total cold run time: 72531 ms
Total hot run time: 66013 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150215 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f31b3981178d58c48e0fc2efb054071bc57781d, data reload: false

query5	4322	600	450	450
query6	439	210	193	193
query7	4820	439	230	230
query8	319	162	145	145
query9	8763	3046	3082	3046
query10	483	309	254	254
query11	5997	3859	3555	3555
query12	147	90	86	86
query13	1294	454	309	309
query14	6144	3914	3634	3634
query14_1	3363	3369	3326	3326
query15	203	199	177	177
query16	964	411	419	411
query17	842	657	554	554
query18	2409	415	297	297
query19	219	163	130	130
query20	83	83	77	77
query21	211	124	105	105
query22	12920	12992	12792	12792
query23	13583	12828	12429	12429
query23_1	12512	12587	12456	12456
query24	7391	821	461	461
query24_1	470	484	488	484
query25	572	426	365	365
query26	1254	255	142	142
query27	2748	439	276	276
query28	4597	1920	1898	1898
query29	1550	601	448	448
query30	306	228	184	184
query31	1001	930	818	818
query32	185	97	106	97
query33	533	306	247	247
query34	893	844	488	488
query35	741	754	636	636
query36	828	807	731	731
query37	127	100	83	83
query38	1806	1723	1671	1671
query39	814	806	740	740
query39_1	735	731	744	731
query40	240	120	96	96
query41	70	67	68	67
query42	88	90	84	84
query43	342	366	315	315
query44	1315	679	690	679
query45	183	178	164	164
query46	844	954	566	566
query47	2942	2860	2725	2725
query48	296	326	227	227
query49	591	402	298	298
query50	683	274	213	213
query51	10590	10498	10494	10494
query52	77	81	72	72
query53	201	218	153	153
query54	258	200	207	200
query55	73	73	63	63
query56	280	238	215	215
query57	1669	1628	1577	1577
query58	314	273	281	273
query59	2138	2140	1890	1890
query60	292	257	234	234
query61	171	160	165	160
query62	395	335	287	287
query63	193	160	161	160
query64	2834	952	733	733
query65	3294	3248	3231	3231
query66	1795	410	292	292
query67	20246	20015	19831	19831
query68	2977	1006	610	610
query69	400	292	261	261
query70	814	768	759	759
query71	298	239	195	195
query72	2784	2635	2269	2269
query73	504	514	301	301
query74	4672	4514	4356	4356
query75	2343	2308	1907	1907
query76	2312	1019	593	593
query77	335	375	260	260
query78	9707	9662	9178	9178
query79	989	823	513	513
query80	1221	413	360	360
query81	590	317	271	271
query82	1272	141	103	103
query83	304	191	182	182
query84	318	120	104	104
query85	870	432	384	384
query86	425	243	231	231
query87	1971	1928	1800	1800
query88	3764	2671	2636	2636
query89	341	278	243	243
query90	1875	170	170	170
query91	165	155	133	133
query92	98	125	101	101
query93	1037	1001	579	579
query94	749	316	271	271
query95	562	386	301	301
query96	671	476	234	234
query97	2462	2429	2311	2311
query98	168	148	158	148
query99	765	755	665	665
Total cold run time: 236395 ms
Total hot run time: 150215 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f31b3981178d58c48e0fc2efb054071bc57781d, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.10	0.10
query5	0.24	0.25	0.25
query6	1.17	0.68	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.34	0.35
query10	0.58	0.56	0.60
query11	0.19	0.14	0.14
query12	0.18	0.14	0.15
query13	0.49	0.48	0.48
query14	1.04	1.01	1.02
query15	0.63	0.60	0.60
query16	0.34	0.35	0.33
query17	1.11	1.15	1.14
query18	0.23	0.20	0.19
query19	2.12	1.93	1.94
query20	0.02	0.01	0.01
query21	15.45	0.20	0.14
query22	4.75	0.05	0.06
query23	16.24	0.31	0.12
query24	3.11	0.50	0.42
query25	0.14	0.04	0.05
query26	1.27	0.20	0.15
query27	0.08	0.04	0.05
query28	3.86	0.77	0.35
query29	12.54	4.28	3.37
query30	0.27	0.12	0.10
query31	2.77	0.55	0.32
query32	3.22	0.60	0.50
query33	3.14	3.17	3.20
query34	15.48	3.97	3.43
query35	3.34	3.35	3.34
query36	0.56	0.45	0.42
query37	0.11	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.57 s
Total hot run time: 24.4 s

@deardeng
deardeng merged commit c3c7af5 into apache:master Oct 26, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants