Skip to content

[fix](cache) Invalidate query caches when replaying table changes - #63612

Merged
924060929 merged 13 commits into
apache:masterfrom
924060929:fix/sql-cache-replay-invalidate
Nov 9, 2026
Merged

924060929 merged 13 commits into
apache:masterfrom
924060929:fix/sql-cache-replay-invalidate

Conversation

@924060929

@924060929 924060929 commented May 25, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

A query cached on a follower or observer can keep returning an old schema after a table change. For example, a heavy ALTER TABLE ... DROP COLUMN switches the base index without changing the table's data visible version. The master's ALTER path clears its local SQL cache, while another FE replays the schema change and retains the old cached result.

This change invalidates derived query caches while applying existing metadata journals. Ordinary DML publishes table and partition visible versions, which SQL cache admission and lookup validate without scanning the cache under transaction table write locks. It replaces the earlier implementation of this PR that emitted a separate metadata-change journal.

Implementation

Env.onTableChanged is the shared entry point. Every cache invalidation runs in its own try/catch, logs a failure, and allows the remaining cache invalidations to run. Checkpoint replay skips these local derived caches.

Change Invalidation
Table metadata SQL result contexts, sorted partition ranges, dependent MTMV rewrite plans, and exact simple-aggregate caches
Visible DML Shared admission/lookup validation rejects changed table-wide or partition dependencies; unaffected partition dependencies remain reusable. No full-cache scan during transaction publication or replay
Database/catalog namespace Matching SQL, sorted partition, and MTMV rewrite caches, including cold external objects

The hook runs at the existing mutation boundary: schema-change/rollup completion, table drop/replacement, and truncate. Transaction visibility retains its original partition-before-table version publication; data validation is performed when admitting or looking up a cache entry. Remaining replay handlers notify it after applying their mutation. Renames and replacements notify after applying the name mapping, preserving the original names and table IDs under the existing locks. Batch partition changes notify each affected table once.

Replay coverage includes table/view creation and removal, recovery, column/table/partition/rollup renames, heavy/light schema and index changes, partition changes/replacement, table replacement/engine/properties/distribution changes, administrative version changes, constraints, MTMV changes, branch/tag operations, and database/catalog changes. External refreshes reuse their existing invalidation protocol and retain the row-count cache's final invalidation barrier.

SQL cache publication and lookup share dependency validation: local metadata revisions, table identity/schema/data versions, view definitions, and external catalog properties. Revisions are scoped by table ID/name, database and catalog; each statement retains its own dependency snapshot, with no registry of executing contexts. Publication uses a fresh statement and connector scope, restores the executing statement in finally, and skips cache admission if validation fails. Lookup also rejects a late put that races invalidation. The BE cache key includes the base index/schema identity, and MTMV rewrite invalidation reuses the existing generation barrier.

Partition-version validation constructs temporary partition storage only after the metadata and unchanged-version fast checks.

No journal opcode, extra journal record, persisted payload, or FE/BE protocol field is added. Approximate statistics and transaction-scoped auto-partition routing retain their existing lifecycles.

Release note

Fix stale cached query results on follower and observer FEs after table metadata changes.

Check List (For Author)

  • Test: Latest-head preset FE UT run: 76 tests across 6 focused classes, 0 failures/errors/skips; build.sh --fe and integrated Checkstyle pass. Includes native/Cloud partition-version checks, actual cache lookup, serialized transaction replay, FE/BE publication, metadata revisions and data policy. Before/after compiler and JDK bytecode confirms the unchanged-version branch skips partition-list construction; no heap/GC/throughput benchmark is claimed. Earlier broader and multi-FE validation is historical and was not rerun for this allocation-only follow-up.
  • Behavior changed: Yes. Existing replayed table changes invalidate their local dependent query caches.
  • Does this need documentation: No

Earlier local multi-FE validation used the then-patched master FE with an existing 4.1.3 BE. The isolated cluster sets be_exec_version=10 and disables the newer local shuffle planner/local exchange to accommodate that BE. This is not a same-head FE/BE build or a Cloud cluster validation.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 55.56% (45/81) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 32203 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a61db331bb784765865901120e601340b8731ef1, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17774	4165	4143	4143
q2	q3	10868	1465	820	820
q4	4753	484	351	351
q5	9679	2312	2125	2125
q6	383	179	138	138
q7	961	812	647	647
q8	9590	1788	1667	1667
q9	6949	5054	4956	4956
q10	6532	2251	1929	1929
q11	447	273	246	246
q12	691	437	302	302
q13	18237	3416	2880	2880
q14	270	265	237	237
q15	q16	834	785	717	717
q17	1010	1015	1036	1015
q18	7111	5678	6195	5678
q19	1343	1242	1134	1134
q20	519	443	261	261
q21	6043	2723	2641	2641
q22	436	374	316	316
Total cold run time: 104430 ms
Total hot run time: 32203 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4972	4851	4928	4851
q2	q3	4865	5390	4669	4669
q4	2158	2226	1415	1415
q5	4803	4922	4707	4707
q6	242	207	128	128
q7	1850	1745	1615	1615
q8	2487	2087	1992	1992
q9	7439	7491	7508	7491
q10	4781	4702	4226	4226
q11	585	388	374	374
q12	740	750	540	540
q13	3077	3398	2828	2828
q14	283	285	252	252
q15	q16	696	715	610	610
q17	1324	1315	1303	1303
q18	7329	6889	6815	6815
q19	1125	1135	1134	1134
q20	2244	2232	1964	1964
q21	5368	4655	4553	4553
q22	525	483	409	409
Total cold run time: 56893 ms
Total hot run time: 51876 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 172962 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a61db331bb784765865901120e601340b8731ef1, data reload: false

query5	4316	664	527	527
query6	336	222	206	206
query7	4218	587	321	321
query8	326	233	219	219
query9	8862	4076	4128	4076
query10	458	341	303	303
query11	5791	2435	2238	2238
query12	174	129	127	127
query13	1289	651	437	437
query14	6178	5490	5221	5221
query14_1	4534	4523	4459	4459
query15	209	205	182	182
query16	999	450	447	447
query17	996	746	592	592
query18	2472	489	362	362
query19	211	200	163	163
query20	135	131	128	128
query21	208	146	116	116
query22	13595	13551	13341	13341
query23	17489	16648	16351	16351
query23_1	16321	16377	16396	16377
query24	7537	1808	1346	1346
query24_1	1341	1328	1353	1328
query25	571	505	472	472
query26	1317	323	178	178
query27	2736	579	355	355
query28	4449	2024	2066	2024
query29	1038	682	541	541
query30	305	239	206	206
query31	1132	1083	949	949
query32	94	82	76	76
query33	553	372	317	317
query34	1192	1168	671	671
query35	790	817	703	703
query36	1423	1420	1252	1252
query37	156	105	94	94
query38	3253	3176	3083	3083
query39	944	922	886	886
query39_1	899	881	906	881
query40	236	152	133	133
query41	75	71	70	70
query42	116	115	113	113
query43	340	342	303	303
query44	
query45	235	209	206	206
query46	1083	1200	707	707
query47	2381	2429	2272	2272
query48	407	439	302	302
query49	660	508	407	407
query50	1092	357	257	257
query51	4380	4314	4283	4283
query52	112	107	97	97
query53	254	293	221	221
query54	335	291	275	275
query55	101	101	89	89
query56	331	325	313	313
query57	1443	1435	1372	1372
query58	314	287	289	287
query59	1684	1705	1456	1456
query60	343	365	313	313
query61	156	158	156	156
query62	703	653	591	591
query63	247	200	196	196
query64	2408	803	651	651
query65	
query66	1712	472	366	366
query67	29992	29958	29913	29913
query68	
query69	463	346	307	307
query70	1038	990	1010	990
query71	301	274	271	271
query72	2972	2723	2458	2458
query73	879	793	438	438
query74	5172	4942	4799	4799
query75	2702	2622	2263	2263
query76	2274	1164	765	765
query77	410	411	330	330
query78	12440	12433	11884	11884
query79	1508	1059	789	789
query80	631	548	451	451
query81	460	281	245	245
query82	1338	158	121	121
query83	366	285	253	253
query84	292	141	115	115
query85	889	544	439	439
query86	403	316	364	316
query87	3444	3429	3256	3256
query88	3686	2774	2742	2742
query89	462	389	348	348
query90	1951	189	179	179
query91	178	168	139	139
query92	79	76	75	75
query93	1521	1446	872	872
query94	534	368	308	308
query95	659	385	444	385
query96	1097	820	339	339
query97	2753	2775	2623	2623
query98	239	233	238	233
query99	1191	1168	1046	1046
Total cold run time: 255412 ms
Total hot run time: 172962 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 76.54% (62/81) 🎉
Increment coverage report
Complete coverage report

@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 56.63% (47/83) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 32080 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9a91daabeabb276c3f73f53c18edef17439a6a2, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	16652	4217	4221	4217
q2	q3	10376	1374	843	843
q4	4622	482	353	353
q5	7982	2300	2162	2162
q6	251	190	139	139
q7	951	820	630	630
q8	9367	1890	1638	1638
q9	5221	5014	5042	5014
q10	6403	2211	1895	1895
q11	438	287	250	250
q12	640	442	306	306
q13	18094	3402	2795	2795
q14	279	264	240	240
q15	q16	809	784	715	715
q17	908	907	986	907
q18	7059	5799	5609	5609
q19	1207	1327	1202	1202
q20	520	401	269	269
q21	5644	2588	2601	2588
q22	446	370	308	308
Total cold run time: 97869 ms
Total hot run time: 32080 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4524	4451	4422	4422
q2	q3	4618	5054	4332	4332
q4	2202	2287	1461	1461
q5	4532	4459	4683	4459
q6	284	218	152	152
q7	2229	1992	1721	1721
q8	2634	2435	2379	2379
q9	8389	8121	8243	8121
q10	4844	4778	4423	4423
q11	597	457	411	411
q12	793	786	558	558
q13	3306	3640	3010	3010
q14	296	302	284	284
q15	q16	736	730	660	660
q17	1419	1587	1394	1394
q18	7996	7410	7451	7410
q19	1167	1088	1132	1088
q20	2225	2254	1960	1960
q21	5424	4816	4749	4749
q22	527	470	412	412
Total cold run time: 58742 ms
Total hot run time: 53406 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 53.33% (64/120) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 173563 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9a91daabeabb276c3f73f53c18edef17439a6a2, data reload: false

query5	4321	695	526	526
query6	351	225	201	201
query7	4258	620	296	296
query8	346	233	230	230
query9	8860	4229	4266	4229
query10	461	361	317	317
query11	5817	2418	2267	2267
query12	178	133	131	131
query13	1308	607	446	446
query14	6176	5659	5309	5309
query14_1	4608	4605	4586	4586
query15	222	210	191	191
query16	1020	468	466	466
query17	1169	759	643	643
query18	2499	513	373	373
query19	224	215	177	177
query20	144	137	132	132
query21	219	141	123	123
query22	13669	13591	13452	13452
query23	17416	16643	16363	16363
query23_1	16314	16340	16415	16340
query24	7361	1858	1310	1310
query24_1	1331	1322	1374	1322
query25	556	491	423	423
query26	1330	335	177	177
query27	2664	572	347	347
query28	4338	2047	2028	2028
query29	1001	640	517	517
query30	309	240	196	196
query31	1156	1110	958	958
query32	91	84	79	79
query33	553	398	301	301
query34	1194	1195	649	649
query35	805	811	692	692
query36	1415	1411	1240	1240
query37	156	110	91	91
query38	3244	3194	3103	3103
query39	933	957	902	902
query39_1	891	896	877	877
query40	226	150	131	131
query41	67	65	67	65
query42	115	112	109	109
query43	351	352	306	306
query44	
query45	219	206	198	198
query46	1078	1220	748	748
query47	2344	2411	2175	2175
query48	438	423	314	314
query49	632	489	396	396
query50	972	353	279	279
query51	4444	4550	4338	4338
query52	107	108	95	95
query53	257	291	207	207
query54	314	284	257	257
query55	101	98	87	87
query56	297	311	306	306
query57	1444	1418	1329	1329
query58	314	285	277	277
query59	1662	1716	1493	1493
query60	327	329	317	317
query61	162	156	161	156
query62	696	656	595	595
query63	250	208	208	208
query64	2442	819	652	652
query65	
query66	1699	475	373	373
query67	29850	29768	29600	29600
query68	
query69	476	343	301	301
query70	1065	1057	1018	1018
query71	312	280	269	269
query72	3032	2780	2498	2498
query73	842	792	428	428
query74	5146	4967	4858	4858
query75	2707	2633	2279	2279
query76	2048	1150	813	813
query77	410	422	350	350
query78	12542	12401	11844	11844
query79	1245	1110	770	770
query80	598	567	506	506
query81	464	280	244	244
query82	239	159	129	129
query83	281	282	257	257
query84	286	144	110	110
query85	877	535	446	446
query86	380	339	360	339
query87	3425	3411	3266	3266
query88	3682	2781	2775	2775
query89	430	394	348	348
query90	2053	195	189	189
query91	178	168	142	142
query92	80	84	79	79
query93	1447	1438	830	830
query94	549	350	329	329
query95	701	407	360	360
query96	1075	819	392	392
query97	2722	2764	2595	2595
query98	237	230	230	230
query99	1163	1157	1041	1041
Total cold run time: 253345 ms
Total hot run time: 173563 ms

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for a failover correctness gap in the new table-meta-change journal signal.

Critical checkpoint conclusions:

  • Goal/test: The PR targets follower FE cache invalidation after table metadata replay. The approach covers the normal happy path, but no automated regression or failover test is added, and the implementation is not safe across master failover.
  • Scope: The change is focused, but it introduces a separate persistent operation for a transient cache side effect, which makes ordering/atomicity part of correctness.
  • Concurrency/lifecycle: No new threads; cache invalidation can race with query serving as before, but the larger issue is journal ordering across crash/failover.
  • Configuration/compatibility: No new config. A new edit-log opcode is added and read/write dispatch exists.
  • Parallel paths: The generic replay signal avoids updating many replay paths, but because it is not part of those metadata logs, any crash before the follow-up signal loses the invalidation.
  • Transaction/persistence: Blocking issue found: metadata and invalidation are persisted independently, so master failover between them can replay the metadata without ever replaying the invalidation.
  • Tests: No automated test covers follower replay invalidation or the failover window.
  • Observability/performance: No blocking observability or performance issue found for the current cache sizes.

User focus: No additional user-provided review focus was supplied.

return;
}
TableMetaChange change =
TableMetaChange.fromTable(table);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the invalidation durable only as a second edit-log record after the real metadata mutation has already been logged by the caller. If the master crashes after the metadata log is committed but before this OP_TABLE_META_CHANGE is committed, the promoted FE replays the schema/property change but never replays this invalidation, so its existing SQL cache can keep serving stale results indefinitely. For example, ALTER TABLE ... RENAME COLUMN logs the rename in the alter path, then reaches notifyTableMetaChange; a crash in this window recreates the original follower-cache bug this PR is fixing. The invalidation needs to be tied to replaying the metadata operation itself, or otherwise made atomic with the metadata journal entry, rather than relying on a best-effort follow-up journal record.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a complementary fix in the latest commit: store OlapTable.getBaseSchemaVersion() in the cache entry and check it on lookup (tablesOrDataChanged). This provides a crash-window-free safety net for OlapTable schema DDLs (rename/add/drop column) — if the schema version changed, the cache entry is invalidated on next access regardless of whether OP_TABLE_META_CHANGE was received.

Now there are two layers:

  • Layer 2 (schemaVersion, lazy): no crash window, covers OlapTable schema changes automatically
  • Layer 3 (OP_TABLE_META_CHANGE, eager): covers all table types and non-schema DDLs (alter properties, etc.), has a tiny crash window

The crash window in Layer 3 only matters for non-OlapTable or non-schema DDLs — and for those, the existing tablesOrDataChanged checks (visibleVersion, table id, etc.) already catch most cases.

@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

924060929 added a commit to 924060929/incubator-doris that referenced this pull request May 26, 2026
…replay (apache#63612)

Cherry-pick of apache#63612 to branch-4.1.

Two-layer fix:
1. Check OlapTable.baseSchemaVersion on cache lookup (lazy, no crash window)
2. OP_TABLE_META_CHANGE journal broadcast (eager, all table types)

Add enable_write_op_table_meta_change FE config (default false) to gate
the OP_TABLE_META_CHANGE edit log writing. When false, master still does
local cache invalidation but does not broadcast to followers via journal.
The schemaVersion check still works regardless of this config.
924060929 added a commit to 924060929/incubator-doris that referenced this pull request May 26, 2026
…replay (apache#63612)

Cherry-pick of apache#63612 to branch-4.0.

Two-layer fix:
1. Check OlapTable.baseSchemaVersion on cache lookup (lazy, no crash window)
2. OP_TABLE_META_CHANGE journal broadcast (eager, all table types)

Add enable_write_op_table_meta_change FE config (default false) to gate
the OP_TABLE_META_CHANGE edit log writing. When false, master still does
local cache invalidation but does not broadcast to followers via journal.
The schemaVersion check still works regardless of this config.
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 21.90% (60/274) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 31993 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dfc8e2c9079955b77196d55a98a6e56c1c72eecc, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	16871	4160	4195	4160
q2	q3	10494	1407	856	856
q4	4661	479	343	343
q5	7816	2275	2092	2092
q6	248	174	142	142
q7	923	790	635	635
q8	9395	1717	1776	1717
q9	5250	5022	4989	4989
q10	6387	2172	1871	1871
q11	431	268	249	249
q12	629	439	299	299
q13	18145	3331	2746	2746
q14	266	257	243	243
q15	q16	821	770	705	705
q17	898	978	1011	978
q18	7050	5687	5581	5581
q19	1171	1259	1116	1116
q20	522	542	307	307
q21	6220	2822	2654	2654
q22	454	373	310	310
Total cold run time: 98652 ms
Total hot run time: 31993 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4904	4893	5131	4893
q2	q3	4826	5240	4675	4675
q4	2169	2276	1467	1467
q5	4965	4749	4923	4749
q6	236	181	133	133
q7	1894	1715	1563	1563
q8	2531	2162	2103	2103
q9	7565	7521	7465	7465
q10	4749	4732	4283	4283
q11	555	398	365	365
q12	742	750	538	538
q13	3051	3480	2827	2827
q14	273	282	263	263
q15	q16	689	714	608	608
q17	1323	1275	1279	1275
q18	7341	6941	7009	6941
q19	1118	1106	1087	1087
q20	2238	2228	1967	1967
q21	5375	4671	4608	4608
q22	531	479	413	413
Total cold run time: 57075 ms
Total hot run time: 52223 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 172625 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dfc8e2c9079955b77196d55a98a6e56c1c72eecc, data reload: false

query5	4396	651	528	528
query6	349	218	200	200
query7	4232	582	318	318
query8	333	241	217	217
query9	8859	4082	4119	4082
query10	441	337	294	294
query11	5785	2621	2263	2263
query12	184	131	124	124
query13	1258	606	427	427
query14	6122	5515	5211	5211
query14_1	4496	4547	4491	4491
query15	212	205	186	186
query16	1015	455	423	423
query17	1121	743	600	600
query18	2468	503	365	365
query19	223	213	168	168
query20	137	134	134	134
query21	216	151	129	129
query22	13771	13631	13541	13541
query23	17313	16576	16185	16185
query23_1	16378	16354	16519	16354
query24	7441	1812	1315	1315
query24_1	1348	1316	1307	1307
query25	588	505	434	434
query26	1312	328	176	176
query27	2697	581	341	341
query28	4388	2026	2020	2020
query29	1011	652	537	537
query30	315	244	201	201
query31	1116	1101	970	970
query32	96	80	78	78
query33	554	364	315	315
query34	1208	1192	676	676
query35	789	813	688	688
query36	1384	1419	1300	1300
query37	156	107	94	94
query38	3239	3190	3124	3124
query39	962	919	893	893
query39_1	881	877	878	877
query40	231	149	133	133
query41	74	71	69	69
query42	113	111	111	111
query43	341	343	303	303
query44	
query45	221	210	207	207
query46	1090	1212	730	730
query47	2375	2416	2237	2237
query48	418	441	317	317
query49	661	536	409	409
query50	982	363	272	272
query51	4358	4285	4341	4285
query52	108	108	96	96
query53	266	287	214	214
query54	336	294	283	283
query55	96	95	93	93
query56	334	315	313	313
query57	1436	1443	1349	1349
query58	316	287	289	287
query59	1656	1735	1525	1525
query60	319	344	306	306
query61	160	156	153	153
query62	705	653	593	593
query63	238	206	204	204
query64	2393	834	635	635
query65	
query66	1719	480	350	350
query67	30131	29896	29633	29633
query68	
query69	464	346	313	313
query70	1020	990	990	990
query71	304	277	282	277
query72	3021	2724	2471	2471
query73	847	745	431	431
query74	5134	4974	4821	4821
query75	2702	2617	2288	2288
query76	2143	1154	784	784
query77	407	425	338	338
query78	12357	12372	11872	11872
query79	1524	1126	782	782
query80	815	536	472	472
query81	475	286	252	252
query82	1283	160	124	124
query83	352	275	250	250
query84	275	150	111	111
query85	923	535	450	450
query86	453	364	322	322
query87	3441	3401	3272	3272
query88	3666	2719	2749	2719
query89	460	397	346	346
query90	1776	196	179	179
query91	178	168	139	139
query92	83	83	69	69
query93	1454	1445	835	835
query94	631	372	320	320
query95	665	380	337	337
query96	1099	795	347	347
query97	2759	2762	2608	2608
query98	247	227	231	227
query99	1163	1151	1033	1033
Total cold run time: 254830 ms
Total hot run time: 172625 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 43 changed files and the relevant mutation/replay, cache publication/lookup, external-catalog and MTMV interactions at head 7310857. One new P2 performance regression is reported inline: unchanged partition-aware cache hits allocate partition-sized temporary storage before their version fast return. No new data-correctness defect or confirmed existing P0/P1 blocker remains. Review converged after two rounds; all three reviewers reported NO_NEW_VALUABLE_FINDINGS in the latest round, and every candidate and initial risk has an evidence-based disposition.

Prior context: catalog-retirement P1 #4480838554 is fixed by named retirement through onCatalogChanged; catalog/MV lock-cycle P1 #4488328455 is fixed by copying dependency names before external resolution; transaction-scan P2 #4489252243 is fixed by retaining lazy dependency validation. Earlier second-journal crash-window/version-skew concerns refer to the superseded implementation. The new inline addresses a distinct per-hit allocation. No additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and proof: existing metadata journals now invalidate local SQL, sorted-partition, MTMV rewrite and exact aggregate caches; shared admission/lookup checks fence stale candidates. Serialized replay and race tests were inspected, with runtime success unverified in this static review.
  2. Scope and reuse: changes are focused on derived-cache invalidation and dependency validation, with common Env fanout and one shared validator.
  3. Concurrency and locks: lookup retains ID-sorted table read locks; MV builds and external dependency resolution occur outside MV locks. Admission releases its own metadata locks around cloud version IO. Native VISIBLE publication retains partition-before-table updates without scanning cached contexts under write locks. No additional concrete lock cycle was established.
  4. Lifecycle and initialization: snapshots retain their revision cells without registering executing contexts. Temporary admission scopes close independently and the executing statement is restored in finally; prepared executions get fresh snapshots. The ID fallback no longer retains a connection. No new static initialization dependency was found.
  5. Configuration: no new production setting; the regression restores its modified FE configuration in finally.
  6. Compatibility: no new journal opcode, persisted payload or storage format. Existing journals receive direct replay invalidation; expanded ephemeral BE cache keys produce safe misses against old entries.
  7. Thrift/protobuf: no definitions changed; field-ID comparison is not applicable.
  8. Parallel paths: FE/BE admission, live/replay metadata, cloud/native restore, warm/cold external refresh and catalog retirement were traced. The existing cloud source of visible versions is preserved.
  9. Special conditions: checkpoint skips, cold name-only notifications, legacy internal CREATE payloads, row-TTL rejection and table-wide dependency fallbacks are consistent with their inspected lifecycle.
  10. Test coverage: inspected metadata/schema replay, renamed/replaced identities, unaffected partitions/tables, COMMITTED versus VISIBLE, late puts, scope cleanup and error containment. The multi-FE regression skips without a follower/observer; it explicitly warms drop/rename cases, while view/truncate cases do not assert their own prior cache admission.
  11. Expected results: the new output agrees with the SQL projection, insert, view arithmetic and truncate sequence. Output generation provenance cannot be established statically.
  12. Observability: invalidation/admission failures log warnings with cache/query context; normal hits add no INFO logging. No missing critical metric was identified for these existing cache paths.
  13. Persistence and failover: fanout follows existing replayed mutations without a second-record crash window; checkpoint replay skips disposable local caches. Original transaction publication semantics are retained.
  14. Data writes and core invariants: no storage write, delete-bitmap or native-memory implementation changes. Identity/schema/data checks reject stale cache reuse; original metadata mutation locks and restore finish-log/showState ordering remain intact.
  15. FE/BE passing: captured schema/base-index or partition identity contributes to the existing context MD5 shared by BE fetch/update; no new wire variable is required.
  16. Performance: the inline identifies the avoidable O(tracked partitions) allocation on unchanged-version hits. Result-row limits do not bound scan-partition dependencies. No benchmark or measured allocation claim is made.
  17. Further recheck: all changed paths, prior threads, candidate dispositions, risk focuses and the inline anchor were swept after the second round. No unresolved production question or uncovered changed path remains.

Validation limits: source and test inspection only. No builds, tests, benchmarks, runtime reproduction, source edits or submodule initialization were performed. The test-admission and output-provenance limits above remain; they are not unreviewed production scope.

Review converged after 2 round(s).

### Background

Partition-aware SQL cache admission and lookup compare metadata and table versions before revalidating individual partition dependencies. Unchanged-version hits can return without building a list of those partitions.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#63612
PR Type: Bugfix

Problem Summary: Splitting the metadata read locks moved construction of the partition-sized ArrayList before the unchanged-version fast return. Every repeated hit could therefore allocate storage proportional to tracked partitions, even for a one-row aggregate. Move the existing local initialization after the metadata and unchanged-version checks while retaining the locking and validation flow.

### Details

- Frequency: Every partition-aware unchanged-version SQL cache validation.
- Root cause: The local list was initialized before all early returns rather than only on the revalidation path.
- Why existing tests missed it: Existing semantic tests validate cache freshness and reuse; they do not measure temporary allocations.
- Fix: Declare the invocation-local list before the first read lock and initialize it after both early-return checks inside that lock. Keep the finally unlocks, Cloud IO between metadata lock sections, partition identity/schema recheck and validated version update.
- Other paths and branches: Native/Cloud admission and lookup share this helper. No new state, API, cache key, journal, executing-context tracking or protocol field.

### Testing

- ./run-fe-ut.sh --run with six focused FE classes: 76 cases, zero failures/errors/skips. Includes 14 native/Cloud version cases and 27 cache/replay cases.
- ./build.sh --fe: success, including integrated Checkstyle with zero violations.
- git diff --check: passed.
- Before/after JDK17 compiler bytecode: the original partition ArrayList allocation precedes the unchanged-version return; the repaired fast return skips that construction. JDK ArrayList(int) bytecode confirms positive capacity constructs Object[]. This is control-flow evidence, not a heap/GC/throughput measurement.
- Reused existing semantic coverage; no new implementation-shaped test, benchmark, full FE UT, remote regression or BE/Cloud runtime run for this follow-up.

### Release note

Reduce temporary partition storage allocation on unchanged-version SQL cache hits.

### Check List (For Author)

- Test: Focused unit tests, FE build/style and compiler artifact inspection above.
- Behavior changed: Yes. Unchanged-version validation skips the unnecessary partition-list allocation.
- Does this need documentation: No
- Pre-submission status: Partially verified: local focused validation passes; new-head remote CI remains separate.

### AI Disclosure

- Tool: Codex, gpt-6.1-sol
- Generated: Code and commit description.
- Human review: Maintainer review pending.
@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@924060929

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#63612
commit: 77a128ed5c2b710bd19834e5201d136d438d4040
base: b6c67510385391d0dd913589f1b323ec86ba7bd6
reviewed_at: 2026-11-06T17:23+08:00
reviewer: 924060929
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 2
converged: true

Notes for maintainers

  • Solo incremental review of all 43 final paths: 42 prepared patches are identical to qualified head 7310857; the complete updated OlapTableSqlCacheVersion patch, full matches method, shared admission/lookup consumer and actual JDK ArrayList constructor were freshly inspected. The valid remote P2 was accepted, repaired and followed by one clean focused round; convergence at round 2, no remaining finding. No subagents.
  • Fixed inline 4489745186: declare the local before the first read lock, but construct partition storage only after metadata and unchanged-version checks. Both bounded metadata locks/finally cleanup, Cloud IO outside the metadata lock, second-phase schema/partition identity checks and validated-version update remain. One production file changed (2 insertions, 1 deletion); no new context tracking, state, cache key, journal or wire format.
  • Preset FE validation passes 6 classes / 76 cases, zero failures/errors/skips, including 14 partition-version and 27 cache/replay cases. build.sh --fe, integrated Checkstyle and git diff --check pass; tested source hash equals this commit. Existing semantic tests were reused; no brittle test tied to local statement placement. Builds/tests occurred in the repair phase; final review was read-only.
  • Saved JDK17 compiler/JDK bytecode confirms the original partition-list allocation preceded the unchanged-version return, while the repaired fast return skips it. This is control-flow/artifact proof, not a new failing-before semantic test or a measured JIT/heap/GC/throughput result, and not a claim that all metadata allocations disappear. Initial-head CI had no failed check but several pending jobs; new remote CI/review remains separate. No full FE, new multi-FE/remote-connector, BE/Cloud runtime or benchmark run is claimed this turn.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28679 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77a128ed5c2b710bd19834e5201d136d438d4040, data reload: false

------ Round 1 ----------------------------------
============================================
q1	16436	5092	5138	5092
q2	2171	314	268	268
q3	9648	1393	822	822
q4	4396	460	331	331
q5	7047	808	541	541
q6	184	187	154	154
q7	761	811	596	596
q8	9069	1365	1110	1110
q9	5532	4238	4200	4200
q10	6824	1301	1031	1031
q11	421	254	226	226
q12	641	395	282	282
q13	17720	3659	3078	3078
q14	273	272	248	248
q15	q16	740	715	646	646
q17	1190	752	578	578
q18	7007	6212	6143	6143
q19	1106	952	571	571
q20	375	333	228	228
q21	5298	2259	2651	2259
q22	400	345	275	275
Total cold run time: 97239 ms
Total hot run time: 28679 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5962	5945	5915	5915
q2	739	740	623	623
q3	5690	5800	5446	5446
q4	1998	2131	1560	1560
q5	5734	5797	5742	5742
q6	238	185	140	140
q7	2202	1991	1884	1884
q8	3021	2667	2712	2667
q9	8120	7880	7538	7538
q10	3644	3624	3127	3127
q11	582	412	385	385
q12	671	664	514	514
q13	3261	3634	3038	3038
q14	292	286	263	263
q15	q16	668	677	624	624
q17	7676	7057	6915	6915
q18	12993	12214	12861	12214
q19	883	770	777	770
q20	2142	2134	1925	1925
q21	5574	4635	4709	4635
q22	492	482	410	410
Total cold run time: 72582 ms
Total hot run time: 66335 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 77.71% (265/341) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 154062 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77a128ed5c2b710bd19834e5201d136d438d4040, data reload: false

query5	4303	508	385	385
query6	432	218	197	197
query7	4798	432	233	233
query8	321	173	153	153
query9	8785	3202	3203	3202
query10	471	307	257	257
query11	5960	3857	3573	3573
query12	149	99	88	88
query13	1266	449	326	326
query14	6151	3957	3679	3679
query14_1	3397	3427	3395	3395
query15	208	209	180	180
query16	966	441	410	410
query17	883	666	545	545
query18	2413	427	319	319
query19	197	166	151	151
query20	84	86	89	86
query21	214	125	109	109
query22	13013	13038	12730	12730
query23	13478	12718	12500	12500
query23_1	12587	12431	12631	12431
query24	7105	840	472	472
query24_1	476	492	476	476
query25	510	400	336	336
query26	1253	255	137	137
query27	2784	433	275	275
query28	4452	2067	2036	2036
query29	1527	591	466	466
query30	285	225	181	181
query31	1002	933	807	807
query32	149	91	91	91
query33	525	293	235	235
query34	876	849	505	505
query35	736	751	637	637
query36	815	830	771	771
query37	126	92	87	87
query38	1779	1761	1663	1663
query39	791	790	756	756
query39_1	735	722	731	722
query40	223	113	110	110
query41	68	64	69	64
query42	89	88	89	88
query43	346	365	307	307
query44	1380	722	724	722
query45	184	181	166	166
query46	852	997	562	562
query47	2974	2947	2690	2690
query48	306	323	227	227
query49	601	397	311	311
query50	713	274	214	214
query51	10396	10761	10591	10591
query52	80	83	68	68
query53	187	213	155	155
query54	260	207	179	179
query55	76	70	65	65
query56	229	214	204	204
query57	1660	1623	1566	1566
query58	295	266	262	262
query59	2171	2216	2113	2113
query60	282	250	228	228
query61	149	149	147	147
query62	380	334	291	291
query63	189	158	171	158
query64	2610	921	759	759
query65	3362	3305	3250	3250
query66	1741	411	342	342
query67	23023	22624	22547	22547
query68	3211	1020	619	619
query69	384	284	256	256
query70	845	772	760	760
query71	307	241	211	211
query72	2758	2608	2315	2315
query73	514	532	293	293
query74	4608	4532	4327	4327
query75	2352	2286	1932	1932
query76	2058	1007	584	584
query77	343	397	269	269
query78	9712	9605	9176	9176
query79	893	858	525	525
query80	495	433	346	346
query81	512	320	271	271
query82	220	149	118	118
query83	203	198	184	184
query84	293	123	101	101
query85	773	433	380	380
query86	262	234	230	230
query87	1973	1941	1818	1818
query88	3684	2786	2753	2753
query89	311	290	246	246
query90	1986	174	167	167
query91	160	154	127	127
query92	105	94	90	90
query93	999	1026	601	601
query94	551	322	282	282
query95	569	324	362	324
query96	672	488	243	243
query97	2447	2448	2350	2350
query98	163	159	154	154
query99	759	759	677	677
Total cold run time: 235622 ms
Total hot run time: 154062 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.19 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77a128ed5c2b710bd19834e5201d136d438d4040, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.04	0.05
query3	0.26	0.09	0.09
query4	1.60	0.10	0.10
query5	0.23	0.25	0.24
query6	1.16	0.66	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.35	0.36
query10	0.57	0.56	0.56
query11	0.20	0.15	0.14
query12	0.19	0.15	0.15
query13	0.48	0.49	0.49
query14	1.03	1.01	1.01
query15	0.62	0.61	0.61
query16	0.34	0.35	0.34
query17	1.13	1.12	1.15
query18	0.20	0.18	0.19
query19	2.07	1.97	2.00
query20	0.02	0.01	0.02
query21	15.41	0.19	0.14
query22	5.00	0.05	0.05
query23	16.27	0.31	0.13
query24	1.06	0.25	0.20
query25	0.09	0.05	0.04
query26	0.76	0.21	0.16
query27	0.03	0.03	0.03
query28	3.67	0.82	0.37
query29	12.52	4.18	3.34
query30	0.30	0.14	0.10
query31	2.77	0.56	0.32
query32	3.23	0.60	0.50
query33	3.22	3.31	3.19
query34	15.41	4.01	3.39
query35	3.34	3.34	3.32
query36	0.56	0.46	0.43
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.04
query40	0.17	0.15	0.13
query41	0.08	0.03	0.03
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 94.87 s
Total hot run time: 24.19 s

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review recovery stopped: shared time budget or helper timeout exhausted
Workflow run: https://github.com/apache/doris/actions/runs/42194372865

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@yujun777 yujun777 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@924060929
924060929 merged commit f066153 into apache:master Nov 9, 2026
50 of 51 checks passed
@924060929
924060929 deleted the fix/sql-cache-replay-invalidate branch November 9, 2026 05:42
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Nov 12, 2026
### What problem does this PR solve?

Related PR: apache#63612

Problem Summary: With a one-entry external database object cache, constructing an MV plan that references two databases evicts a database object. The eviction was reported as a metadata change, invalidating the MV generation and restarting every successful build indefinitely. Separate automatic local-object retirement from explicit metadata invalidation while retaining typed engine cleanup and row-count fencing. Bound rewrite-cache construction to four attempts so sustained metadata changes fall back through the callers' existing optional-optimization exception handling.

### Release note

Avoid repeated query planning caused by external database object cache eviction during materialized view rewriting.

### Check List (For Author)

- Test: Unit Test - 192 focused FE tests passed with zero failures, errors or skips; FE build and integrated Checkstyle passed. The unfixed 4.1 reproduction had 6 failures among 11 tests.
- Behavior changed: Yes - automatic database object eviction no longer reports a semantic metadata change, and unstable optional MV cache construction terminates after four attempts.
- Does this need documentation: No
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Nov 12, 2026
### What problem does this PR solve?

Related PR: apache#63612

Problem Summary: With a one-entry external database object cache, constructing an MV plan that references two databases evicts a database object. The eviction was reported as a metadata change, invalidating the MV generation and restarting every successful build indefinitely. Separate automatic local-object retirement from explicit metadata invalidation while retaining typed engine cleanup and row-count fencing. Bound rewrite-cache construction to four attempts so sustained metadata changes fall back through the callers' existing optional-optimization exception handling.

### Release note

Avoid repeated query planning caused by external database object cache eviction during materialized view rewriting.

### Check List (For Author)

- Test: Unit Test - 192 focused FE tests passed with zero failures, errors or skips; FE build and integrated Checkstyle passed. The unfixed 4.1 reproduction had 6 failures among 11 tests.
- Behavior changed: Yes - automatic database object eviction no longer reports a semantic metadata change, and unstable optional MV cache construction terminates after four attempts.
- Does this need documentation: No
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Nov 13, 2026
### What problem does this PR solve?

Related PR: apache#63612

Problem Summary: The synchronous-MV cache regression warmed base-table queries before publishing rollups and expected their old SQL cache entries to survive. Rollup publication now notifies table metadata changes and correctly invalidates those entries. Share one check across the rename, insert and drop scenarios to verify cache misses after publication, then rerun each query and verify cache hits. Preserve the later rename, insert and drop assertions.

### Release note

None

### Check List (For Author)

- Test: 40 focused FE unit tests passed; regression framework and modified Groovy suite compiled; FE build with integrated Checkstyle passed. End-to-end regression was not run locally because this worktree has no BE binary.
- Behavior changed: No production behavior change; update regression expectations for metadata invalidation.
- Does this need documentation: No
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Nov 13, 2026
### What problem does this PR solve?

Related PR: apache#63612

Problem Summary: The synchronous-MV cache regression warmed base-table queries before publishing rollups and expected their old SQL cache entries to survive. Rollup publication now notifies table metadata changes and correctly invalidates those entries. Share one check across the rename, insert and drop scenarios to verify cache misses after publication, then rerun each query and verify cache hits. Preserve the later rename, insert and drop assertions.

### Release note

None

### Check List (For Author)

- Test: 40 focused FE unit tests passed; regression framework and modified Groovy suite compiled; FE build with integrated Checkstyle passed. End-to-end regression was not run locally because this worktree has no BE binary.
- Behavior changed: No production behavior change; update regression expectations for metadata invalidation.
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.