Skip to content

[feature](iceberg) Support required column additions with defaults - #70220

Open
hubgeter wants to merge 8 commits into
apache:masterfrom
hubgeter:analyze/doris-30020-master
Open

hubgeter wants to merge 8 commits into
apache:masterfrom
hubgeter:analyze/doris-30020-master

Conversation

@hubgeter

@hubgeter hubgeter commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: #65851, #69488, #69499

Problem Summary:

ALTER TABLE ... ADD COLUMN added INT NOT NULL DEFAULT 7 is rejected even on an Iceberg format-version 3 table. Nested additions also reject DEFAULT in their separate grammar and metadata validation.

Reuse the full column definition for nested paths and share SDK staging for top-level single/batch additions, STRUCT children, ARRAY element children and MAP value children. Preserve requiredness and set both initial and write defaults. Validate the loaded table's format version before committing; retain batch atomicity, path/position validation and statement-bound write-schema checks. NULL parents remain NULL, and explicit NULL in a supplied required field remains an error.

Normalize defaults to their typed metadata representation: trim valid Boolean spellings, exactly rescale decimals, canonicalize compact UUIDs, decode and quote strings according to SQL mode, and preserve SQL temporal precision and the existing nanosecond gap policy. Unqualified TIMESTAMPTZ values retain civil text across the generic connector boundary; Iceberg resolves them using the existing session timezone API, then applies the requested precision so equivalent explicit/session zones agree across DST carries. Compact fractional timestamps retain all six digits before zone binding. Validated compact spellings with or without T or a decimal point are normalized lexically before zone detection, retaining the original offset or named zone and declared precision. DATETIME defaults and final UTC TIMESTAMPTZ defaults must fit years 0001 through 9999 before staging, matching physical Parquet timestamp reads as well as historical default materialization. Year-zero physical timestamps would otherwise read as NULL even when the historical default is valid. Reject non-finite floating defaults before they can publish schema JSON that cannot reload. Iceberg SHOW CREATE, DESC and information_schema default presentation retain their current behavior.

Release note

Iceberg format-version 3 tables support adding top-level and nested NOT NULL fields with non-null literal defaults. Typed defaults preserve their SQL values through metadata serialization and reads/writes. Compact DATETIME and TIMESTAMPTZ defaults accept fractional seconds without a decimal point. Format-version 1 and 2 tables continue to reject non-null column defaults. Non-finite floating defaults are rejected before commit, as are DATETIME defaults or final UTC TIMESTAMPTZ defaults outside years 0001 through 9999.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Validation:

  • Final focused run-fe-ut.sh run: 93 tests passed (37 connector, 56 core/parser/command), with no failures or skips. HadoopCatalog tests commit and reload real metadata, including decimal/string/UUID/session-zone defaults, DST carries, negative epoch rounding, UTC bounds, rejection of year-zero values and failure atomicity.

  • Final build.sh --fe, Checkstyle and both connector architecture gates: passed (0 Checkstyle violations).

  • run-regression-test.sh --run -d external_table_p0/iceberg: all four suites passed in normal comparison: the new defaults suite and the three existing suites listed below. The new 146-line output was generated by the preset runner against the existing newer REST/MinIO fixture; Parquet/ORC old/new files, nested defaults, compact fractional timestamps with and without a decimal point, copied non-null nested defaults in physical new files, nanosecond DEFAULT versus CAST, explicit/implicit DST equality, UTC/date rejection, minimum/maximum civil and UTC timestamp reads/writes, SQL modes, string HEX values and failure atomicity were verified.

  • Existing REST suites iceberg_schema_change_ddl, test_iceberg_nested_schema_evolution_ddl and test_iceberg_schema_metadata_atomicity_matrix passed.

  • Local read/write validation uses the branch-built FE with an ASAN BE cache (BE code is unchanged). A full local BE build was blocked while cloning a required submodule.

  • Behavior changed:

    • No.
    • Yes. Required v3 additions and nested scalar defaults are supported; literal defaults preserve their typed values and invalid metadata defaults fail before commit.
  • Does this need documentation?

    • No.
    • Yes. Connector contract and examples are included in fe/fe-connector/docs/iceberg-add-column-defaults.md.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65851, apache#69488, apache#69499

Problem Summary: ADD COLUMN rejects required fields even when a format-version 3 table can supply a non-null initial and write default. Nested additions also reject defaults in the parser and metadata layer. Reuse full column-definition parsing for nested paths and stage optional or required additions through one SDK helper for top-level, batch, STRUCT, ARRAY element and MAP value additions. Check the loaded table format before committing defaults and preserve existing parent, collision, position and write-schema validation. Correct Boolean and SQL temporal literal conversion on the same default path. Keep the existing SHOW CREATE and catalog default presentation.

### Release note

Iceberg format-version 3 tables support adding top-level and nested NOT NULL fields with non-null defaults. Boolean 0/1 and SQL date/timestamp default literals are converted correctly. Older formats continue to reject non-null column defaults.

### Check List (For Author)

- Test: Targeted run-fe-ut.sh connector, parser and command suites passed (313 tests); additional typed default and real SDK schema tests passed (30 tests, including repeated coverage). Full build.sh --fe and architecture gates passed before the final typed conversion update. Manual SQL DDL positive and negative cases passed. Parquet/ORC regression suite added; local read/write execution is pending preparation of the native runtime.
- Behavior changed: Yes; required v3 additions and nested scalar defaults are supported, and invalid Boolean defaults fail instead of silently becoming false.
- Does this need documentation: Yes; the connector contract and examples are included under fe/fe-connector/docs/iceberg-add-column-defaults.md.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65851, apache#69488

Problem Summary: External ADD COLUMN accepts compact SQL dates and timestamps such as 20240102 and 20240102030405, but their raw numeric strings can be mistaken for connector epoch values. Extend the existing civil TIMESTAMP_NS default normalization to DATE and DATETIME with the same parser used by validation. Preserve fractional precision and keep TIMESTAMPTZ and OLAP behavior unchanged. Add focused normalization tests and top-level/nested regression cases.

### Release note

External DATE and DATETIME column defaults preserve the date/time represented by compact SQL literals.

### Check List (For Author)

- Test: run-fe-ut.sh passed ColumnDefinitionTest, AlterTableCommandTest and IcebergNestedSchemaEvolutionParserTest, including existing nanosecond normalization coverage. The previous full FE build passed; the final rebuild and Parquet/ORC regression are pending this normalization update.
- Behavior changed: Yes; civil external DDL defaults are normalized before connector conversion.
- Does this need documentation: Yes; the connector contract is included in this branch.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65851, apache#69488

Problem Summary: Persisted Iceberg defaults must preserve the instant of a TIMESTAMPTZ SQL literal and the fractional precision of its target type. Resolve zoned literals with the existing session-time-zone conversion helper and convert other parsed temporal literals through the typed literal bridge before passing their canonical value to connectors. Extend real SDK tests and the Parquet/ORC regression with zoned, compact and rounded defaults. Prepare local Hadoop test data directories and check NULL rejection independently of the validation layer that reports it.

### Release note

External temporal column defaults preserve compact SQL dates, target fractional precision and the DDL session's time zone for TIMESTAMPTZ values without an explicit offset.

### Check List (For Author)

- Test: run-fe-ut.sh passed 81 focused connector, parser, ALTER command and column-definition tests. Manual Parquet/ORC old/new file read-write and required-NULL rejection tests passed. A complete preliminary regression generated outputs successfully; final output generation and comparison await deployment of this last temporal update. Previous full FE builds and architecture gates passed; the final FE rebuild follows this commit.
- Behavior changed: Yes; external temporal defaults are canonicalized with their target type and session-zone semantics before connector conversion.
- Does this need documentation: Yes; the connector contract is updated in this branch.
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65851, apache#69488

Problem Summary: Record script-generated expected results for required top-level and nested column additions, including old/new Parquet and ORC files, NULL parents, typed and compact literals, session-zone defaults, fractional rounding, negative writes, older formats and batch atomicity. Clarify literal defaults and the optional local filesystem test fixture.

### Release note

None

### Check List (For Author)

- Test: run-regression-test.sh --run -d external_table_p0/iceberg -s test_iceberg_add_column_defaults -genOut passed, followed by a successful normal comparison. Final build.sh --fe and Checkstyle passed, as did 81 focused FE tests and both connector architecture gates. Local read/write testing used the branch-built FE and an ASAN BE cache; no BE code changed.
- Behavior changed: No; this commit adds generated test results and documentation for the feature in this branch.
- Does this need documentation: Yes; the connector contract is included and clarified.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

hubgeter commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28714 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2906bda0c6efae33e6506a60680499e0dc8f26cc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17590	5148	5033	5033
q2	2130	311	251	251
q3	10255	1405	835	835
q4	4680	477	351	351
q5	7437	800	526	526
q6	183	190	154	154
q7	776	797	623	623
q8	9247	1341	1041	1041
q9	5467	4204	4183	4183
q10	6815	1307	1048	1048
q11	424	241	228	228
q12	628	404	283	283
q13	18097	3669	3059	3059
q14	283	264	249	249
q15	q16	727	714	666	666
q17	1203	764	572	572
q18	7043	6200	6127	6127
q19	1252	957	579	579
q20	383	345	228	228
q21	5799	2497	2407	2407
q22	387	323	271	271
Total cold run time: 100806 ms
Total hot run time: 28714 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6014	5894	6083	5894
q2	749	601	598	598
q3	5671	5835	5271	5271
q4	2034	2171	1559	1559
q5	5800	5882	5642	5642
q6	235	191	142	142
q7	2213	2051	1795	1795
q8	2962	2625	2658	2625
q9	8044	7546	7442	7442
q10	3615	3588	3143	3143
q11	587	410	371	371
q12	660	681	511	511
q13	3313	3597	3080	3080
q14	288	299	265	265
q15	q16	657	673	611	611
q17	7654	7063	6893	6893
q18	12961	12231	12837	12231
q19	899	783	837	783
q20	2144	2188	1911	1911
q21	5534	4533	4655	4533
q22	494	463	422	422
Total cold run time: 72528 ms
Total hot run time: 65722 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151707 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2906bda0c6efae33e6506a60680499e0dc8f26cc, data reload: false

query5	4311	604	436	436
query6	436	226	187	187
query7	4825	443	229	229
query8	315	168	154	154
query9	8788	3176	3163	3163
query10	471	312	255	255
query11	5978	3911	3571	3571
query12	151	91	90	90
query13	1250	460	358	358
query14	6050	3846	3635	3635
query14_1	3404	3424	3406	3406
query15	198	205	180	180
query16	971	403	402	402
query17	848	630	512	512
query18	2412	410	300	300
query19	201	159	134	134
query20	86	85	81	81
query21	208	124	105	105
query22	12927	13040	12983	12983
query23	13486	12890	12662	12662
query23_1	12592	12547	12732	12547
query24	7384	813	463	463
query24_1	474	476	477	476
query25	542	425	361	361
query26	1254	263	144	144
query27	2763	441	274	274
query28	4671	2030	2004	2004
query29	1583	612	463	463
query30	304	222	193	193
query31	1006	923	820	820
query32	154	91	92	91
query33	534	314	248	248
query34	1009	850	502	502
query35	744	761	648	648
query36	844	830	755	755
query37	126	94	85	85
query38	1762	1737	1672	1672
query39	788	778	738	738
query39_1	738	728	746	728
query40	221	114	100	100
query41	72	71	69	69
query42	88	86	81	81
query43	352	360	318	318
query44	1323	700	717	700
query45	181	181	161	161
query46	843	972	573	573
query47	2973	2990	2607	2607
query48	314	323	233	233
query49	586	410	315	315
query50	693	284	230	230
query51	10742	10699	10436	10436
query52	82	82	71	71
query53	193	218	159	159
query54	246	239	188	188
query55	75	72	67	67
query56	251	236	226	226
query57	1655	1626	1529	1529
query58	292	274	270	270
query59	2169	2202	1985	1985
query60	307	264	242	242
query61	170	174	168	168
query62	394	353	290	290
query63	192	159	164	159
query64	2933	952	738	738
query65	3359	3264	3280	3264
query66	1788	416	303	303
query67	20124	20293	20128	20128
query68	2921	1026	621	621
query69	386	295	258	258
query70	862	772	766	766
query71	310	238	210	210
query72	2748	2615	2260	2260
query73	508	539	290	290
query74	4619	4514	4338	4338
query75	2342	2286	1957	1957
query76	2367	1018	583	583
query77	324	363	273	273
query78	9713	9768	9221	9221
query79	992	852	539	539
query80	1203	414	359	359
query81	591	309	268	268
query82	628	129	104	104
query83	301	194	180	180
query84	315	125	101	101
query85	868	459	365	365
query86	393	239	233	233
query87	1989	1945	1811	1811
query88	3838	2780	2742	2742
query89	329	281	249	249
query90	1869	172	169	169
query91	166	157	136	136
query92	103	82	88	82
query93	1014	993	582	582
query94	631	318	270	270
query95	588	324	308	308
query96	691	496	243	243
query97	2452	2445	2313	2313
query98	160	153	151	151
query99	785	749	640	640
Total cold run time: 235835 ms
Total hot run time: 151707 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.45 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2906bda0c6efae33e6506a60680499e0dc8f26cc, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.04	0.04
query3	0.26	0.08	0.09
query4	1.60	0.10	0.10
query5	0.26	0.24	0.23
query6	1.17	0.68	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.35	0.34
query10	0.58	0.57	0.58
query11	0.19	0.14	0.14
query12	0.17	0.14	0.15
query13	0.48	0.49	0.49
query14	1.03	1.02	1.01
query15	0.60	0.60	0.61
query16	0.35	0.34	0.36
query17	1.14	1.13	1.10
query18	0.22	0.20	0.20
query19	2.08	2.03	2.05
query20	0.02	0.01	0.01
query21	15.50	0.19	0.13
query22	4.87	0.06	0.05
query23	16.25	0.30	0.13
query24	2.81	0.54	0.49
query25	0.10	0.06	0.07
query26	1.61	0.21	0.15
query27	0.05	0.05	0.04
query28	3.69	0.79	0.36
query29	12.48	4.14	3.24
query30	0.27	0.14	0.11
query31	2.77	0.56	0.33
query32	3.23	0.60	0.50
query33	3.12	3.18	3.19
query34	15.28	3.99	3.41
query35	3.36	3.34	3.34
query36	0.55	0.50	0.44
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 97.2 s
Total hot run time: 24.45 s

…load

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65851, apache#69488, apache#69499

Problem Summary: Required and nested ADD COLUMN defaults must retain their SQL values when converted to Iceberg metadata. Normalize decimal scale, valid Boolean whitespace, compact UUIDs and escaped strings; preserve nanosecond gap semantics and temporal precision while keeping unqualified zoned values compatible with existing connector inputs. Resolve Iceberg zoned defaults through the session timezone and reject non-finite floating defaults before metadata publication. Verify real Hadoop metadata commits/reloads, extend read/write regression coverage and update both existing v2 rejection assertions.

### Release note

Typed literal defaults for required and nested Iceberg column additions preserve their values through metadata serialization and reads/writes. Non-finite floating defaults fail before commit.

### Check List (For Author)

- Test: 89 focused FE tests passed; full build.sh --fe, Checkstyle and both connector architecture gates passed. The preset regression runner generated the new expected output and then successfully compared all four affected external suites against isolated REST/MinIO services. Local read/write validation used the branch-built FE and an ASAN BE cache; BE code is unchanged.
- Behavior changed: Yes; correct typed default conversion and prevent publication of unreadable floating defaults. Existing SHOW CREATE default presentation remains unchanged.
- Does this need documentation: Yes; the connector contract and fixture requirements are updated in this commit.
@hubgeter

hubgeter commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28466 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a5332d71e7c3174ab4369ad0aa8153368ae06795, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17605	5153	5035	5035
q2	2127	311	271	271
q3	10229	1378	817	817
q4	4679	463	336	336
q5	7428	817	523	523
q6	183	187	153	153
q7	762	809	613	613
q8	9259	1326	1064	1064
q9	5483	4176	4188	4176
q10	6825	1288	1028	1028
q11	426	246	241	241
q12	638	408	290	290
q13	18110	3688	3061	3061
q14	286	267	249	249
q15	q16	743	710	663	663
q17	1197	747	584	584
q18	7015	6192	6113	6113
q19	1101	955	619	619
q20	378	334	222	222
q21	5410	2155	2284	2155
q22	349	297	253	253
Total cold run time: 100233 ms
Total hot run time: 28466 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5620	5599	5576	5576
q2	693	565	541	541
q3	5218	5369	4900	4900
q4	1994	2169	1554	1554
q5	5252	5130	5570	5130
q6	303	209	155	155
q7	2358	2123	1925	1925
q8	3110	2875	2728	2728
q9	8125	8152	7985	7985
q10	3782	3698	3234	3234
q11	631	445	415	415
q12	676	669	515	515
q13	3581	3926	3207	3207
q14	303	306	298	298
q15	q16	694	738	700	700
q17	8336	7567	6992	6992
q18	12965	12152	12904	12152
q19	937	789	787	787
q20	2114	2135	1902	1902
q21	5649	4716	4686	4686
q22	540	440	402	402
Total cold run time: 72881 ms
Total hot run time: 65784 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151615 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a5332d71e7c3174ab4369ad0aa8153368ae06795, data reload: false

query5	4312	605	434	434
query6	427	218	191	191
query7	4802	442	237	237
query8	328	168	157	157
query9	8757	3178	3193	3178
query10	469	326	265	265
query11	6029	3865	3579	3579
query12	145	90	86	86
query13	1281	450	353	353
query14	6198	3943	3678	3678
query14_1	3437	3419	3415	3415
query15	219	209	187	187
query16	943	427	404	404
query17	863	652	541	541
query18	2408	436	313	313
query19	197	175	143	143
query20	92	85	82	82
query21	223	128	111	111
query22	12893	13023	12789	12789
query23	13524	12831	12322	12322
query23_1	12641	12577	12532	12532
query24	7357	831	480	480
query24_1	483	489	493	489
query25	533	431	365	365
query26	1258	259	140	140
query27	2769	446	285	285
query28	4606	2035	2060	2035
query29	1590	609	464	464
query30	305	228	193	193
query31	1006	935	830	830
query32	140	96	92	92
query33	528	300	249	249
query34	887	903	506	506
query35	728	752	654	654
query36	814	841	752	752
query37	126	101	82	82
query38	1761	1748	1661	1661
query39	802	809	740	740
query39_1	720	727	739	727
query40	217	117	98	98
query41	70	71	66	66
query42	87	85	87	85
query43	352	364	319	319
query44	1318	700	726	700
query45	183	185	170	170
query46	835	1002	586	586
query47	2927	2832	2620	2620
query48	338	306	213	213
query49	574	401	285	285
query50	684	265	214	214
query51	10447	10648	10566	10566
query52	78	83	69	69
query53	182	212	154	154
query54	251	190	192	190
query55	77	66	61	61
query56	222	212	209	209
query57	1661	1654	1588	1588
query58	280	261	249	249
query59	2143	2253	2005	2005
query60	289	247	226	226
query61	167	148	147	147
query62	396	338	284	284
query63	191	160	159	159
query64	2749	987	765	765
query65	3344	3282	3263	3263
query66	1786	426	337	337
query67	19960	20194	20293	20194
query68	3249	1004	631	631
query69	392	297	249	249
query70	834	792	760	760
query71	302	235	211	211
query72	2859	2553	2282	2282
query73	539	531	300	300
query74	4690	4498	4355	4355
query75	2345	2292	1912	1912
query76	2407	1013	618	618
query77	345	370	285	285
query78	9700	9669	9157	9157
query79	1021	915	541	541
query80	789	429	346	346
query81	575	318	278	278
query82	576	141	106	106
query83	301	202	186	186
query84	312	131	103	103
query85	929	448	385	385
query86	357	233	228	228
query87	2039	1941	1811	1811
query88	3775	2813	2785	2785
query89	338	287	247	247
query90	1764	167	169	167
query91	169	153	132	132
query92	98	89	93	89
query93	1013	1016	582	582
query94	509	317	264	264
query95	569	373	296	296
query96	677	477	238	238
query97	2465	2435	2308	2308
query98	160	151	145	145
query99	756	761	651	651
Total cold run time: 234979 ms
Total hot run time: 151615 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a5332d71e7c3174ab4369ad0aa8153368ae06795, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.05	0.05
query3	0.25	0.08	0.10
query4	1.62	0.08	0.10
query5	0.24	0.24	0.25
query6	1.18	0.67	0.68
query7	0.04	0.01	0.00
query8	0.05	0.04	0.03
query9	0.39	0.35	0.35
query10	0.59	0.62	0.57
query11	0.19	0.14	0.13
query12	0.18	0.14	0.15
query13	0.48	0.48	0.48
query14	1.04	1.02	1.00
query15	0.63	0.60	0.62
query16	0.31	0.36	0.35
query17	1.14	1.13	1.16
query18	0.19	0.19	0.18
query19	2.04	1.97	1.95
query20	0.01	0.01	0.01
query21	15.48	0.18	0.13
query22	4.94	0.05	0.05
query23	16.30	0.31	0.13
query24	3.21	0.43	0.39
query25	0.10	0.07	0.06
query26	0.88	0.21	0.16
query27	0.05	0.04	0.06
query28	2.72	0.76	0.35
query29	12.55	4.13	3.25
query30	0.30	0.13	0.09
query31	2.77	0.55	0.33
query32	3.22	0.59	0.50
query33	3.19	3.20	3.19
query34	15.24	3.99	3.39
query35	3.35	3.33	3.34
query36	0.54	0.42	0.43
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 96.03 s
Total hot run time: 24.18 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 95.83% (23/24) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65851, apache#69488

Problem Summary: Equivalent explicit and session-zone TIMESTAMPTZ defaults can differ by one hour when fractional rounding crosses a DST overlap. A valid civil timestamp can also resolve outside the UTC range supported by Doris and become an unreadable default after schema commit. Preserve civil precision until the existing Iceberg session boundary, round the resolved instant at the original declared precision, and reject unsupported final UTC years before staging. Share the conversion across single, batch and nested additions and retain unparameterized SPI epoch microseconds.

### Release note

Iceberg zoned column defaults consistently interpret explicit and implicit time zones across DST rounding and reject unsupported UTC values before committing schema changes.

### Check List (For Author)

- Test: 91 focused FE tests passed; full build.sh --fe, Checkstyle and both connector architecture gates passed. The preset regression runner generated expected results and passed the four affected external suites, including old/new files, explicit/session CET equality, UTC upper/lower rejection and batch atomicity. Local native validation uses an existing ASAN BE cache; C++ code is unchanged.
- Behavior changed: Yes; zoned ADD COLUMN defaults resolve time zones before target rounding and validate the final UTC range. Existing default presentation is preserved.
- Does this need documentation: Yes; the connector contract describes the precision and UTC-range policy.
@hubgeter

hubgeter commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28772 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c72b3357a7c4135c5a12ecd7855238ec68ae4cd, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17562	5058	4981	4981
q2	2122	315	270	270
q3	10212	1355	796	796
q4	4678	465	335	335
q5	7426	787	542	542
q6	175	182	151	151
q7	747	790	616	616
q8	9249	1299	1039	1039
q9	5396	4159	4161	4159
q10	6811	1276	1017	1017
q11	426	254	228	228
q12	637	406	297	297
q13	18075	3612	3016	3016
q14	284	262	242	242
q15	q16	724	707	674	674
q17	1201	760	591	591
q18	6953	6132	6082	6082
q19	1102	954	603	603
q20	395	338	224	224
q21	4711	2708	2641	2641
q22	389	343	268	268
Total cold run time: 99275 ms
Total hot run time: 28772 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5915	5872	5897	5872
q2	717	614	566	566
q3	5577	5687	5284	5284
q4	2030	2161	1609	1609
q5	6004	5720	5696	5696
q6	252	189	148	148
q7	2232	2015	1821	1821
q8	3022	2478	2491	2478
q9	7485	7479	7388	7388
q10	3582	3554	3149	3149
q11	573	404	378	378
q12	679	664	505	505
q13	3306	3590	3021	3021
q14	290	289	261	261
q15	q16	668	680	613	613
q17	7686	6988	6859	6859
q18	12976	12157	12812	12157
q19	940	831	830	830
q20	2143	2137	1908	1908
q21	5661	4762	4698	4698
q22	506	433	416	416
Total cold run time: 72244 ms
Total hot run time: 65657 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151587 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c72b3357a7c4135c5a12ecd7855238ec68ae4cd, data reload: false

query5	4313	600	451	451
query6	442	208	201	201
query7	4809	445	230	230
query8	326	167	144	144
query9	8812	3188	3194	3188
query10	473	321	262	262
query11	5935	3861	3568	3568
query12	147	89	89	89
query13	1258	456	351	351
query14	6114	3912	3666	3666
query14_1	3436	3394	3441	3394
query15	199	202	184	184
query16	949	424	432	424
query17	838	650	547	547
query18	2419	424	310	310
query19	205	179	145	145
query20	86	87	80	80
query21	210	132	110	110
query22	13024	13051	13015	13015
query23	13439	12882	12409	12409
query23_1	12707	12929	12470	12470
query24	7355	857	473	473
query24_1	482	495	488	488
query25	540	426	356	356
query26	1282	255	143	143
query27	2762	449	280	280
query28	4599	2058	2052	2052
query29	1592	604	456	456
query30	304	225	183	183
query31	1014	939	804	804
query32	147	97	93	93
query33	519	311	249	249
query34	888	859	500	500
query35	723	747	651	651
query36	815	828	735	735
query37	125	105	83	83
query38	1777	1708	1672	1672
query39	783	799	735	735
query39_1	730	725	754	725
query40	228	117	102	102
query41	72	69	68	68
query42	90	84	84	84
query43	345	356	307	307
query44	1372	710	703	703
query45	180	179	168	168
query46	841	984	578	578
query47	2860	2834	2840	2834
query48	312	320	230	230
query49	623	397	283	283
query50	700	284	213	213
query51	10510	10505	10475	10475
query52	78	80	70	70
query53	192	225	151	151
query54	246	212	183	183
query55	79	77	67	67
query56	246	205	215	205
query57	1647	1614	1595	1595
query58	288	257	254	254
query59	2174	2147	2020	2020
query60	278	240	218	218
query61	148	144	139	139
query62	389	335	282	282
query63	192	162	158	158
query64	2746	962	778	778
query65	3349	3340	3263	3263
query66	1801	417	302	302
query67	20183	19966	19884	19884
query68	3013	1041	619	619
query69	380	287	254	254
query70	869	791	770	770
query71	301	240	220	220
query72	2674	2556	2314	2314
query73	516	536	294	294
query74	4626	4530	4398	4398
query75	2331	2284	1916	1916
query76	2317	1044	606	606
query77	347	372	261	261
query78	9713	9657	9197	9197
query79	1003	861	540	540
query80	1225	429	357	357
query81	579	322	265	265
query82	593	129	108	108
query83	301	204	185	185
query84	306	131	110	110
query85	891	424	372	372
query86	395	238	227	227
query87	1979	1947	1829	1829
query88	3862	2784	2763	2763
query89	331	276	241	241
query90	1915	173	170	170
query91	162	152	131	131
query92	98	85	89	85
query93	1096	1032	578	578
query94	646	316	265	265
query95	566	333	295	295
query96	691	504	246	246
query97	2436	2437	2322	2322
query98	159	154	144	144
query99	777	753	648	648
Total cold run time: 235642 ms
Total hot run time: 151587 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7c72b3357a7c4135c5a12ecd7855238ec68ae4cd, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.04	0.06
query3	0.26	0.08	0.09
query4	1.60	0.09	0.09
query5	0.24	0.24	0.25
query6	1.17	0.68	0.68
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.34	0.35
query10	0.57	0.58	0.61
query11	0.20	0.13	0.14
query12	0.18	0.14	0.14
query13	0.50	0.49	0.50
query14	1.01	1.00	1.01
query15	0.64	0.60	0.60
query16	0.34	0.35	0.35
query17	1.14	1.14	1.11
query18	0.21	0.19	0.20
query19	2.07	2.00	1.94
query20	0.02	0.01	0.01
query21	15.51	0.20	0.13
query22	4.94	0.05	0.05
query23	16.27	0.33	0.14
query24	3.11	0.38	0.36
query25	0.13	0.07	0.05
query26	0.97	0.21	0.16
query27	0.04	0.06	0.06
query28	3.61	0.83	0.36
query29	12.47	4.09	3.27
query30	0.29	0.12	0.09
query31	2.77	0.58	0.34
query32	3.23	0.60	0.50
query33	3.19	3.22	3.25
query34	15.32	4.05	3.41
query35	3.35	3.35	3.34
query36	0.57	0.44	0.44
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.04 s
Total hot run time: 24.33 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 95.83% (23/24) 🎉
Increment coverage report
Complete coverage report

…bounds

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#70220

Problem Summary: Compact TIMESTAMPTZ defaults such as 20240102030405.123456 lose their fractional seconds when civil precision is inferred as zero. Parse unqualified civil defaults explicitly at microsecond precision before session-zone binding and target rounding. Zone conversion or rounding can also produce year-zero dates that schema metadata accepts but physical Parquet timestamp reads turn into NULL. Validate DATETIME and final UTC TIMESTAMPTZ values against the readable 0001-9999 range before staging single, batch or nested additions.

### Release note

Iceberg column defaults preserve compact timestamp fractions and reject DATETIME and final UTC TIMESTAMPTZ values outside years 0001 through 9999 before committing schema changes.

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - 92 focused FE tests passed; full FE build and Checkstyle passed.
    - Generated defaults-suite output and normal comparison of four Iceberg suites passed, including Parquet/ORC old/new files and minimum/maximum civil and UTC timestamp defaults.
    - Both connector architecture gates passed. Read/write verification uses the branch-built FE and an ASAN BE cache; a full local BE build remains blocked by a required submodule clone.
- Behavior changed: Yes; preserve compact fractions and reject unsupported final UTC dates before metadata publication.
- Does this need documentation: Yes; update the included connector contract.
@hubgeter

hubgeter commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28769 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2065cc91fe82e4bf3d01408251fd9079f3e06eea, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17583	5127	5036	5036
q2	2166	312	268	268
q3	10219	1401	845	845
q4	4681	470	334	334
q5	7428	806	536	536
q6	187	190	160	160
q7	745	808	601	601
q8	9245	1332	1009	1009
q9	5663	4214	4170	4170
q10	6840	1301	1017	1017
q11	430	252	225	225
q12	630	410	298	298
q13	18105	3676	3084	3084
q14	274	277	251	251
q15	q16	745	734	679	679
q17	1201	762	582	582
q18	6931	6164	6119	6119
q19	1108	976	620	620
q20	393	346	236	236
q21	5445	2432	2439	2432
q22	370	344	267	267
Total cold run time: 100389 ms
Total hot run time: 28769 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5925	5950	6111	5950
q2	747	614	572	572
q3	5651	5773	5384	5384
q4	2016	2156	1562	1562
q5	5702	5850	5718	5718
q6	243	188	144	144
q7	2261	2049	1789	1789
q8	3019	2658	2681	2658
q9	8075	7647	7558	7558
q10	3666	3551	3139	3139
q11	573	400	372	372
q12	690	687	507	507
q13	3274	3579	3014	3014
q14	292	279	248	248
q15	q16	658	680	610	610
q17	7839	7064	6837	6837
q18	12926	12238	12932	12238
q19	906	772	775	772
q20	2133	2149	1905	1905
q21	5612	4584	4714	4584
q22	509	463	410	410
Total cold run time: 72717 ms
Total hot run time: 65971 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151480 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2065cc91fe82e4bf3d01408251fd9079f3e06eea, data reload: false

query5	4313	596	450	450
query6	431	213	189	189
query7	4803	460	228	228
query8	317	168	152	152
query9	8775	3216	3228	3216
query10	461	317	257	257
query11	6030	3889	3594	3594
query12	159	91	88	88
query13	1244	484	346	346
query14	6144	3942	3635	3635
query14_1	3442	3430	3412	3412
query15	203	214	184	184
query16	962	436	410	410
query17	865	659	546	546
query18	2420	453	298	298
query19	189	168	133	133
query20	87	82	87	82
query21	219	133	108	108
query22	13059	12949	12897	12897
query23	13464	12816	12283	12283
query23_1	12574	12573	12404	12404
query24	7349	846	468	468
query24_1	484	477	471	471
query25	509	399	347	347
query26	1245	254	138	138
query27	2780	446	279	279
query28	4624	2055	2045	2045
query29	1526	588	451	451
query30	300	223	189	189
query31	1000	937	804	804
query32	132	92	93	92
query33	525	293	237	237
query34	913	862	525	525
query35	736	759	641	641
query36	845	805	713	713
query37	123	93	83	83
query38	1773	1749	1678	1678
query39	782	781	736	736
query39_1	736	732	737	732
query40	217	134	95	95
query41	66	65	67	65
query42	87	84	83	83
query43	370	359	314	314
query44	1338	721	715	715
query45	186	182	166	166
query46	823	966	587	587
query47	2939	2888	2758	2758
query48	297	305	226	226
query49	586	406	308	308
query50	702	288	221	221
query51	10610	10832	10392	10392
query52	83	84	88	84
query53	195	220	166	166
query54	250	211	216	211
query55	77	69	68	68
query56	253	222	243	222
query57	1751	1646	1564	1564
query58	288	277	263	263
query59	2139	2194	1989	1989
query60	297	262	245	245
query61	174	163	171	163
query62	401	347	293	293
query63	192	166	169	166
query64	2864	1052	895	895
query65	3318	3301	3335	3301
query66	1835	415	299	299
query67	20058	20419	19868	19868
query68	3432	1020	627	627
query69	406	292	266	266
query70	868	774	775	774
query71	302	236	210	210
query72	2861	2578	2309	2309
query73	530	532	304	304
query74	4625	4520	4353	4353
query75	2338	2291	1919	1919
query76	2412	1029	602	602
query77	353	376	283	283
query78	9744	9837	9182	9182
query79	991	924	550	550
query80	794	432	349	349
query81	578	330	285	285
query82	581	150	116	116
query83	277	200	182	182
query84	305	131	101	101
query85	996	417	376	376
query86	361	240	243	240
query87	2001	1947	1792	1792
query88	3834	2800	2785	2785
query89	330	284	239	239
query90	1727	179	176	176
query91	166	177	135	135
query92	102	97	89	89
query93	1027	1027	589	589
query94	511	299	262	262
query95	582	400	308	308
query96	700	492	243	243
query97	2472	2421	2317	2317
query98	167	151	146	146
query99	773	774	670	670
Total cold run time: 235787 ms
Total hot run time: 151480 ms

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#70220

Problem Summary: The existing DEFAULT validator accepts compact fractional timestamps without a decimal point, including forms with or without T and explicit offsets or named zones. External DATETIME and TIMESTAMPTZ normalization reparses the raw spelling with a formatter that requires the decimal point, rejecting valid ADD COLUMN statements. Insert the fractional separator lexically before timezone detection while retaining all validated digits, the original zone and declared precision. Keep existing zone-first rounding and timestamp range checks. Add equivalent-spelling unit controls and persistent read/write cases across single, batch and nested additions; copy non-null nested defaults into new physical files to verify their values survive.

### Release note

External DATETIME and TIMESTAMPTZ column defaults accept validated compact fractional timestamps without a decimal point.

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - 93 focused FE tests passed; full FE build and Checkstyle passed.
    - Generated defaults-suite output and normal comparison of four Iceberg suites passed, including Parquet/ORC old/new files and non-null nested defaults copied into physical files.
    - Both connector architecture gates passed. Read/write checks use the branch-built FE and an ASAN BE cache; a full local BE build remains blocked by a required submodule clone.
- Behavior changed: Yes; retain accepted compact timestamp DEFAULT spellings and their precision/timezone semantics.
- Does this need documentation: Yes; update the included connector contract.
@hubgeter

hubgeter commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2065cc91fe82e4bf3d01408251fd9079f3e06eea, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.11	0.10
query5	0.24	0.24	0.23
query6	1.16	0.68	0.67
query7	0.04	0.02	0.01
query8	0.05	0.04	0.04
query9	0.40	0.36	0.35
query10	0.59	0.60	0.58
query11	0.20	0.14	0.16
query12	0.19	0.14	0.15
query13	0.48	0.51	0.50
query14	1.02	1.00	1.01
query15	0.64	0.61	0.60
query16	0.36	0.36	0.37
query17	1.12	1.19	1.14
query18	0.20	0.18	0.20
query19	2.14	1.95	1.99
query20	0.02	0.02	0.01
query21	15.49	0.18	0.14
query22	4.91	0.05	0.06
query23	16.25	0.32	0.12
query24	2.88	0.57	0.62
query25	0.12	0.05	0.06
query26	1.61	0.22	0.15
query27	0.04	0.05	0.03
query28	3.38	0.70	0.37
query29	12.47	4.11	3.22
query30	0.29	0.14	0.12
query31	2.77	0.56	0.33
query32	3.22	0.60	0.50
query33	3.24	3.21	3.28
query34	15.45	4.01	3.41
query35	3.36	3.36	3.32
query36	0.56	0.44	0.45
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.37 s
Total hot run time: 24.52 s

@hubgeter

hubgeter commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70220
commit: 431539613dc21f83e330776d146c3345ce471108
base: 380912cc7344e92375c3de58ec37c99602cad013
reviewed_at: 2026-11-05T05:22+00:00
reviewer: hubgeter
model: gpt-6.1-sol
effort: max
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java:605-606 — Generic DATETIMEV2 defaults use declared precision. Paimon 2.1 supplied-NULL default fallback can differ from its former raw-text flooring; ordinary Doris omitted/DEFAULT writes already rounded. This declared normalization policy is detailed in the local reports.
  • regression-test/suites/external_table_p0/iceberg/test_iceberg_add_column_defaults.groovy:193-194 and regression-test/suites/external_table_p0/iceberg/test_iceberg_add_column_defaults.groovy:253-254 — New physical nested controls copy complete materialized STRUCTs; they do not establish partial-STRUCT missing-child binding. Nested NULL-literal controls refuse in FE casts rather than exercising the BE nested-null validator branch.
  • be/src/core/data_type_serde/data_type_timestamptz_serde.cpp:548-551 — The existing ORC restriction on pre-epoch fractions in [-0.999, 0) seconds remains; its error directs callers to Parquet. This feature does not widen that writer value domain.
  • This was read-only static review; no reviewer builds, tests or cluster operations ran. Author evidence covers 93 focused FE tests, a full FE build and four regression comparisons. The ASAN BE cache has no verified exact-source build receipt; local native build was ORC-download blocked. Current-head CI remains separate.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28897 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 431539613dc21f83e330776d146c3345ce471108, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17571	5153	5056	5056
q2	2101	337	260	260
q3	10227	1431	833	833
q4	4681	465	348	348
q5	7425	794	526	526
q6	181	182	152	152
q7	766	802	593	593
q8	9255	1320	1113	1113
q9	5665	4242	4198	4198
q10	6827	1314	1057	1057
q11	432	241	228	228
q12	646	405	288	288
q13	18093	3918	3036	3036
q14	280	274	248	248
q15	q16	716	712	668	668
q17	1187	745	571	571
q18	7017	6226	6150	6150
q19	1255	958	627	627
q20	404	347	228	228
q21	5893	2422	2471	2422
q22	393	314	295	295
Total cold run time: 101015 ms
Total hot run time: 28897 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5969	5930	6146	5930
q2	790	590	603	590
q3	5681	5900	5306	5306
q4	2067	2152	1578	1578
q5	5785	5907	5588	5588
q6	244	185	144	144
q7	2235	2057	1786	1786
q8	2995	2789	2705	2705
q9	8070	7558	7431	7431
q10	3648	3586	3113	3113
q11	584	405	383	383
q12	669	671	514	514
q13	3306	3667	3041	3041
q14	281	284	269	269
q15	q16	680	703	605	605
q17	7713	7079	7010	7010
q18	12940	12184	12841	12184
q19	913	782	777	777
q20	2165	2132	1931	1931
q21	5594	4641	4710	4641
q22	476	460	413	413
Total cold run time: 72805 ms
Total hot run time: 65939 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152234 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 431539613dc21f83e330776d146c3345ce471108, data reload: false

query5	4307	603	454	454
query6	428	207	193	193
query7	4797	438	232	232
query8	316	174	153	153
query9	8785	3247	3234	3234
query10	481	304	253	253
query11	5927	3853	3588	3588
query12	149	91	86	86
query13	1273	452	331	331
query14	6164	3972	3742	3742
query14_1	3442	3428	3449	3428
query15	213	205	182	182
query16	950	437	426	426
query17	870	653	538	538
query18	2414	420	318	318
query19	191	164	132	132
query20	90	87	83	83
query21	214	127	107	107
query22	13126	13029	12855	12855
query23	13510	12749	12559	12559
query23_1	12613	12556	12566	12556
query24	7355	823	488	488
query24_1	494	495	483	483
query25	548	427	362	362
query26	1265	263	139	139
query27	2766	439	284	284
query28	4625	2064	2053	2053
query29	1565	609	461	461
query30	297	224	185	185
query31	1013	934	823	823
query32	146	98	93	93
query33	539	330	248	248
query34	912	866	519	519
query35	748	745	651	651
query36	818	800	721	721
query37	129	99	88	88
query38	1778	1730	1664	1664
query39	792	797	753	753
query39_1	744	733	731	731
query40	223	119	105	105
query41	73	69	67	67
query42	85	82	84	82
query43	349	361	314	314
query44	1321	708	723	708
query45	182	188	163	163
query46	843	977	571	571
query47	2899	3000	2943	2943
query48	298	306	225	225
query49	564	398	297	297
query50	676	267	215	215
query51	10581	10601	10499	10499
query52	83	78	72	72
query53	197	208	151	151
query54	231	200	181	181
query55	78	65	60	60
query56	242	226	216	216
query57	1675	1631	1569	1569
query58	282	264	262	262
query59	2107	2177	1987	1987
query60	279	249	223	223
query61	150	145	150	145
query62	390	338	287	287
query63	190	160	165	160
query64	2737	991	780	780
query65	3316	3279	3260	3260
query66	1792	412	297	297
query67	20267	20477	19965	19965
query68	3338	996	623	623
query69	432	295	253	253
query70	850	809	810	809
query71	312	234	204	204
query72	2864	2684	2386	2386
query73	529	530	305	305
query74	4653	4551	4361	4361
query75	2401	2286	1926	1926
query76	2394	1030	595	595
query77	337	373	268	268
query78	9836	9759	9136	9136
query79	990	848	534	534
query80	1142	428	343	343
query81	586	331	274	274
query82	572	133	108	108
query83	325	198	182	182
query84	313	123	105	105
query85	884	428	372	372
query86	366	239	236	236
query87	1983	1978	1826	1826
query88	3764	2808	2787	2787
query89	339	281	246	246
query90	1822	176	171	171
query91	168	158	128	128
query92	100	88	91	88
query93	1018	967	582	582
query94	548	302	264	264
query95	566	386	308	308
query96	681	496	245	245
query97	2425	2423	2359	2359
query98	170	154	159	154
query99	777	759	653	653
Total cold run time: 236168 ms
Total hot run time: 152234 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 431539613dc21f83e330776d146c3345ce471108, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.09	0.08
query4	1.61	0.10	0.10
query5	0.24	0.25	0.25
query6	1.19	0.69	0.68
query7	0.03	0.00	0.00
query8	0.06	0.04	0.04
query9	0.40	0.36	0.35
query10	0.58	0.59	0.61
query11	0.20	0.15	0.15
query12	0.19	0.15	0.15
query13	0.48	0.49	0.48
query14	1.02	1.01	1.00
query15	0.63	0.61	0.60
query16	0.36	0.35	0.35
query17	1.13	1.08	1.10
query18	0.21	0.18	0.19
query19	2.06	1.99	2.02
query20	0.02	0.02	0.01
query21	15.49	0.19	0.13
query22	4.90	0.06	0.05
query23	16.28	0.31	0.13
query24	2.61	0.44	0.42
query25	0.14	0.05	0.06
query26	1.41	0.21	0.16
query27	0.05	0.05	0.04
query28	3.30	0.74	0.36
query29	12.49	4.12	3.23
query30	0.28	0.14	0.10
query31	2.78	0.56	0.33
query32	3.23	0.59	0.52
query33	3.13	3.30	3.18
query34	15.40	3.94	3.37
query35	3.30	3.34	3.34
query36	0.55	0.46	0.43
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 96.63 s
Total hot run time: 24.35 s

@hubgeter
hubgeter marked this pull request as ready for review November 9, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants