Skip to content

[Test](iceberg-writer) Add iceberg partition transform unit tests and fix some issues. - #36889

Merged
morningman merged 1 commit into
apache:masterfrom
kaka11chen:add_iceberg_partition_transformers_unit_tests
Jul 3, 2024
Merged

morningman merged 1 commit into
apache:masterfrom
kaka11chen:add_iceberg_partition_transformers_unit_tests

Conversation

@kaka11chen

Copy link
Copy Markdown
Contributor

Proposed changes

  • Add iceberg partition transform unit tests.
  • Change ColumnWithTypeAndName apply(Block& block, int column_pos) to ColumnWithTypeAndName apply(const Block& block, int column_pos).
  • Fix and change string truncate partition transform issue.
  • Fix bucket partition transform calculation error.
  • Fix year/month partition transform calculation error due to leap year issue.

@doris-robot

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@kaka11chen

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 40192 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3feb3805c3a72491c797b33f4ceb66e8e05694af, data reload: false

------ Round 1 ----------------------------------
q1	17609	4555	4282	4282
q2	2019	197	197	197
q3	10459	1240	1180	1180
q4	10184	771	783	771
q5	7509	2653	2614	2614
q6	215	135	139	135
q7	953	604	595	595
q8	9234	2120	2097	2097
q9	9212	6518	6490	6490
q10	9090	3809	3729	3729
q11	462	236	230	230
q12	477	237	243	237
q13	18862	2956	3017	2956
q14	277	223	219	219
q15	533	479	467	467
q16	516	382	386	382
q17	990	696	757	696
q18	8059	7629	7591	7591
q19	1674	1455	1417	1417
q20	663	315	324	315
q21	4981	3866	3263	3263
q22	389	334	329	329
Total cold run time: 114367 ms
Total hot run time: 40192 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4382	4221	4215	4215
q2	392	260	255	255
q3	2946	2750	2881	2750
q4	1974	1748	1695	1695
q5	5611	5564	5533	5533
q6	233	132	135	132
q7	2230	1840	1868	1840
q8	3299	3424	3406	3406
q9	8799	8684	8874	8684
q10	4140	3889	3702	3702
q11	578	487	488	487
q12	816	631	672	631
q13	15782	3174	3172	3172
q14	307	272	277	272
q15	534	473	476	473
q16	522	433	424	424
q17	1854	1532	1525	1525
q18	8188	7832	7848	7832
q19	4022	1692	1590	1590
q20	2710	1889	1850	1850
q21	5149	5042	4920	4920
q22	643	540	545	540
Total cold run time: 75111 ms
Total hot run time: 55928 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 172907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3feb3805c3a72491c797b33f4ceb66e8e05694af, data reload: false

query1	923	382	371	371
query2	6439	2491	2154	2154
query3	6647	204	217	204
query4	19164	17607	16984	16984
query5	3649	476	508	476
query6	262	178	173	173
query7	4583	293	293	293
query8	286	271	274	271
query9	8486	2415	2420	2415
query10	569	311	283	283
query11	10453	10069	10065	10065
query12	114	85	87	85
query13	1653	377	375	375
query14	9408	6357	7126	6357
query15	227	187	190	187
query16	7731	269	265	265
query17	1719	546	538	538
query18	1937	267	271	267
query19	192	149	149	149
query20	87	80	81	80
query21	204	132	126	126
query22	4393	4115	4238	4115
query23	33623	33675	33877	33675
query24	10897	2898	2911	2898
query25	615	369	365	365
query26	710	158	154	154
query27	2368	329	328	328
query28	5963	2158	2133	2133
query29	917	630	656	630
query30	244	164	154	154
query31	979	763	739	739
query32	97	53	54	53
query33	653	287	294	287
query34	880	472	488	472
query35	743	646	642	642
query36	1097	993	987	987
query37	150	71	77	71
query38	3009	2835	2799	2799
query39	884	861	815	815
query40	205	133	126	126
query41	52	50	51	50
query42	119	99	116	99
query43	586	542	535	535
query44	1048	744	718	718
query45	190	163	170	163
query46	1059	693	696	693
query47	1857	1773	1767	1767
query48	374	315	327	315
query49	847	405	408	405
query50	761	385	393	385
query51	6843	6814	6758	6758
query52	103	96	88	88
query53	356	293	292	292
query54	870	443	453	443
query55	76	77	75	75
query56	280	254	255	254
query57	1107	1041	1048	1041
query58	246	244	253	244
query59	3265	3404	3043	3043
query60	293	274	267	267
query61	97	103	96	96
query62	594	485	442	442
query63	316	287	301	287
query64	8443	2267	1757	1757
query65	3155	3067	3078	3067
query66	738	334	344	334
query67	15329	14960	14880	14880
query68	6219	560	547	547
query69	637	473	437	437
query70	1200	1160	1069	1069
query71	482	277	274	274
query72	7278	5614	5465	5465
query73	785	326	326	326
query74	5848	5563	5434	5434
query75	3801	2705	2651	2651
query76	3250	933	928	928
query77	620	308	301	301
query78	10436	9729	9815	9729
query79	3008	522	514	514
query80	1394	468	486	468
query81	549	218	221	218
query82	1005	111	110	110
query83	197	174	170	170
query84	266	89	89	89
query85	1189	295	275	275
query86	447	298	335	298
query87	3294	3096	3095	3095
query88	3976	2359	2394	2359
query89	466	392	376	376
query90	1734	188	192	188
query91	132	99	98	98
query92	60	49	50	49
query93	3864	524	519	519
query94	1026	191	192	191
query95	420	320	324	320
query96	607	273	265	265
query97	3253	3060	3086	3060
query98	231	205	210	205
query99	1194	820	865	820
Total cold run time: 271032 ms
Total hot run time: 172907 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3feb3805c3a72491c797b33f4ceb66e8e05694af, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.07	0.08
query5	0.50	0.49	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.49	0.48
query10	0.53	0.52	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.79	0.77	0.78
query15	0.83	0.84	0.83
query16	0.37	0.36	0.38
query17	0.95	1.05	1.04
query18	0.24	0.24	0.25
query19	1.89	1.76	1.73
query20	0.01	0.00	0.00
query21	15.43	0.75	0.66
query22	3.74	7.50	2.32
query23	18.30	1.39	1.26
query24	2.13	0.24	0.22
query25	0.16	0.08	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.17	1.02	1.00
query29	12.61	3.63	3.54
query30	0.26	0.06	0.06
query31	2.85	0.39	0.38
query32	3.30	0.46	0.47
query33	2.90	2.95	2.90
query34	17.17	4.42	4.47
query35	4.54	4.45	4.59
query36	0.66	0.46	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.16
query39	0.04	0.03	0.04
query40	0.18	0.14	0.16
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.11 s
Total hot run time: 31.32 s

@ghkang98 ghkang98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman morningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Jul 3, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

@wsjz wsjz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit defb32f into apache:master Jul 3, 2024
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 12, 2024
… fix some issues. (apache#36889)

## Proposed changes

- Add iceberg partition transform unit tests.
- Change `ColumnWithTypeAndName apply(Block& block, int column_pos)` to
`ColumnWithTypeAndName apply(const Block& block, int column_pos)`.
- Fix and change string truncate partition transform issue.
- Fix bucket partition transform calculation error.
- Fix year/month partition transform calculation error due to leap year
issue.
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 12, 2024
… fix some issues. (apache#36889)

- Add iceberg partition transform unit tests.
- Change `ColumnWithTypeAndName apply(Block& block, int column_pos)` to
`ColumnWithTypeAndName apply(const Block& block, int column_pos)`.
- Fix and change string truncate partition transform issue.
- Fix bucket partition transform calculation error.
- Fix year/month partition transform calculation error due to leap year
issue.
morningman added a commit that referenced this pull request Jul 13, 2024
…7692)

## Proposed changes

Cherry-pick iceberg partition transform functionality. #36289 #36889

---------

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
… fix some issues. (#36889)

## Proposed changes

- Add iceberg partition transform unit tests.
- Change `ColumnWithTypeAndName apply(Block& block, int column_pos)` to
`ColumnWithTypeAndName apply(const Block& block, int column_pos)`.
- Fix and change string truncate partition transform issue.
- Fix bucket partition transform calculation error.
- Fix year/month partition transform calculation error due to leap year
issue.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
… fix some issues. (apache#36889)

## Proposed changes

- Add iceberg partition transform unit tests.
- Change `ColumnWithTypeAndName apply(Block& block, int column_pos)` to
`ColumnWithTypeAndName apply(const Block& block, int column_pos)`.
- Fix and change string truncate partition transform issue.
- Fix bucket partition transform calculation error.
- Fix year/month partition transform calculation error due to leap year
issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants