You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the Plan phase, when multiple UNION combinations are encountered, multiple rounds of iteration will increase the plan time. In my test, it may cost 2-3 seconds to analyze and plan a SQL with 30 UNIONs.
Too many Fragments
A SQL may generate hundreds or thousands of Fragments, and all Fragments are currently sent to the corresponding BE separately. The common fields of these Fragments will undergo multiple serialization and deserialization operations, which results in a very high RPC processing time.
In my test, a SQL with 30 UNIONs generates 300+ fragments, and total size of these fragments is over 100MB, mainly
occupied by DescriptorTable.
Optimization Strategy:
For problem 1, it is necessary to analyze the corresponding code hot spots through the Java Profiler tool to optimize the analysis process. (TODO)
For problem 2, multiple fragments sent to the same BE can be merged and sent in batch, reducing the number of RPCs and reducing the amount of data sent. (WIP)
Activity
[-][union all]现在Doris的union all好像是会集中到一台机器上进行计算,导致效率不高[/-][+][feature wanted]现在Doris的union all好像是会集中到一台机器上进行计算,导致效率不高[/+]xinghuayu007 commentedon Sep 27, 2020
能给一个复现的例子吗
li-ang-666 commentedon Oct 13, 2020
我给超勇哥提供了例子~
li-ang-666 commentedon Oct 13, 2020
@chaoyli 勇哥
morningman commentedon Nov 12, 2020
There are two main problems here.
In the Plan phase, when multiple UNION combinations are encountered, multiple rounds of iteration will increase the plan time. In my test, it may cost 2-3 seconds to analyze and plan a SQL with 30 UNIONs.
A SQL may generate hundreds or thousands of Fragments, and all Fragments are currently sent to the corresponding BE separately. The common fields of these Fragments will undergo multiple serialization and deserialization operations, which results in a very high RPC processing time.
In my test, a SQL with 30 UNIONs generates 300+ fragments, and total size of these fragments is over 100MB, mainly
occupied by
DescriptorTable.Optimization Strategy: