You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Speedup execution of sqllogictests with more parallelization #20524
Is your feature request related to a problem or challenge?
@blaginin is trying to speed up execution of tests
Right now, when running the tests by default it takes 8 wallclock seconds with 16 cores.
cargo test --profile=ci --test sqllogictests
...
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds
However, if you run time, you can see this only consumes 53 coresec out of a theoretical 128 coresec ( 16 cores * 8 seconds = 128 core*sec)
time cargo test --profile=ci --test sqllogictests
Finished `ci` profile [unoptimized] target(s) in 0.34s
Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-91ecf623e0e3b627)
Running with 16 test threads (available parallelism: 16)
Completed 407 test files in 8 seconds
real 0m8.232s
user 0m53.201s
sys 0m1.247s
This means the tests could in theory go twice as fast with more cores
I when looked briefly at this, I think one problem is that the push_down_filter.slt test takes a long time and is single threaded for that time
[00:00:05] ####################-------------------- 52/107 "push_down_filter.slt" - 4 took > 500 ms
Describe the solution you'd like
I would like the sqllogictests to run on all cores
Describe alternatives you've considered
Maybe we can start by splitting up push_down_filter.slt into a few smaller files that can run in parallel
Activity
Split `push_down_filter.slt` into standalone sqllogictest files to re…
Add deterministic per-file timing summary to sqllogictest runner (#20569
Improve sqllogicteset speed by creating only a single large file rath…
Speedup sqllogictests by running long running tests first (#20576)
Speedup push_down_filter_regression.slt by using uncompressed parquet (…
Split `push_down_filter.slt` into standalone sqllogictest files to re…
Add deterministic per-file timing summary to sqllogictest runner (apa…
Improve sqllogicteset speed by creating only a single large file rath…
Speedup sqllogictests by running long running tests first (apache#20576)
Speedup push_down_filter_regression.slt by using uncompressed parquet (…