make localvalidation does not produce consistent results

Running the make localvalidation against a sample runtime multiple times produces noticeably different results on each run.

For example, I ran the suite against with the packaged runc in Fedora 34 (runc-1.0.2-2.fc34.x86_64) repeatedly with the command:

$ sudo --preserve-env=PATH make RUNTIME=runc localvalidation

Results included:

Suites:   16 failed, 34 passed, 8 skip, 58 of 58 completed
Asserts:  31 failed, 3533 passed, 444 skip, of 4008
Time:     32s

Suites:   15 failed, 35 passed, 8 skip, 58 of 58 completed
Asserts:  29 failed, 3527 passed, 444 skip, of 4000
Time:     36s

Suites:   15 failed, 35 passed, 8 skip, 58 of 58 completed
Asserts:  30 failed, 3534 passed, 444 skip, of 4008
Time:     32s

Suites:   14 failed, 36 passed, 8 skip, 58 of 58 completed
Asserts:  29 failed, 3801 passed, 481 skip, of 4311
Time:     34s

Likewise results vary if I run the suite against crun (specifically crun-1.0-1.fc34.x86_64):

Suites:   13 failed, 37 passed, 8 skip, 58 of 58 completed
Asserts:  18 failed, 3714 passed, 481 skip, of 4213
Time:     26s

Suites:   15 failed, 35 passed, 8 skip, 58 of 58 completed
Asserts:  20 failed, 3702 passed, 481 skip, of 4203
Time:     27s

Suites:   14 failed, 36 passed, 8 skip, 58 of 58 completed
Asserts:  14 failed, 3564 passed, 481 skip, of 4059
Time:     25s

Navigation Menu