Repository navigation
Conversation
The kafka integration tests take about 199s in the nightly run. In September they took about 97s. Three changes added about 100s of serial test time: - #4843 enabled 8 schema_registry tests that were skipped (about 61s). - #4923 added 2 partition_assignment_strategy tests (about 30s). - #4959 added the tombstone test (about 9s). No top-level test in the package calls t.Parallel(), so these tests run one after the other. Each schema_registry test starts its own pair of Redpanda containers. Add t.Parallel() to these 11 tests. They use their own containers or unique names on the shared container. One Redpanda container reserves about 11000 Linux AIO events, and the default host limit is 65536. More than about 5 containers at the same time exit with code 133. Start the test containers with --max-networking-io-control-blocks=1000, which lowers this to about 2000. Locally, the 11 tests take 24s instead of 93s. Ref: CON-565
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
kafkaintegration tests took about 97s in the September nightly runs. Now they take about 199s.Three PRs added about 100s of test time:
No top-level test in the package calls
t.Parallel(), so these tests ran one after the other.Each schema_registry test starts its own pair of Redpanda containers.
TestSchemaRegistry*Integration (8 tests) TestRedpandaOutputTombstoneIntegration TestIntegrationRedpandaPartitionAssignment{Strategy,Migration} integration.CheckSkip(t) + t.Parallel()These tests use their own containers, or unique topic and group names on the shared container.
The parallel start needs one more change.
One Redpanda container reserves about 11000 Linux AIO events, and the default host limit (
fs.aio-max-nr) is 65536.When more than about 5 containers run at the same time, the next one exits with code 133.
The new
redpandatest.WithLowAIOUsage()option adds--max-networking-io-control-blocks=1000to the Redpanda command.This lowers the reservation to about 2000 events per container.
Evidence
✓ kafka (3m18.8s).After:
workflow_dispatchwith filterkafkaon this branch, 39811314264:✓ kafka (2m2.3s), with no failures.Both runs use the same runner (
ubuntu-latest-32) and the same Redpanda image (sha256:badca2d9…, v26.2.5).main: 93s and 93s.t.Parallel()without the low AIO option: 24s and 23s, with 8 to 9 failures (container exited with code 133)./proc/sys/fs/aio-nrin the VM): 11026 with the default flags, 2026 with--max-networking-io-control-blocks=1000.kafka/enterpriseandconfluentintegration tests pass with the new flag.go vet -tags integrationpasses for all packages that importredpandatest.Merge Danger
Door: two-way
The change is in test code only. A revert brings back the old behavior.
Blast Radius: tests
The low AIO flag applies to every test that uses
redpandatest(confluent,kafka,kafka/enterprise,redpanda,redpanda/migrator).I did not run the
redpandaandredpanda/migratorintegration tests with it.The tests are tiny, so the smaller networking AIO budget is not expected to matter.
If one of the parallel tests is flaky, it can now run at the same time as the other 10. This could make the flake show more often.
Ref: CON-565