Skip to content

kafka: run the slow integration tests in parallel - #5024

Open
Leward wants to merge 1 commit into
mainfrom
claude/kafka-it-slowdown-d072c9
Open

Leward wants to merge 1 commit into
mainfrom
claude/kafka-it-slowdown-d072c9

Conversation

@Leward

@Leward Leward commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Summary

The kafka integration tests took about 97s in the September nightly runs. Now they take about 199s.
Three PRs added about 100s of test time:

No top-level test in the package calls t.Parallel(), so these tests ran one after the other.
Each schema_registry test starts its own pair of Redpanda containers.

 TestSchemaRegistry*Integration (8 tests)
 TestRedpandaOutputTombstoneIntegration
 TestIntegrationRedpandaPartitionAssignment{Strategy,Migration}
   integration.CheckSkip(t)
+  t.Parallel()

These tests use their own containers, or unique topic and group names on the shared container.

The parallel start needs one more change.
One Redpanda container reserves about 11000 Linux AIO events, and the default host limit (fs.aio-max-nr) is 65536.
When more than about 5 containers run at the same time, the next one exits with code 133.
The new redpandatest.WithLowAIOUsage() option adds --max-networking-io-control-blocks=1000 to the Redpanda command.
This lowers the reservation to about 2000 events per container.

 redpandatest.StartSingleBrokerWithConfig
-  opts := nil
+  opts := [WithLowAIOUsage()]
 kafka_test sharedRedpanda, startRedpandaWithSASL
+  redpandatest.WithLowAIOUsage()

Evidence

  • Before: nightly 39526346316 (2026-10-21): ✓ kafka (3m18.8s).
    After: workflow_dispatch with filter kafka on this branch, 39811314264: ✓ kafka (2m2.3s), with no failures.
    Both runs use the same runner (ubuntu-latest-32) and the same Redpanda image (sha256:badca2d9…, v26.2.5).
  • On a local Colima VM (8 CPUs), I ran only the 11 tests, two times for each version:
    • main: 93s and 93s.
    • t.Parallel() without the low AIO option: 24s and 23s, with 8 to 9 failures (container exited with code 133).
    • This PR: 24s and 24s, with no failures.
  • AIO use of one container (/proc/sys/fs/aio-nr in the VM): 11026 with the default flags, 2026 with --max-networking-io-control-blocks=1000.
  • kafka/enterprise and confluent integration tests pass with the new flag. go vet -tags integration passes for all packages that import redpandatest.

Merge Danger

Door: two-way

The change is in test code only. A revert brings back the old behavior.

Blast Radius: tests

The low AIO flag applies to every test that uses redpandatest (confluent, kafka, kafka/enterprise, redpanda, redpanda/migrator).
I did not run the redpanda and redpanda/migrator integration tests with it.
The tests are tiny, so the smaller networking AIO budget is not expected to matter.
If one of the parallel tests is flaky, it can now run at the same time as the other 10. This could make the flake show more often.

Ref: CON-565

The kafka integration tests take about 199s in the nightly run. In September they took about 97s.
Three changes added about 100s of serial test time:

- #4843 enabled 8 schema_registry tests that were skipped (about 61s).
- #4923 added 2 partition_assignment_strategy tests (about 30s).
- #4959 added the tombstone test (about 9s).

No top-level test in the package calls t.Parallel(), so these tests run one after the other.
Each schema_registry test starts its own pair of Redpanda containers.

Add t.Parallel() to these 11 tests.
They use their own containers or unique names on the shared container.

One Redpanda container reserves about 11000 Linux AIO events, and the default host limit is 65536.
More than about 5 containers at the same time exit with code 133.
Start the test containers with --max-networking-io-control-blocks=1000, which lowers this to about 2000.

Locally, the 11 tests take 24s instead of 93s.

Ref: CON-565

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant