Skip to content

test: parallelize leaf_pushdown_fuzz with dynamic work-stealing pool (#26759) - #26843

Closed
Sruhvx-jpg wants to merge 1 commit into
apache:mainfrom
Sruhvx-jpg:perf/leaf-pushdown-fuzz-parallel
Closed

Sruhvx-jpg wants to merge 1 commit into
apache:mainfrom
Sruhvx-jpg:perf/leaf-pushdown-fuzz-parallel

Conversation

@Sruhvx-jpg

@Sruhvx-jpg Sruhvx-jpg commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes part of #26759.

Rationale for this change

In CI merge queues, leaf_pushdown_fuzz_extended bottlenecks 32-vCPU runners for 229–238 seconds by sequentially evaluating 5,000 in-memory and 1,000 Parquet differential queries on a single thread.

What changes are included in this PR?

  1. Dynamic Work-Stealing Pool (Arc<AtomicU64>):
    • Spawns worker tasks scaled to host vCPUs via available_parallelism(), clamped to 16 workers to avoid thrashing against DataFusion's internal target_partitions = 2 plans.
    • Workers dynamically claim seeds via fetch_add(1, Ordering::Relaxed) with a pre-check load(Relaxed) guard, eliminating chunk-slicing math and straggler delays.
  2. Safe Concurrent Contexts:
    • SessionContext is safely cloned across Tokio tasks since schemas and tables are pre-registered and queries only take read locks.
  3. Refactor run() with Strongly Typed Newtypes:
    • Refactors async fn run(...) from a loose 6-argument signature (&SessionContext, &SessionContext, &str, &str, usize, u64) into strongly typed parameters: run(left: TestEngine<'_>, right: TestEngine<'_>, cases: Cases, seed0: Seed).
    • TestEngine couples each SessionContext with its diagnostic label, eliminating label-engine mismatch bugs at compile time.
    • Call sites (run_memory, run_parquet, leaf_pushdown_parquet_schema_evolution) updated accordingly.
    • Measures and reports individual fuzz test durations (std::time::Instant).
  4. Deterministic Aggregation:
    • Workers record local metrics and their earliest failing seed. The coordinator reduces by minimal seed number, preserving 100% deterministic failure reporting and reproduction (LEAF_PUSHDOWN_FUZZ_SEED=<seed> LEAF_PUSHDOWN_FUZZ_CASES=1).

Are these changes tested?

Yes, tested against extended_tests.

Benchmark Results

1. Executive Summary & Comparison

Test Target Legacy Sequential Baseline Redesigned Work-Stealing Pool Speedup / Improvement
Extended Suite (leaf_pushdown_fuzz_extended) 141.16 s (2m 21s) 15.20 s 9.29x faster (-89.2% time)
├─ In-Memory Fuzz (5,000 cases) ~110.00 s 11.55 s 9.52x faster
└─ Parquet Pushdown Fuzz (1,000 cases) ~31.16 s 3.63 s 8.58x faster
Standard Suite (leaf_pushdown_fuzz, 250 cases) 5.38 s 0.56 s (557 ms) 9.14x faster (-89.1% time)
Parquet Suite (leaf_pushdown_parquet_fuzz, 125 cases) ~2.70 s 0.44 s (437 ms) 6.14x faster
CPU Saturation 105% (1 core bottleneck) 1522% (16 concurrent threads) 14.5x core scaling
Peak Memory Footprint (Max RSS) 133.7 MiB (136,956 KB) 153.2 MiB (156,892 KB) Negligible (+19.5 MiB)
Correctness & Determinism 6000/6000 ok 6000/6000 ok (minimal seed) 100% identical repro

(Note: On 32-vCPU CI runners, leaf_pushdown_fuzz_extended baseline ran for 229–238 s, dropping to ~15 s for an estimated ~15.6x speedup).

2. Statistical Side-by-Side Benchmark (hyperfine)

Benchmarking leaf_pushdown_fuzz (250 differential queries, 3 runs):

Benchmark 1: Redesign (Parallel Work-Stealing) (3 runs)
                mean     ±       σ          min     …     max
  Wall Time    588.9 ms  ±    21.2 ms     568.7 ms  …   611.0 ms
  Memory       134.6 MiB ±     1.5 MiB    132.8 MiB …   135.6 MiB
 
Benchmark 2: Baseline (Legacy Sequential) (3 runs)
                mean     ±       σ          min     …     max
  Wall Time   5384.2 ms  ±    89.4 ms    5287.3 ms  …  5463.5 ms          +814.2%  (9.1x slower)
  Memory       120.4 MiB ±     0.3 MiB    120.2 MiB …   120.7 MiB          -10.5%

Benchmarking leaf_pushdown_fuzz_extended (6,000 differential queries, 3 runs):

Benchmark 1: Redesign (Parallel Work-Stealing) (3 runs)
                mean     ±       σ          min     …     max
  Wall Time   15.917 s   ±   0.531 s     15.312 s   …  16.303 s
  Memory       152.7 MiB ±     0.8 MiB    151.8 MiB …   153.5 MiB

3. Hardware Footprint & OS Scheduler Profiling (/usr/bin/time -v)

Baseline (Legacy Sequential):

Command: extended_tests fuzz_cases::leaf_pushdown_fuzz::leaf_pushdown_fuzz_extended --exact
User time: 147.99 s
System time: 0.64 s
Percent of CPU this job got: 105%
Elapsed (wall clock) time: 2:21.16 (141.16 s)
Maximum resident set size: 136956 KB (~133.7 MiB)
Voluntary context switches: 110933
Involuntary context switches: 5298

Redesign (Parallel Work-Stealing):

Command: extended_tests fuzz_cases::leaf_pushdown_fuzz::leaf_pushdown_fuzz_extended --exact
User time: 239.61 s
System time: 0.70 s
Percent of CPU this job got: 1522%
Elapsed (wall clock) time: 0:15.78 (15.78 s)
Maximum resident set size: 156892 KB (~153.2 MiB)
Voluntary context switches: 11204
Involuntary context switches: 54633

Co-Author: Gemini 3.8

@github-actions github-actions Bot added the core Core DataFusion crate label Oct 28, 2026
@Sruhvx-jpg
Sruhvx-jpg force-pushed the perf/leaf-pushdown-fuzz-parallel branch 3 times, most recently from ae4e942 to bc95b65 Compare October 28, 2026 11:09
@Sruhvx-jpg
Sruhvx-jpg force-pushed the perf/leaf-pushdown-fuzz-parallel branch from bc95b65 to b305183 Compare October 28, 2026 11:15
@2010YOUY01

Copy link
Copy Markdown
Member

I think this approach is over-engineered, here is an alternative: #26846

@Sruhvx-jpg

Sruhvx-jpg commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

@2010YOUY01 If right your approach chooses random seeds, right? but my approach runs all the test in less amount. I do not think its over-engineered, cuz pretty sure u run fuzz test concurrently,r right?

Anyways why do u want to trade test coverage with simplicity? My solution preserve the 5k test, I have added comment to explain stuff too.

@2010YOUY01

Copy link
Copy Markdown
Member

@2010YOUY01 If right your approach chooses random seeds, right? but my approach runs all the test in less amount. I do not think its over-engineered, cuz pretty sure u run fuzz test concurrently,r right?

Anyways why do u want to trade test coverage with simplicity? My solution preserve the 5k test, I have added comment to explain stuff too.

Adding a work-stealing pool for a single UT looks a bit too much.

The total run time (aggregated exec time across all CPUs) don't change by parallelizing, and our CI resource can't afford to continuously add new UTs that each take 5 min to run.

A better way is to make all fuzzing tests configurable by time budget:

  • per-PR CI: fuzzing for 10 min
  • daily CI: fuzzing for 1 hr

Though more work to be done for that goal beyond that PR, I'll try to investigate further.

@alamb

alamb commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.

https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful

This has the hallmarks of AI code that was not reviewed by the author: an "Executive Summary" description, a work-stealing pool and a TestEngine abstraction for a single fuzz test, and a simpler alternative already proposed in #26846.

If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.

@alamb alamb closed this Oct 29, 2026
@Sruhvx-jpg

Sruhvx-jpg commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@alamb I think there is a mis-understanding, I have mentioned that I have co-authored it with Gemini 3.8.

I do understand the design, cause I was the one to brainstorm it myself and then build on that design. I did take AI help, but did not intent to or did "AI Dump". I have read the policies.

And yes, I did review the code myself. My design tried to preserve the 5k Test.

@Sruhvx-jpg

Copy link
Copy Markdown
Contributor Author

Also the TestEngine abstraction was a refactor, nothing much. I reckon that it should have been addressed in other PR or a issue but I thought why not this PR itself cuz its was in the radius

@Sruhvx-jpg

Copy link
Copy Markdown
Contributor Author

In conclusion, i have added a "Co-Author: Gemini 3.8" and find insulting to say its "AI Dump" becuase I did not just ask my clanker to fo everything on its own.

The code wasn't written by me - as I said, but it was reviewed. Anyways I reckon this design might be over-engineered and a simple test was something the ASF prefers, but please criticise my design, not my ai usage.

Thanks for your precious time and apologies if you found me rude 😀

@Sruhvx-jpg

Copy link
Copy Markdown
Contributor Author

Again the abstractions added was all something I decided to do, not my clanker.

To give you a wider context, I was reading the Rust Style Guide bt Google, thought I could apply it here.

Reccon I should have asked if I should add it, but the issuer seemed busy, Anyways I still see a scope to work on this PR. I do not know why would you judge me like that. There could be other caveats, maybe the clamping part, but AI Dump?

@Sruhvx-jpg

Copy link
Copy Markdown
Contributor Author

@alamb @2010YOUY01

leaf_pushdown_fuzz_extended: run the seeds in a few concurrent chunks. The report already counts failures across all seeds and keeps the first one, so the chunks' results can be merged into the same report.

For context, the issuer did ask for concurrent chunking. I reccon the newtypes part was over engineering, which could be reverted in this PR

@Sruhvx-jpg

Sruhvx-jpg commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Anyways, i will propose a different solution I guess if thats what you want.

@alamb

alamb commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Sorry for the somewhat abrupt close here @Sruhvx-jpg -- our challenge is that we have only a small amount of review bandwidth and have been overwhelmed with so many suggestions for changes that we can't possible keep up reviewing them.

I am sorry that your good faith efforts got caught up in my mass closings of PRs today

However that being said I think @2010YOUY01 's feedback is vaild here -- this seems a substantial over engineering.

@alamb

alamb commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

a more reasonable change might be, for example, to split up the failing test into smaller tests so they run in parallel

@Sruhvx-jpg

Copy link
Copy Markdown
Contributor Author

@alamb I understand it, maybe i scrap the atomic counter and worker stealing. I did glaze over the fack that extra compute was being used as my main focus back than was mostly just making the CI fast, anyways am do not intended to give up on this one, I will submit another PR.

Thanks for the prompt reply, I appreciate your kindness here :>

@Sruhvx-jpg

Copy link
Copy Markdown
Contributor Author

However that being said I think @2010YOUY01 's feedback is vaild here -- this seems a substantial over engineering.

Yes, i reccon it is. Not over-engineered to say but the style change was highly cosmetic to say. I think it my fault for letting my clanker exercise "larping" or rather to say use "buzz-words".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants