Repository navigation
test: parallelize leaf_pushdown_fuzz with dynamic work-stealing pool (#26759) - #26843
Sruhvx-jpg wants to merge 1 commit into
Conversation
ae4e942 to
bc95b65
Compare
bc95b65 to
b305183
Compare
|
I think this approach is over-engineered, here is an alternative: #26846 |
|
@2010YOUY01 If right your approach chooses random seeds, right? but my approach runs all the test in less amount. I do not think its over-engineered, cuz pretty sure u run fuzz test concurrently,r right? Anyways why do u want to trade test coverage with simplicity? My solution preserve the 5k test, I have added comment to explain stuff too. |
Adding a work-stealing pool for a single UT looks a bit too much. The total run time (aggregated exec time across all CPUs) don't change by parallelizing, and our CI resource can't afford to continuously add new UTs that each take 5 min to run. A better way is to make all fuzzing tests configurable by time budget:
Though more work to be done for that goal beyond that PR, I'll try to investigate further. |
|
Thanks for the contribution! I am closing this under DataFusion's AI Dump policy. This has the hallmarks of AI code that was not reviewed by the author: an "Executive Summary" description, a work-stealing pool and a TestEngine abstraction for a single fuzz test, and a simpler alternative already proposed in #26846. If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review. |
|
@alamb I think there is a mis-understanding, I have mentioned that I have co-authored it with Gemini 3.8. I do understand the design, cause I was the one to brainstorm it myself and then build on that design. I did take AI help, but did not intent to or did "AI Dump". I have read the policies. And yes, I did review the code myself. My design tried to preserve the 5k Test. |
|
Also the TestEngine abstraction was a refactor, nothing much. I reckon that it should have been addressed in other PR or a issue but I thought why not this PR itself cuz its was in the radius |
|
In conclusion, i have added a "Co-Author: Gemini 3.8" and find insulting to say its "AI Dump" becuase I did not just ask my clanker to fo everything on its own. The code wasn't written by me - as I said, but it was reviewed. Anyways I reckon this design might be over-engineered and a simple test was something the ASF prefers, but please criticise my design, not my ai usage. Thanks for your precious time and apologies if you found me rude 😀 |
|
Again the abstractions added was all something I decided to do, not my clanker. To give you a wider context, I was reading the Rust Style Guide bt Google, thought I could apply it here. Reccon I should have asked if I should add it, but the issuer seemed busy, Anyways I still see a scope to work on this PR. I do not know why would you judge me like that. There could be other caveats, maybe the clamping part, but AI Dump? |
For context, the issuer did ask for concurrent chunking. I reccon the newtypes part was over engineering, which could be reverted in this PR |
|
Anyways, i will propose a different solution I guess if thats what you want. |
|
Sorry for the somewhat abrupt close here @Sruhvx-jpg -- our challenge is that we have only a small amount of review bandwidth and have been overwhelmed with so many suggestions for changes that we can't possible keep up reviewing them. I am sorry that your good faith efforts got caught up in my mass closings of PRs today However that being said I think @2010YOUY01 's feedback is vaild here -- this seems a substantial over engineering. |
|
a more reasonable change might be, for example, to split up the failing test into smaller tests so they run in parallel |
|
@alamb I understand it, maybe i scrap the atomic counter and worker stealing. I did glaze over the fack that extra compute was being used as my main focus back than was mostly just making the CI fast, anyways am do not intended to give up on this one, I will submit another PR. Thanks for the prompt reply, I appreciate your kindness here :> |
Yes, i reccon it is. Not over-engineered to say but the style change was highly cosmetic to say. I think it my fault for letting my clanker exercise "larping" or rather to say use "buzz-words". |
Which issue does this PR close?
Closes part of #26759.
Rationale for this change
In CI merge queues,
leaf_pushdown_fuzz_extendedbottlenecks 32-vCPU runners for 229–238 seconds by sequentially evaluating 5,000 in-memory and 1,000 Parquet differential queries on a single thread.What changes are included in this PR?
Arc<AtomicU64>):available_parallelism(), clamped to 16 workers to avoid thrashing against DataFusion's internaltarget_partitions = 2plans.fetch_add(1, Ordering::Relaxed)with a pre-checkload(Relaxed)guard, eliminating chunk-slicing math and straggler delays.SessionContextis safely cloned across Tokio tasks since schemas and tables are pre-registered and queries only take read locks.run()with Strongly Typed Newtypes:async fn run(...)from a loose 6-argument signature (&SessionContext, &SessionContext, &str, &str, usize, u64) into strongly typed parameters:run(left: TestEngine<'_>, right: TestEngine<'_>, cases: Cases, seed0: Seed).TestEnginecouples eachSessionContextwith its diagnostic label, eliminating label-engine mismatch bugs at compile time.run_memory,run_parquet,leaf_pushdown_parquet_schema_evolution) updated accordingly.std::time::Instant).LEAF_PUSHDOWN_FUZZ_SEED=<seed> LEAF_PUSHDOWN_FUZZ_CASES=1).Are these changes tested?
Yes, tested against
extended_tests.Benchmark Results
1. Executive Summary & Comparison
leaf_pushdown_fuzz_extended)leaf_pushdown_fuzz, 250 cases)leaf_pushdown_parquet_fuzz, 125 cases)(Note: On 32-vCPU CI runners,
leaf_pushdown_fuzz_extendedbaseline ran for 229–238 s, dropping to ~15 s for an estimated ~15.6x speedup).2. Statistical Side-by-Side Benchmark (
hyperfine)Benchmarking
leaf_pushdown_fuzz(250 differential queries, 3 runs):Benchmarking
leaf_pushdown_fuzz_extended(6,000 differential queries, 3 runs):3. Hardware Footprint & OS Scheduler Profiling (
/usr/bin/time -v)Baseline (Legacy Sequential):
Redesign (Parallel Work-Stealing):
Co-Author: Gemini 3.8