Skip to content

fix: Improve slow fuzz test leaf_pushdown_fuzz - #26846

Draft
2010YOUY01 wants to merge 1 commit into
apache:mainfrom
2010YOUY01:fix-slow-fuzz
Draft

2010YOUY01 wants to merge 1 commit into
apache:mainfrom
2010YOUY01:fix-slow-fuzz

Conversation

@2010YOUY01

Copy link
Copy Markdown
Member

Which issue does this PR close?

case 1 of #26759

Rationale for this change

This section I'll try to explain the idea at high-level.

The simplest way to write a fuzzer is:

# Fixed seed: deterministic and easy to reproduce
run_fuzz_cases(rounds=100, seed=FIXED_SEED)

Given limited CI resources, we typically want to run as many rounds as possible within a small time budget, e.g. 5 seconds.

However, some coverage goals cannot be reached within that budget. For example:

leaf_pushdown_fuzz_extended takes around 4 minutes in total CI time, as only a fuzzing unit test.

A better approach would be:

# Random seed: broaden coverage across CI runs
run_fuzz_cases(rounds=100, seed=current_time())

# On failure, report the failing case seed, e.g. 42
# To reproduce or add a regression test manually:
run_case(seed=42)

This trades some convenience in reproduction for broader coverage over time, and also uses way less CI resources.

What changes are included in this PR?

Apply the above idea for leaf_pushdown_fuzz test

Locally this test runs from 42s to 2s

cargo test --profile ci -p datafusion --features extended_tests \
                                                               --test extended_tests -- leaf_pushdown --nocapture

What is the testing strategy for this PR?

Are there any user-facing changes?

@2010YOUY01

Copy link
Copy Markdown
Member Author

Some refactoring is needed to make individual test cases reproducible from a seed.

For example, if a test case fails:

# Random seed determined at runtime
run_case(357)

The error message should print the failing seed, so we can add it to the regression tests:

def run_regression():
    # All previously failing seeds
    run_case(43)
    run_case(357)

I think this is probably an acceptable maintenance overhead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant