Skip to content

feature request: order tests by slowest-first #89

Open

Description

@charles-cooper

in long running test suites (especially with xdist), it's useful to run the slowest tests first so that you end up with better load. the following code seems to do so:

def pytest_collection_modifyitems(config, items):
    try:
        with open(".test_durations") as f:
            s = f.read()
            durations = json.loads(s)
    except FileNotFoundError:
        durations = {}

    timings = {}
    for item in items:
        path, _, reqname = item.location
        request_id = f"{path}::{reqname}"
        timing = durations.get(request_id, None)
        if timing is not None:
            timings[item] = timing

    meantime = sum(timings.values()) / len(timings)
    # sort by highest time first
    items.sort(key=lambda item: -timings.get(item, meantime))

but i was thinking it might be good for this to be available in the pytest-split plugin itself, so we don't need to add this same function across different projects.

alternatively, i was thinking publishing this as a plugin myself, if pytest-split does not want to add it.

Activity

  1. jerry-git commented on Mar 18, 2024

    @jerry-git
    Owner

    Hey 👋 ! There's already support for different splitting algorithms, see

    I believe what you are asking is actually quite close to "least duration" algo that already exists.

    Happy to take a contribution which introduces "slowest first" splitting algorithm 🙂

  2. charles-cooper commented on Mar 18, 2024

    @charles-cooper
    Author

    yea these are interesting. the use case i am imagining is actually most useful with --splits 1 --group 1 because it reorders the items on the local worker. do you think "slowest first" makes sense with more splits?

  3. jerry-git commented on Mar 18, 2024

    @jerry-git
    Owner

    If the main motivation is to optimise the usage with xdist, I guess there could be also use cases which prefer to have multiple test runner nodes (e.g. GHA workers). So, pytest-split for splitting the work to multiple nodes and then xdist for splitting the work inside each node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions