Skip to content

Possibility to split tests on Class basis #82

Open

Description

@adaptivecodes

I am currently using this plugin for splitting up my tests.
A lot of my pytests consist of multiple test cases which are depended on one another.
Is there a possibility to split the tests on class basis?
Example:
Testclass A: test_01: test_02:
Testclass B: test_01: test_02: test_03:
Testclass C: test_01:
Splitting those in two would do something like this:
Batch A: A(test_01, test_02) C(test_01)
Batch B: B (test_01, test_02, test_03)

Currently I am getting something like this:
Batch A: A(test_01, test_02) B(test_01)
Batch B: B(test_02, test_03), C(test_01)

Maybe I am missing something. If this is not possible at the moment are you willing to accept a Pull Request?

Greetings,
Andreas

Activity

  1. christianbundy commented on Apr 5, 2024

    @christianbundy

    Adding my use-case here: Django's TestCase runs one transaction per class, so it's optimal to run each class on a single worker.

  2. jerry-git commented on Jun 19, 2024

    @jerry-git
    Owner

    Happy to take a PR for this 🙂 I this the best would be to have a CLI flag for it such that it could be used with all the current (and future) splitting algorithms. Another option would be to implement a new splitting algorithm which is tailored for this class use case.

  3. long-git-hash commented on Dec 24, 2024

    @long-git-hash

    I am currently working on an implementation for that.

  4. jerry-git commented on Dec 29, 2024

    @jerry-git
    Owner

    FYI there's also demand for keeping tests from a single module within the same group: #108. Thus, I think introducing a CLI flag for both use cases is better than new splitting algorithms. For #108 I suggested --no-module-splits and for this one I think --no-class-splits would make sense. I believe there are plenty of synergies for these two features considering what kind of changes are needed in the current implementation.

  5. long-git-hash commented on Jan 2, 2025

    @long-git-hash

    My current implementation goes this way, that I want to support the following splitting boundaries:

    class Boundary(enum.Enum):
        FUNCTION = pytest.Function
        CLASS = pytest.Class
        MODULE = pytest.Module
        PACKAGE = pytest.Package
        FILE = pytest.File
        DIR = pytest.Dir
    

    And I thought about a parameter called boundary supporting the values

    • function
    • class
    • module
    • package
    • file
    • dir
  6. long-git-hash commented on Jan 3, 2025

    @long-git-hash

    While implementing the PR for this "splitting on boundaries" feature I stumbled over an issue with the test coverage.

    I get the warning:

    ~/git/pytest-split/.venv/lib/python3.12/site-packages/coverage/inorout.py:519: CoverageWarning: Module pytest_split was previously imported, but not measured (module-not-measured)
      self.warn(msg, slug="module-not-measured")
    

    And in the coverage report below I see that the first lines of each file are always not covered, which (from my limited understanding) should usually not be the case, because they are only imports and definitions.

    ---------- coverage: platform linux, python 3.12.3-final-0 -----------
    Name                                      Stmts   Miss  Cover   Missing
    -----------------------------------------------------------------------
    src/pytest_split/__init__.py                  0      0   100%
    src/pytest_split/algorithms.py               83     30    64%   1-33, 36, 42-59, 114-127, 155, 167, 176, 185-190
    src/pytest_split/cli.py                      13      0   100%
    src/pytest_split/folding.py                  47     41    13%   1-26, 30, 36-46, 56-93
    src/pytest_split/ipynb_compatibility.py      27      5    81%   1-9, 35, 47, 57
    src/pytest_split/plugin.py                   95     23    76%   1-25, 87-88, 116, 129-130, 152-153, 165-166, 185, 196, 215-220
    -----------------------------------------------------------------------
    TOTAL                                       265     99    63%
    

    I don't have an idea how to resolve this.
    From what I read online it might be connected to the fact, that this plugin somehow uses an old version of the pytest plugin testing API.
    Any suggestions how to remove this warning (and increase our test coverage)?

  7. long-git-hash commented on Jun 4, 2025

    @long-git-hash

    @jerry-git What shall I do with this test coverage issue mentioned in #82 (comment) (just above)?
    Shall I submit my changes with some minor refactoring in tests or in separate commits?

  8. mrohleff commented on Sep 12, 2025

    @mrohleff

    @jerry-git any update on this topic?

  9. long-git-hash commented on Sep 12, 2025

    @long-git-hash

    My custom version splitting on class boundaries is now running for half a year in our test rack.

  10. long-git-hash commented on Jun 11, 2026

    @long-git-hash

    I recently remembered that I didn't yet created a pull request for my solution to split on class boundaries (and other boundaries).
    Shall I base it on the current master or will you integrate the fixes of vokracko soon?
    I would love to see the improvements of vokracko in (although this would be some effort for me to rebase my changes to the newer version.

    Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions