ResultCollector exception handling: improvement for pytest.raises() failures

Problem Description

This issue has been observed in practice when testing the 02_control_flow.ipynb notebook, but cannot be reliably reproduced in current testing environments. When a student's solution function failed to raise an expected exception in a pytest.raises() context, the test was incorrectly classified as TestOutcome.TEST_ERROR instead of TestOutcome.FAIL, causing the notebook to display a misleading "🚨 Syntax Error" message instead of "❌ Failed".

Current Status

Reproduction: Intermittent/unreproducible - the issue was observed once but cannot be triggered consistently.

Likely cause: The current code theoretically should work because pytest.fail.Exception and _pytest.outcomes.Failed are the same class object (pytest aliases them). However, the intermittent nature suggests there may be edge cases related to pytest version differences, hook execution order, or specific test scenarios.

Analysis

In tutorial/tests/testsuite/helpers.py at lines 759-778, the ResultCollector.pytest_exception_interact() method catches AssertionError and pytest.fail.Exception, but does NOT explicitly reference _pytest.outcomes.Failed, which is the canonical name for the exception raised when pytest.raises() fails.

python-tutorial/tutorial/tests/testsuite/helpers.py

Lines 767 to 772 in 047f9b4

    
           outcome = ( 
        
               TestOutcome.FAIL 
        
               if exc.errisinstance(AssertionError) 
        
               or exc.errisinstance(pytest.fail.Exception) 
        
               else TestOutcome.TEST_ERROR 
        
           )

Why is it happening?

When a test uses pytest.raises() but the student's solution doesn't raise the expected exception:

Test with pytest.raises(): Student's solution should raise ValueError, but doesn't
pytest behavior: pytest.raises() context manager's __exit__ method detects no exception was raised
pytest raises Failed: Internally raises _pytest.outcomes.Failed with message like "DID NOT RAISE ValueError"
Exception handling: pytest_exception_interact() is called but doesn't recognize Failed exception
Misclassification: Defaults to TestOutcome.TEST_ERROR
Escalation: run_pytest_for_function() (in testsuite.py) sees TEST_ERROR and returns IPytestOutcome.PYTEST_ERROR
Display: Notebook shows "🚨 Syntax Error" (orange) instead of "❌ Failed" (red)

Impact

Affected tests:

Any test using pytest.raises(SomeException) where the student's code fails to raise that exception

Examples:

test_02_control_flow.py:492 - test_base_converter_invalid_bases
test_12_functions_advanced.py:213,231 - test_once_twice
test_magic_example.py:55 - test_power2_raise

Students receive confusing feedback suggesting a syntax/compile error rather than understanding they failed to raise the expected exception.

Proposed Hotfix

Even though pytest.fail.Exception and _pytest.outcomes.Failed are technically the same class object in pytest's current implementation, we recommend explicitly referencing _pytest.outcomes.Failed for the following reasons:

Code clarity: Makes it explicit that we're handling the canonical exception from pytest.raises() failures
Documentation: Self-documents that this code path handles failed pytest.raises() assertions
Defensive programming: Guards against potential pytest version differences or internal refactoring
Edge case protection: May prevent issues related to import order, hook execution timing, or environment-specific behaviors
Direct import: Uses the actual exception class rather than relying on the .Exception attribute pattern

Changes

In tutorial/tests/testsuite/helpers.py:

Add import somewhere (preferably in the function's scope):

from _pytest.outcomes import Failed

Update lines 767-772:

outcome = (
    TestOutcome.FAIL
    if isinstance(exc.value, AssertionError | Failed)
    else TestOutcome.TEST_ERROR
)

This explicitly catches all pytest assertion failures (direct assertions, pytest.fail(), and pytest.raises() failures) as FAIL, while preserving TEST_ERROR classification for actual syntax errors, import errors, etc.

Technical Note

In pytest's current implementation, pytest.fail.Exception and _pytest.outcomes.Failed reference the same class:

_pytest.outcomes.Failed is the actual exception class
pytest.fail.Exception is created by the @_with_exception decorator that attaches the exception class to the fail() function
pytest.raises.Exception is just an alias of fail.Exception

Testing Considerations

Test scenario to verify behavior:

Use existing tests like test_02_control_flow.py::test_base_converter_invalid_bases:

# In notebook cell (02_control_flow.ipynb):
%%ipytest
def solution_base_converter(number, from_base, to_base):
    # Intentionally doesn't raise ValueError for invalid bases
    return "42"

Expected behavior:

With the fix, the test should consistently show "❌ Failed" with a clear message about not raising the expected ValueError
The defensive fix ensures this behavior is stable across different pytest versions and environments

Relevant second reproduction of the same visible symptom via a different path, found while reviewing #369:

Another trigger: student stubs that return `None`

Any notebook where the initial stub is return (or a student's in-progress code returns None) hits the same "🚨 Syntax Error" misdisplay – no pytest.raises() needed. Example from test_31_image_classification.py:

def solution_scale_image(img, scale_factor):
    # Start your code here
    return
    # End your code here

The test then does image_test.shape, which raises AttributeError: 'NoneType' object has no attribute 'shape'. That AttributeError is neither AssertionError nor pytest.fail.Exception nor _pytest.outcomes.Failed, so even after the fix proposed in this issue it still routes to TestOutcome.TEST_ERROR and renders as "🚨 Syntax Error". PR #369 worked around this test-side by prepending assert isinstance(image_test, np.ndarray), "Your function should return an image." to every test – but that's a per-test workaround for what is really a display-layer bug.

Two additional things the `Failed`-import fix alone doesn't address

1. The label itself is wrong for any non-SyntaxError runtime failure. helpers.py:276-279 unconditionally renders TestOutcome.TEST_ERROR as "🚨 Syntax Error":

case TestOutcome.TEST_ERROR:
    status_class = "test-error"
    icon = "🚨"
    status_text = "Syntax Error"

This is correct for the IPytestOutcome.COMPILE_ERROR path (cell genuinely fails to parse) but wrong for every runtime error that survives the pytest_exception_interact routing. The exception type and message are already shown in the error block below (helpers.py:302-313), so the "Syntax Error" header is both cosmetically misleading and contradicts the rendered traceback.

Minimal fix – branch on the actual exception:

case TestOutcome.TEST_ERROR:
    status_class = "test-error"
    if isinstance(self.exception, (SyntaxError, IndentationError)):
        icon = "🚨"
        status_text = "Syntax Error"
    else:
        icon = "⚠️"
        exc_name = type(self.exception).__name__ if self.exception else "Error"
        status_text = f"Runtime Error ({exc_name})"

2. testsuite.py:68-82 discards passing parametrizations whenever any test errors. The pytest.ExitCode.TESTS_FAILED branch checks if any(test.outcome == TEST_ERROR ...) and, if so, downgrades the entire IPytestResult to IPytestOutcome.PYTEST_ERROR, throwing away every TestCaseResult and keeping only the exceptions list. A student whose scale-image stub errors on all three parametrizations, or whose solution_once_twice errors on one of the two cases in test_12_functions_advanced.py:213,231, currently sees a single generic error panel instead of per-parametrization granularity. The IPytestOutcome.FINISHED display path at helpers.py:656-695 already renders a mix of PASS/FAIL/TEST_ERROR correctly per row, so the simplification is:

case pytest.ExitCode.TESTS_FAILED:
    return IPytestResult(
        function=function,
        status=IPytestOutcome.FINISHED,
        test_results=list(result_collector.tests.values()),
    )

IPytestOutcome.PYTEST_ERROR remains valid – it's still used by the INTERNAL_ERROR branch (testsuite.py:88-93).

Suggested consolidation

The Failed-import fix proposed in this issue is still correct and worth landing – it's strictly additive. I'd propose broadening this issue's scope to cover the three things together:

Promote _pytest.outcomes.Failed to FAIL in pytest_exception_interact (the original proposal here).
Branch the TEST_ERROR label on the actual exception class in TestCaseResult.to_html().
Stop downgrading TESTS_FAILED runs to PYTEST_ERROR when any test errored.

(2) alone already removes the visible mislabel that students hit; (1) + (3) together let a mixed run show "2 ✅, 1 ❌ DID NOT RAISE ValueError" instead of "everything is a 🚨 Syntax Error". These are all small, surgical display-layer changes and don't touch the routing walk proposed in the TODO at helpers.py:764, which remains the longer-term fix.

Route pytest.raises failures to FAIL instead of TEST_ERROR (#345)

Navigation Menu

Repository navigation

`ResultCollector` exception handling: improvement for `pytest.raises()` failures #345

Problem Description

Current Status

Analysis

Why is it happening?

Impact

Proposed Hotfix

Changes

Technical Note

Testing Considerations

Another trigger: student stubs that return `None`

Two additional things the `Failed`-import fix alone doesn't address

Suggested consolidation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	outcome = (
	TestOutcome.FAIL
	if exc.errisinstance(AssertionError)
	or exc.errisinstance(pytest.fail.Exception)
	else TestOutcome.TEST_ERROR
	)

ResultCollector exception handling: improvement for pytest.raises() failures #345

Description

Problem Description

Current Status

Analysis

Why is it happening?

Impact

Proposed Hotfix

Changes

Technical Note

Testing Considerations

Activity

edoardob90 commented on Apr 15, 2026

Another trigger: student stubs that return None

Two additional things the Failed-import fix alone doesn't address

Suggested consolidation

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`ResultCollector` exception handling: improvement for `pytest.raises()` failures #345

Another trigger: student stubs that return `None`

Two additional things the `Failed`-import fix alone doesn't address