Skip to content

Add support for soft assertions #19

Open

Description

@baev
No description provided.

Activity

  1. added this to the AJava Release milestone on Mar 14, 2017
  2. changed the title [-][TestNG] Add support for soft asserions[/-] [+]Add support for soft assertions[/+] on Nov 12, 2018
  3. scripnichenko commented on Dec 17, 2018

    @scripnichenko

    Hi @baev. Any plans to get it implemented? 💡

  4. baev commented on Dec 17, 2018

    @baev
    MemberAuthor

    @scripnichenko we are not using soft assertions in our code and you are the first one who are asking for such issue during last few years. Seems like everyone uses AssertJ instead of TestNG assertions this days. But I'm understand that issue is important for people who support old projects written years ago and have no possibility for refactoring.

    BTW I have no idea how to fix this, some research work is required. Probably the easiest way to fix the issue add an aspect. If you have any ideas feel free to share.

  5. Little-Elephant commented on Feb 12, 2019

    @Little-Elephant

    I'll be the second one.

  6. Lyisko commented on Jun 27, 2019

    @Lyisko

    I'll be the third one.

  7. vlayon commented on Feb 25, 2020

    @vlayon

    Hello there.
    I want to ask also for implementing SoftAsserts in Allure. Are you going to do it?

    ..because now the bahavior is : current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

  8. bereg2k commented on Nov 3, 2020

    @bereg2k

    Yep, that's an important feature.

  9. Guter1988 commented on Jan 19, 2021

    @Guter1988

    I use it too.
    And due to this, add attachments when it failed to find a better solution.

  10. sskorol commented on Jan 19, 2021

    @sskorol
    Contributor

    Can anyone explain the expected behavior and attach the minimal code to reproduce the issue? As I don't really understand what kind of SA support is required.

  11. Guter1988 commented on Jan 21, 2021

    @Guter1988

    I will try from my point of usage:
    When in the an of API (for example), I'm checking the JSON response, I need to test all fields that exist as expected.
    I can run soft assertion or run a new test to validate one filed.
    Or in UI - when by click on an element will be an open window with 3 parameters that I need to validate, I will use SA.

    I hope it helped :)

  12. sskorol commented on Jan 21, 2021

    @sskorol
    Contributor

    @guterscooter I know how SA works. I wonder what do you expect to see in Allure while using SA?

  13. Guter1988 commented on Jan 24, 2021

    @Guter1988

    @sskorol To see all parts where it failed.
    I see only the first one, while the next can failed will not be shown at Allure.

  14. twixdouble commented on Feb 17, 2021

    @twixdouble

    @sskorol mark each failed step (and each of its ancestors) with red color, add assertion message within failed step content, add stack-trace within assertion error content, add summary error-message as test result, e.g. 'N assertions failed: <error 1> (expandable with stack-trace), <error 2> (expandable) ...

  15. lbastil commented on Nov 9, 2021

    @lbastil

    @sskorol: I think in #529 everything is explained quite well:

    What is the current behavior?
    current step, which has failed soft assert in it, does not fail. The case fails when assertAll method is called.

    What is the expected behavior?
    hope it show failed in each step and make the test failed

    Example + Expectation:

    On a very general level an example would be:

    public class MyTestClass {
    
            private SoftAssert softAssert;
        
            @BeforeMethod(alwaysRun = true)
            public void initialize(Method method) {
        
                softAssert = new SoftAssert();
            }
        
            @Test
            public void myComplexTest_1() {
        
                reusableTestcasePart_1();
                reusableTestcasePart_2();
                softAssert.assertAll();
            }
        
            @Step("...")
            protected void reusableTestcasePart_1() {
                // do something directly ...
        
                // use reusable Steps:
                testStep_1();
                testStep_2();
            }
        
            @Step("...")
            protected void reusableTestcasePart_2() {
                // do something directly ...
        
                // use reusable Steps:
                testStep_3();
                testStep_4();
            }
        
            @Step("...")
            protected void testStep_1() {
        
                boolean stepResult = true;// in real case check some sophisticated ui part
                // soft assertion
                softAssert.assertTrue(stepResult);
            }
        
            @Step("...")
            protected void testStep_2() {
        
                boolean stepResult = false;// in real case check some sophisticated ui part
                // soft assertion
                softAssert.assertTrue(stepResult);
            }
        
            @Step("...")
            protected void testStep_3() {
        
                boolean stepResult = false;// in real case check some sophisticated ui part
                // soft assertion
                softAssert.assertTrue(stepResult);
            }
        
            @Step("...")
            protected void testStep_4() {
        
                boolean stepResult = true;// in real case check some sophisticated ui part
                // soft assertion
                softAssert.assertTrue(stepResult);
            }
        }

    One would expect in this case the following report result:

    • (test method) myComplexTest_1 : FAILED
      • (step) reusableTestcasePart_1 : FAILED
        • (step) testStep_1 : SUCCESS
        • (step) testStep_2 : FAILED
      • (step) reusableTestcasePart_2 : FAILED
        • (step) testStep_3 : FAILED
        • (step) testStep_4 : SUCCESS

    But it is currently:

    • (test method) myComplexTest_1 : FAILED
      • (step) reusableTestcasePart_1 : SUCCESS
        • (step) testStep_1 : SUCCESS
        • (step) testStep_2 : SUCCESS
      • (step) reusableTestcasePart_2 : SUCCESS
        • (step) testStep_3 : SUCCESS
        • (step) testStep_4 : SUCCESS

    Which is misleading and hard to review for the test team which evaluates the allure report details

  16. lbastil commented on Nov 9, 2021

    @lbastil

    @sskorol : if I can I would try to support the feature enhancement. Can you give me some advise about entry points where to start? I guess there is a kind of post-processing for the final test / step state (updating the TestResult information).

  17. deisenberg-remedy commented on Jun 12, 2023

    @deisenberg-remedy

    What is the status on implementing Soft Assertion support as described above?

  18. Phlegethonyarre commented on Jul 21, 2023

    @Phlegethonyarre

    Hi guys. A little workaround here I created long time ago, this could be helpful for your own ideas, probably with aspects :)

    public class StepUtil {
    
    	private final List<Throwable> troubles = new ArrayList<>();
    	private final List<String> causes = new ArrayList<>();
    
    	public void runStep(Object name, Runnable runnable) {
    		final String uuid = UUID.randomUUID().toString();
    		final List<String> localCauses = new ArrayList<>();
    		final List<Throwable> localExceptions = new ArrayList<>();
    
    		try {
    			Allure.getLifecycle().startStep(uuid, new StepResult().setName(name.toString()));
    			runnable.run();
    		} catch (Throwable e) {
    			localCauses.add(String.format("\r\nFailed step name: %s", name.toString()));
    			localExceptions.add(e);
    		}
    
    		if (!localExceptions.isEmpty()) {
    			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.FAILED));
    			troubles.addAll(localExceptions);
    			causes.addAll(localCauses);
    		} else {
    			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.PASSED));
    		}
    
    		Allure.getLifecycle().updateStep(step -> {
    			List<StepResult> stepResults = step.getSteps();
    			if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.FAILED))) {
    				step.setStatus(Status.FAILED);
    			} else if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.BROKEN))) {
    				step.setStatus(Status.BROKEN);
    			}
    		});
    
    		Allure.getLifecycle().stopStep(uuid);
    	}
    
    	public void checkTroubles() {
    		if (!troubles.isEmpty()) {
    			throw new AssertJMultipleFailuresError(causes.toString(), troubles);
    		}
    	}
  19. Achitheus commented on Nov 10, 2023

    @Achitheus

    Hi guys. A little workaround here I created long time ago, this could be helpful for your own ideas, probably with aspects :)

    public class StepUtil {
    
    	private final List<Throwable> troubles = new ArrayList<>();
    	private final List<String> causes = new ArrayList<>();
    
    	public void runStep(Object name, Runnable runnable) {
    		final String uuid = UUID.randomUUID().toString();
    		final List<String> localCauses = new ArrayList<>();
    		final List<Throwable> localExceptions = new ArrayList<>();
    
    		try {
    			Allure.getLifecycle().startStep(uuid, new StepResult().setName(name.toString()));
    			runnable.run();
    		} catch (Throwable e) {
    			localCauses.add(String.format("\r\nFailed step name: %s", name.toString()));
    			localExceptions.add(e);
    		}
    
    		if (!localExceptions.isEmpty()) {
    			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.FAILED));
    			troubles.addAll(localExceptions);
    			causes.addAll(localCauses);
    		} else {
    			Allure.getLifecycle().updateStep(uuid, (step) -> step.setStatus(Status.PASSED));
    		}
    
    		Allure.getLifecycle().updateStep(step -> {
    			List<StepResult> stepResults = step.getSteps();
    			if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.FAILED))) {
    				step.setStatus(Status.FAILED);
    			} else if (stepResults.stream().anyMatch(stepResult -> stepResult.getStatus().equals(Status.BROKEN))) {
    				step.setStatus(Status.BROKEN);
    			}
    		});
    
    		Allure.getLifecycle().stopStep(uuid);
    	}
    
    	public void checkTroubles() {
    		if (!troubles.isEmpty()) {
    			throw new AssertJMultipleFailuresError(causes.toString(), troubles);
    		}
    	}

    Maybe I'm wrong, but it looks like the run() method never throws errors because it has a soft assertion inside?

  20. Achitheus commented on Nov 10, 2023

    @Achitheus

    I did this for my needs:

    public class AllureSelenideCustom {
        public static void markOuterStepAsFailedAndStop() {
            AllureLifecycle lifecycle = getLifecycle();
            lifecycle.updateStep(step -> step.setStatus(Status.FAILED));
            lifecycle.stopStep();
        }
    
        public static void stepSoftAssert(final String passMessage, final String failMessage,
                                          final Consumer<String> consumer, final boolean markStepAsFailed) {
            Allure.step(markStepAsFailed ? failMessage : passMessage
                    , () -> {
                        consumer.accept(failMessage);
                        if (markStepAsFailed) {
                            markOuterStepAsFailedAndStop();
                        }
                    });
        }
    }
    

    The bad new is you should pass to this method boolean markStepAsFailed. It means you should additionally ("manually") check result before use it.
    Its ok for me in cases when I looking for "bad items" and pass as argument something like badItem.exists()

  21. added theissue type on Jul 10, 2026
  22. removed
    pr:new featureChange that add something new for end users
    on Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions