How do I stop selenium automation if 20% or 1st 20 test cases test methods are failed?


Problem:

I have 100 test cases, if 20% or first 20 test cases all fail how can I stop the execution ? Already have testng ITestResult where should I break the build?

@Test(retryAnalyzer = ReTryFail.class, dataProvider = "SanityTCTest", dataProviderClass = utility.Xlsdataprovider.class, groups = "Dashboard", alwaysRun = true)
public void Sanity_TC001(LinkedHashMap<String, String> data) throws InterruptedException, SQLException {

 Some code 
}

@Test(retryAnalyzer = ReTryFail.class, dataProvider = "SanityTCTest", dataProviderClass = utility.Xlsdataprovider.class, groups = "Dashboard", alwaysRun = true)
public void Sanity_TC002(LinkedHashMap<String, String> data) throws InterruptedException, SQLException {

 Some code 
}

@Test(retryAnalyzer = ReTryFail.class, dataProvider = "SanityTCTest", dataProviderClass = utility.Xlsdataprovider.class, groups = "Dashboard", alwaysRun = true)
public void Sanity_TC003(LinkedHashMap<String, String> data) throws InterruptedException, SQLException {

 Some code 
}
///////////////////////////////

Where can I break this suite if Result, "FAIL is over 20 ? DO i require to create new class or can I add in same below?

@AfterMethod(alwaysRun = true)
public void reporterDataResults(ITestResult Result) throws InterruptedException {
    boolean flag = false;
    Testfail = TestResultStatus.Testfail;

    /*System.out.println("test fail flag in AfterMethod: " + Testfail); */
    if (Result.getStatus() == ITestResult.SKIP) {
        Resmark.put(Result.getName(), "");
        captureScreenShot(Result, "SKIP", getDriver());
        Reporter.log(Result.getName() + " is SKIPPED");
        Add_Log.info(Result.getName() + " is SKIPPED");
        TestResultTL.put(Result.getName(), "SKIP");
        if (!(getDriver() == null)) {
            closeWebBrowser();
        }
    } else if (Result.getStatus() == ITestResult.FAILURE) {

        Collection<String> values = TestResultStatus.mpaskeysnew.get(Result.getName());
        String resultout = String.join(" | ", values);
        System.out.println(resultout);
        Resmark.put(Result.getName(), resultout);

        captureScreenShot(Result, "FAIL", getDriver());
        Reporter.log(Result.getName() + " is FAIL");
        Add_Log.info(Result.getName() + " is FAIL");
        if (!(getDriver() == null)) {
            closeWebBrowser();
        }
        TestResultTL.put(Result.getName(), "FAIL");


    } else {
        captureScreenShot(Result, "PASS", getDriver());
        Resmark.put(Result.getName(), "");
        Reporter.log(Result.getName() + " is PASS");
        Add_Log.info(Result.getName() + " is PASS");
        if (!(getDriver() == null)) {
            closeWebBrowser();
        }
        TestResultTL.put(Result.getName(), "PASS");

    }
    Testskip = false;
    TestResultStatus.Testfail = false;

}

Solution:

You can implement ISuiteListener and in onFinish method you'll have access to ISuite and ISuiteResult

Then you can do

public void onFinish(ISuite suite) {

        final Map<java.lang.String,ISuiteResult>  res = suite.getResults();

                for (ISuiteResult r : res.values()) {
                  context =  r.getTestContext()  ;
                  failedTestCases =context.getFailedTests().size();
                 }
         }

size() will give you number of failed test for that suite. Once you know that number you can implement to stop execution using strategies in this

If your test cases are in different suites then in each call to onFinish method you can count number of failed test cases per suite and based on that stop execution.

One other alternative is to implement ITestListener . In onTestFailure method you have access to ITestResult

You can count how many times onTestFailure method is called and based on that stop execution. I think implementing ITestListener is more suitable and easy in your case.

Here , I edited to explain how you'd implement listener

import org.testng.ISuiteListener;

public class listener implements Itestlistener {
            public int i = 0;

             public void onTestFailure(ITestResult result) {

                     result.getName();
                     i++;
                     //your break logic goes here
                       if (i ==20){
                              // do something or call some function to stop execution
                             }
         }

     }

You can read more about testng listeners here .

For your EDIT above (if you want to go that way). Though I still think you should implement listener, which is more cleaner. It will be called only when the test fails.

But do the same thing as I did in onTestFailure method , add a counter and increase it in else if .

   public int i = 0;  //do this in your class

then in your method

 else if (Result.getStatus() == ITestResult.FAILURE) {
    i++;  //increase counter here
    Collection<String> values = TestResultStatus.mpaskeysnew.get(Result.getName());
    String resultout = String.join(" | ", values);
    System.out.println(resultout);
    Resmark.put(Result.getName(), resultout);

    captureScreenShot(Result, "FAIL", getDriver());
    Reporter.log(Result.getName() + " is FAIL");
    Add_Log.info(Result.getName() + " is FAIL");
    if (!(getDriver() == null)) {
        closeWebBrowser();
    }
       TestResultTL.put(Result.getName(), "FAIL");
      if (i==20){
         // stop execution here
      }

  }

Recent Tips

  1. Sharepoint with silverlight app
  2. What's the best way to write robots.txt for github pages using multiple repos?
  3. Biopython: Cant use .count() for biopython
  4. How can I find out the token balance of an address?
  5. ref value is undefined in vue (modal, textarea, $refs)
  6. Azure - HDInsight Hbase Data Insertion Failed
  7. SignalR overwriting OnConnected(), OnDisconnected()
  8. DatePickerDialog displays with two borders
  9. "type 'double' is not a subtype of type 'int' in type cast" error in flutter. What should i do?
  10. hiding the autocomplete list when user click outside the textbox is not working as expected
  11. JSF IceFaces basic problem with redisplaying input value
  12. How to validate material ui TextField in reactjs?
  13. Go and MongoDB connection won't work with panic log "no reachable server"
  14. WordPress Posts Pagination Not Working
  15. F# sprintf won't print in interactive console
  16. Spring Integration get FTP files recursively with outbound-gateway
  17. Jade mixins not getting working from external file
  18. Can not access defined exports from the webpack bundle?
  19. Completely new to Node.js - API Programming
  20. Formatting Compare-Object Ouput
  21. Add dynamically added textbox value from User Control to main form
  22. Create a ByteBuf in Netty 4.0
  23. Is it possible to do computation before super() in the constructor?
  24. Q-learning Updating Frequency
  25. Wrong reload order when using Gulp and browserSync
  26. I use hugo build static page. But don't have content
  27. How to change background color and set bar colors based on conditional formatting in VBA?
  28. Problem when comparing two numeric values in SAS
  29. Is ACE reactor timer managment thread safe?
  30. Why Express res.render dumps the render output (EJS template) in console?