Skip to main content

Reporting and Client Side Failures

Introduction​

During an automation test, Assertion or Exceptions can be thrown outside any context of Mobile Studio.

This fails the test but because the exception isn't happening during any execution in the Cloud Server, the test status will be displayed as Passed instead of Failed.

For example, let us consider the following test case. 

DeviceByZeroTest

@Test(priority = 1)
public void dividedByZero() {
    System.out.println(">>>>>> I'm in method divided by zero");
    int e = 1 / 0;
}

As we can see there is a divide by zero error, the cause for it was purely based on the client code with no involvement of the Cloud Server.

However, if we observe in the Cloud Portal test result, there are no errors and the report status will be displayed as "Passed". 

Test Results without Listeners - Test will be displayed as Passed, on any client-side errors.​

In such situations, Continuous Testing recommends integrating Reporting Listeners into the Tests. These listeners will report to the "Grid reporter" any exception or failed assertion and mark the test as failed.

Test Results with Listeners - Test will be displayed as Failed, on any client-side errors.​

Setup​

Each testing framework has been split into 3 jars. Typically these jars are bundled when you install Appium Studio 

  • TestNG 6.9.10
  • Junit 4.12
  • Junit 5.2.0

Libraries bundled with SeeTest Client​

Libraries are located under the client folder reporter <SeeTest installation folder>\clients\reporter\java

  • reporter-testng.jar
  • reporter-junit4.jar
  • reporter-junit5.jar

Code Usage​

TestNG​

There are many ways to configure a listener in TestNG.

Create a build.gradle file and configure dependency

TestNG Gradle Dependency

dependencies {
	compile group: 'org.testng', name: 'testng', version: '6.9.10'
        compile fileTree(dir: '<seeTest\Appium Installation folder>\\clients\\reporter\\java', include:'reporter*.jar')

	// Appium Driver
	compile group: 'io.appium', name: 'java-client', version: '5.0.4'
	compile fileTree(dir: '<seeTest Installation folder>\\clients\\appium\\java', include: '*.jar')
}

Implement a TestNG Class (in this example 'AppiumDriver') with setup and teardown function.  See the @Listeners({com.experitest.reporter.testng.Listener.class}) annotation below in code. Listener communicates to Grid Reporter the Client side failure.

Test Class

package com.experitest.testng;
.....
@Listeners({com.experitest.reporter.testng.Listener.class})
public class AppiumDriver {
    protected AndroidDriver driverAndroid;
    @BeforeMethod
    public void setup(){
	    System.out.println(">>>>>> I'm in Setup");
            DesiredCapabilities dcAdroid = new DesiredCapabilities;
	    dcAdroid.setCapability("deviceQuery", "@os='android'");
            // Please replace YOUR_CLOUD_ACCESSKEY with your cloud access key
            dcAdroid.setCapability("accessKey", "YOUR_CLOUD_ACCESSKEY");
            dcAdroid.setCapability(MobileCapabilityType.APP, "cloud:com.experitest.ExperiBank/.LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_ACTIVITY, ".LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_PACKAGE, "com.experitest.ExperiBank");
		
	    String url = new URL("http://" + HOST_NAME_GRID + "8080:/wd/hub");
	    System.out.println(dcAdroid);
	    driverAndroid = new AndroidDriver(url, dcAdroid);	
    }
	
    
    @AfterMethod
    public void tearDown() {
	    System.out.println(">>>>>> Before closing driver");
	    driverAndroid.quit();
    	    driverAndroid = null;
	    System.out.println(">>>>>> Reports generated");
    }

Implement test functions and include in TestNG class above. See divideByZero function which produces and client-side error.

Test methods

@Test(priority = 0)
public void methodPass() {
    System.out.println(">>>>>> I'm in method pass");
}

@Test(priority = 1)
public void dividedByZero() {
    System.out.println(">>>>>> I'm in method divided by zero");
    int e = 1 / 0;
}

@Test(dependsOnMethods = { "dividedByZero" })
public void methodSkip() {
    System.out.println(">>>>>> I'm in method skip");
}


info

If the test passes in the TestNG framework, the test will be reported as passed, even if a step (or more) had failed in the test itself.

JUnit4 using Runner​

Configure Gradle Dependency

Gradle Junit4 Dependency

//JUnit4
dependencies {
    compile group: 'junit', name: 'junit', version: '4.12'
    compile '<seeTest Installation folder>\\clients\\reporter\\java\\reporter-junit4.jar'

    // Appium Driver
    compile group: 'io.appium', name: 'java-client', version: '5.0.4'
    compile fileTree(dir: '<seeTest Installation folder>\\clients\\appium\\java', include: '*.jar')
}

First set the annotation relevant to the runner as a class annotation

Class Anotation @RunWith

@RunWith(ReporterRunner.class)
public class SeeTestAutomationTestClass {
.....

Then make sure you have one of the following class property, the name of the property is not important

Class Property

	// And/OR
    protected SeeTestClient seeTestClient;
	// And/OR
    protected AndroidDriver driverAndroid;
	// And/OR
    protected IOSDriver driverIOS;

Initialize the driver as you would usually do

Setup Driver

	
    @Before
    public void setup(){
	    System.out.println(">>>>>> I'm in Setup");
            DesiredCapabilities dcAdroid = new DesiredCapabilities;
            dcAdroid.setCapability("deviceQuery", "@os='android'");
            // Please replace YOUR_CLOUD_ACCESSKEY with your cloud access key
            dcAdroid.setCapability("accessKey", "YOUR_CLOUD_ACCESSKEY");
	    dcAdroid.setCapability(MobileCapabilityType.APP, "cloud:com.experitest.ExperiBank/.LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_ACTIVITY, ".LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_PACKAGE, "com.experitest.ExperiBank");
		
	    String url = new URL("http://" + HOST_NAME_GRID + "8080:/wd/hub");
	    System.out.println(dcAdroid);
	    driverAndroid = new AndroidDriver(url, dcAdroid);
    }
  
    
    @After
    public void tearDown() {
	    System.out.println(">>>>>> Before closing driver");
	    driverAndroid.quit();
    	    driverAndroid = null;
            System.out.println(">>>>>> Reports generated");
    }


Run a test that generates an exception

Test method

@Test
public void dividedByZero() {
    System.out.println(">>>>>> I'm in method divided by zero");
    int e = 1 / 0;
}

JUnit4 using a Rule​

Set Gradle Dependency

Gradle Junit4 Dependency

//JUnit4
dependencies {
    compile group: 'junit', name: 'junit', version: '4.12'
    compile '<seeTest Installation folder>\\clients\\reporter\\java\\reporter-junit4.jar'

	// Appium Driver
	compile group: 'io.appium', name: 'java-client', version: '5.0.4'
	compile fileTree(dir: '<seeTest Installation folder>\\clients\\appium\\java', include: '*.jar')
}

First set the Rule declaration

Class Anotation @RunWith

public class SeeTestAutomationTestClass {
	
  	@Rule
	public ReporterRule reporterRule = new ReporterRule(this);

Then make sure you have one of the following class property, the name of the property is not important

Class Property

	// And/OR
    protected SeeTestClient seeTestClient;
	// And/OR
    protected AndroidDriver driverAndroid;
	// And/OR
    protected IOSDriver driverIOS;

Initialize the driver as you would usually do

Setup Driver

	@Before
	public void setup(){
	    System.out.println(">>>>>> I'm in Setup");
            DesiredCapabilities dcAdroid = new DesiredCapabilities;
	    dcAdroid.setCapability("deviceQuery", "@os='android'");
            // Please replace YOUR_CLOUD_ACCESSKEY with your cloud access key
            dcAdroid.setCapability("accessKey", "YOUR_CLOUD_ACCESSKEY");
            dcAdroid.setCapability(MobileCapabilityType.APP, "cloud:com.experitest.ExperiBank/.LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_ACTIVITY, ".LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_PACKAGE, "com.experitest.ExperiBank");
		
	    String url = new URL("http://" + HOST_NAME_GRID + "8080:/wd/hub");
	    System.out.println(dcAdroid);
	    driverAndroid = new AndroidDriver(url, dcAdroid);	}

Replace all references from @After to @AfterWithSeeTestReporter

@AfterWithSeeTestReporter method

    @AfterWithSeeTestReporter
    public void tearDown() {
	    System.out.println(">>>>>> Before closing driver");
	    driverAndroid.quit();
    	driverAndroid = null;
	    System.out.println(">>>>>> Reports generated");
    }
warning

@After can't be used because it runs before we get to the part the failed status is sent to the server thus closing the driver will cause the setReporterStatus request to fail. So another annotation ( @AfterWithSeeTestReporter ) is used to replace it This will call the tearDown right after the status is sent.

Run a test that generates an exception

Test method

@Test
public void dividedByZero() {
    System.out.println(">>>>>> I'm in method divided by zero");
    int e = 1 / 0;
}

JUnit5​

Set Gradle Dependency

Gradle Junit5 Dependency

//JUnit5
dependencies {
    compile 'org.junit.jupiter:junit-jupiter-api:5.2.0'
    compile '<seeTest Installation folder>\\clients\\reporter\\java\\reporter-junit5.jar'


	// Appium Driver
	compile group: 'io.appium', name: 'java-client', version: '5.0.4'
	compile fileTree(dir: '<seeTest Installation folder>\\clients\\appium\\java', include: '*.jar')
}

First set the annotation relevant to the runner as a class annotation

Class Anotation @RunWith

@ExtendWith(SeeTestReporterExceptionExtension.class)
public class SeeTestAutomationTestClass {
.....

Then make sure you have one of the following class property, the name of the property is not important

Class Property

	// And/OR
    protected SeeTestClient seeTestClient;
	// And/OR
    protected AndroidDriver driverAndroid;
	// And/OR
    protected IOSDriver driverIOS;

Initialize the driver as you would usually do

Setup Driver

	@BeforeEach
	public void setup(){
	   System.out.println(">>>>>> I'm in Setup");
            DesiredCapabilities dcAdroid = new DesiredCapabilities;
	    dcAdroid.setCapability("deviceQuery", "@os='android'");
            // Please replace YOUR_CLOUD_ACCESSKEY with your cloud access key
            dcAdroid.setCapability("accessKey", "YOUR_CLOUD_ACCESSKEY");
            dcAdroid.setCapability(MobileCapabilityType.APP, "cloud:com.experitest.ExperiBank/.LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_ACTIVITY, ".LoginActivity");
	    dcAdroid.setCapability(AndroidMobileCapabilityType.APP_PACKAGE, "com.experitest.ExperiBank");
		
	    String url = new URL("http://" + HOST_NAME_GRID + "8080:/wd/hub");
	    System.out.println(dcAdroid);
	    driverAndroid = new AndroidDriver(url, dcAdroid);
	}
    @AfterEach
    public void tearDown() {
	    System.out.println(">>>>>> Before closing driver");
	    driverAndroid.quit();
    	driverAndroid = null;
	    System.out.println(">>>>>> Reports generated");
    }


Run a test that generates an exception

Test method

@Test
public void dividedByZero() {
    System.out.println(">>>>>> I'm in method divided by zero");
    int e = 1 / 0;
}
Last updated on