Parallel Tests Execution

One of the advantages of executing tests in parallel is it speeds up the execution time of Tests which in turn facilitates speeding up the Continuous Integration Process.

Parallelism in Continuous Testing Cloud

Continuous Testing cloud and hosted services supports parallelism of Appium or Selenium tests by the implementation of SeeTestGrid the grid is an extension of the current Selenium Grid model, and it supports queuing tests and allocating agents based of desired capabilities to a test Session. seetest.io has the capability to run concurrent tests for one or more users in your project.

Grid Execution provides more details on this concept. 

Parallelism in Automated Tests

Just having the support of parallelism in the Continuous Testing is not enough. In order to execute tests in parallel for a specific client session,

• Multi-Threading support of the language in which tests are developed can be used. 
  or
• Parallel execution feature of Test Framework using which tests are developed can be made use of and is the preferred solution.

We also need to ensure that to develop tests which can be executed simultaneously, we need to develop tests adhering to some best practices.

They are:

• Independent Tests Methods: Tests should be independent of each other.

• Usage of thread safe references: Tests should use thread safe variables. For example, avoid the usage of static references in the tests.

• Repeatable: Tests should return always the same results for the same version of the application and test inputs.

As an example, let's discuss the support of parallel execution of tests in the TestNG framework. Concepts discussed in the section below will kick-start the process of building a reliable Parallel Tests.

Parallelism Using TestNG

Quick and Easy Way

Now that you are all set with basics of Parallel test execution, try out a sample to demonstrate these concepts. 

1. Find the sample test in our git repository.

2. Fork the repository.

3. Follow the steps in the Readme file packaged in the sample project.

4. Run the test using the command gradlew runTestsParallel.

tip

Before you proceed with fetching the sample tests:

1. If you plan to test a native or hybrid app - Upload your app to your project.
2. Fetch your access key.

Further Reading 

Before we discuss parallel execution, here are some basics of TestNG.

TestNG provides a parallel attribute in testng.xml. Basically, this is a way to instruct TestNG's engine to run tests in multiple threads. 

The behavior of the thread execution depends upon what value this attribute is assigned in testng.xml.

Value	Definition Example in testng.xml	Description
methods	<``suite name``=``"My suite" parallel``=``"methods">	Run all your test methods in separate threads.
tests	<``suite name``=``"My suite" parallel``=``"tests"``>	Run all the methods in the same <test> tag in the same thread
classes	<``suite name``=``"My suite" parallel``=``"classes">	Run all the methods in the same class in the same thread, but each class will be run in a separate thread.
instances	<``suite name``=``"My suite" parallel``=``"instances"``>	Run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads

Here is a simple example of testng.xml with parallel attribute.

TestNG with parallel attribute

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestNGRunParallelClassesExample" parallel="classes" verbose="2">
   <!-- preserve-order=true ensures that the order is  -->
   <test name="TestNGRunParallelClassesExample">
      <classes>
         <class name="io.appium.testng.TestNGRunParallelClassesExample">
      </classes>
   </test>
</suite>

Although this looks simple, the onus is on Test Developers to develop robust and reliable parallel execution test mechanism. Let's take an example to elaborate on this.

Consider the following scenario:

The parallel attribute is defined with a value of methods. This means that every test method defined in TestNG class will be executed in a separate thread.

Parallel at mehods level

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestNGRunParallelClassesExample" parallel="methods" verbose="2">
   <!-- preserve-order=true ensures that the order is  -->
   <test name="TestNGRunParallelClassesExample">
      <classes>
         <class name="io.appium.testng.TestNGRunParallelClassesExample">
      </classes>
   </test>
</suite>

Appium or Selenium Driver is set up using the @Beforeclass Annotation. This means that the Driver is set up on once for a TestNG class.

WebDriver Creation in @BeforeClass

public class TestBase {
	protected AndroidDriver<AndroidElement> driver = null;
		@BeforeClass
		public void setUp() {
			driver = new AndroidDriver<>(new URL("https://cloud.seetest.io/wd/hub"), dc);
		}
		@Test
		public void test1() {
			driver.findelement(By.xpath("//*[@id='test1']"));
    	} 
		@Test
		public void test2() {
			driver.findelement(By.xpath("//*[@id='test2']"));
    	} 
}

The test developed using the strategy above will not give reliable results.

This is because the Appium Driver created in @BeforeClass annotated function is shared in all the test methods. However, every test method is run in separate thread simultaneously. Sharing the same Driver instance makea it thread-unsafe resulting in unreliable results.

In such cases, it is best to use a parallel attribute with "classes" as value. This ensures that all test methods are run in same thread within a TestNG class. 

If you choose to parallelism at methods level, Web Driver needs to be set up in every test methods to make it thread-safe. Of course, this becomes very resource intensive because every test method creates a driver instance.

WebDriver at @Test

public class TestBase {

	private AndroidDriver<AndroidElement> driver = null;
	
		@Test
		public void test1() {
			AndroidDriver<AndroidElement> driver = new AndroidDriver<>(new URL("https://cloud.seetest.io/wd/hub"), new DesiredCapabilities());
			driver.findelement(By.xpath("//*[@id='test1']"));
    	} 

		@Test
		public void test2() {
			AndroidDriver<AndroidElement> driver = new AndroidDriver<>(new URL("https://cloud.seetest.io/wd/hub"), new DesiredCapabilities());
			driver.findelement(By.xpath("//*[@id='test2']"));
    	} 
}