Skip to main content

Test Status Update in Test Orchestrator

Test Status Update ensures that the final test result displayed in Reporter accurately reflects the actual outcome reported by your test framework, even when it differs from the Appium session result.

With Test Orchestrator, a test's final status is determined by two sources:

  • The outcome of the Appium session on the device.
  • The pass or fail result reported by your test framework (JUnit or TestNG).

Normally, these two results are the same. However, in some situations they can differ. Test Status Update resolves these differences by ensuring that Reporter always displays the final result reported by your test framework.

Why Test Status Update is needed​

Before Test Status Update, Cloud Reporter determined a test's status solely from the Appium session running on the device. As a result, the status shown in Reporter differed from the actual outcome of the test.

For example:

  • The Appium session completes successfully, but a later assertion fails in the test framework.
  • A driver exception occurs during the Appium session, but the test intentionally catches the exception and completes successfully.

In these situations, the Appium session status does not represent the actual test outcome.

The following table illustrates the difference.

Appium sessionTest frameworkReporter before Test Status UpdateReporter with Test Status Update
PassedFailed (assertion failed)PassedFailed
Failed (driver exception)Passed (exception handled)FailedPassed

Test Status Update eliminates these inconsistencies by synchronizing the final test result with the outcome reported by your test framework. As a result, the Reporter always displays the same final status as your JUnit or TestNG test.

Prerequisites​

Test Status Update is available when all of the following conditions are met:

  • Tests are executed using Test Orchestrator.
  • The test framework is JUnit or TestNG.
  • The Appium session has already completed.

No additional configuration is required.

How Test Status Update works​

When a test finishes executing, Test Orchestrator automatically synchronizes the final test status with the Cloud Reporter.

The synchronization process is as follows:

  1. The Appium session completes and the server records the session status.
  2. JUnit or TestNG determines the final outcome of the test.
  3. Test Orchestrator compares the Appium session status with the test framework result.
  4. If the results differ, Test Orchestrator updates the completed report to match the test framework result.
  5. Reporter displays the synchronized test status.

This process is automatic and requires no user intervention.

Supported frameworks​

FrameworkStatus
JUnitSupported
TestNGSupported

What gets synchronized​

When Test Orchestrator detects a difference between the Appium session status and the test framework result, it synchronizes:

  • The final test status (Passed or Failed)
  • Failure details and exception information, when applicable

The synchronized result is reflected in Reporter.

Example scenarios​

Scenario 1 — Assertion fails after the Appium session completes successfully​

Your test launches the application, logs in successfully, performs all required device interactions, and closes the Appium session without errors.

After the session ends, the test verifies that the expected welcome message is displayed.

assertEquals("Welcome John", actualWelcomeText);

The application displays "Welcome User" instead, causing the assertion to fail.

Result

ComponentResult
Appium sessionPassed
Test frameworkFailed
ReporterFailed
AnalyticsFailed

Although the Appium session completed successfully, the overall test failed because the final assertion did not pass. Test Orchestrator updates Reporter and Analytics so they display the correct test result.

Scenario 2 — A handled driver exception does not fail the test​

Your test intentionally performs an invalid operation to verify that the application handles an error correctly.

The driver throws an exception, but the exception is expected and handled by the test.

try {
    driver.findElement(By.id("invalidElement")).click();
} catch (Exception e) {
    // Expected exception
}

The remaining test steps complete successfully.

Result

ComponentResult
Appium sessionFailed
Test frameworkPassed
ReporterPassed
AnalyticsPassed

Although the Appium session recorded a driver error, the test framework considered the test successful because the exception was expected. Test Orchestrator synchronizes the final result so Reporter and Analytics correctly display the test as Passed.

Limitations​

  • Test Status Update is available only for tests executed using Test Orchestrator.
  • Only JUnit and TestNG are currently supported.
  • Update occurs only after the Appium session has completed.
  • Test Status Update updates only the final test status and failure information. Screenshots, videos, logs, and other execution artifacts are not modified.

Important Notes​

  • No additional configuration is required.
  • If the Appium session status and test framework result already match, no synchronization is performed.
  • Reporter always displays the final synchronized result after the update is complete.

Frequently asked questions​

What happens if the Appium session result and the test framework result are the same?​

No synchronization is required. Reporter continues to display the existing test result.

Does Test Status Update require additional configuration?​

No. The feature works automatically when tests are executed using Test Orchestrator with JUnit or TestNG.

Does Test Status Update modify execution artifacts?​

No. Test Status Update updates only the final test status and related failure information. Screenshots, logs, videos, and other execution artifacts remain unchanged.

Last updated on