Skip to main content

Test Status Update in Smart Agent

Test Status Update ensures that the final test result displayed in Reporter accurately reflects the actual outcome reported by your test framework, even when it differs from the Appium session result.

With Smart Agent, a test's final status is determined by two sources:

  • The outcome of the Appium session on the device.
  • The pass or fail result reported by your test framework (JUnit or TestNG).

Normally, these two results are the same. However, in some situations they can differ. Test Status Update resolves these differences by ensuring that Reporter always displays the final result reported by your test framework.

Why Test Status Update is needed​

Before Test Status Update, Cloud Reporter determined a test's status solely from the Appium session running on the device. As a result, the status shown in Reporter differed from the actual outcome of the test.

For example:

  • The Appium session completes successfully, but a later assertion fails in the test framework.
  • A driver exception occurs during the Appium session, but the test intentionally catches the exception and completes successfully.

In these situations, the Appium session status does not represent the actual test outcome.

The following table illustrates the difference.

Appium sessionTest frameworkReporter before Test Status UpdateReporter with Test Status Update
PassedFailed (assertion failed)PassedFailed
Failed (driver exception)Passed (exception handled)FailedPassed

Test Status Update eliminates these inconsistencies by synchronizing the final test result with the outcome reported by your test framework. As a result, the Reporter always displays the same final status as your JUnit or TestNG test.

Prerequisites​

Test Status Update is available when all of the following conditions are met:

  • Tests are executed using Smart Agent.
  • The test framework is JUnit or TestNG.
  • The Appium session has already completed.

No additional configuration is required.

How Test Status Update works​

When a test finishes executing, Smart Agent automatically synchronizes the final test status with the Cloud Reporter.

The synchronization process is as follows:

  1. The Appium session completes and the server records the session status.
  2. JUnit or TestNG determines the final outcome of the test.
  3. Smart Agent compares the Appium session status with the test framework result.
  4. If the results differ, Smart Agent updates the completed report to match the test framework result.
  5. Reporter displays the synchronized test status.

This process is automatic and requires no user intervention.

Supported frameworks​

FrameworkStatus
JUnitSupported
TestNGSupported

What gets synchronized​

When Smart Agent detects a difference between the Appium session status and the test framework result, it synchronizes:

  • The final test status (Passed or Failed)
  • Failure details and exception information, when applicable

The synchronized result is reflected in Reporter.

Example scenarios​

Scenario 1 — Assertion fails after the Appium session completes successfully​

Your test launches the application, logs in successfully, performs all required device interactions, and closes the Appium session without errors.

After the session ends, the test verifies that the expected welcome message is displayed.

assertEquals("Welcome John", actualWelcomeText);

The application displays "Welcome User" instead, causing the assertion to fail.

Result

ComponentResult
Appium sessionPassed
Test frameworkFailed
ReporterFailed
AnalyticsFailed

Although the Appium session completed successfully, the overall test failed because the final assertion did not pass. Smart Agent updates Reporter and Analytics so they display the correct test result.

Scenario 2 — A handled driver exception does not fail the test​

Your test intentionally performs an invalid operation to verify that the application handles an error correctly.

The driver throws an exception, but the exception is expected and handled by the test.

try {
    driver.findElement(By.id("invalidElement")).click();
} catch (Exception e) {
    // Expected exception
}

The remaining test steps complete successfully.

Result

ComponentResult
Appium sessionFailed
Test frameworkPassed
ReporterPassed
AnalyticsPassed

Although the Appium session recorded a driver error, the test framework considered the test successful because the exception was expected. Smart Agent synchronizes the final result so Reporter and Analytics correctly display the test as Passed.

Limitations​

  • Test Status Update is available only for tests executed using Smart Agent.
  • Only JUnit and TestNG are currently supported.
  • Update occurs only after the Appium session has completed.
  • Test Status Update updates only the final test status and failure information. Screenshots, videos, logs, and other execution artifacts are not modified.

Important Notes​

  • No additional configuration is required.
  • If the Appium session status and test framework result already match, no synchronization is performed.
  • Reporter always displays the final synchronized result after the update is complete.

Frequently asked questions​

What happens if the Appium session result and the test framework result are the same?​

No synchronization is required. Reporter continues to display the existing test result.

Does Test Status Update require additional configuration?​

No. The feature works automatically when tests are executed using Smart Agent with JUnit or TestNG.

Does Test Status Update modify execution artifacts?​

No. Test Status Update updates only the final test status and related failure information. Screenshots, logs, videos, and other execution artifacts remain unchanged.

Last updated on