Skip to main content

Test Orchestrator Overview

Test Orchestrator is a Java-based tool that orchestrates automated mobile tests. It acts as an intermediary between your test project and the Digital.ai Testing cloud, enabling dynamic management of execution parameters without code changes.

It eliminates repetitive updates to test scripts for changes like app versions or device queries. This approach keeps automation suites agile and maintainable at scale.

The Test Orchestrator is distributed as a JAR file. Add it to your project as a JVM agent to instrument test execution at runtime. It uses a configuration file (config.yml) to define shared properties and capabilities that are applied consistently across all tests, eliminating the need to modify individual test scripts.

Key Capabilities​

Centralized Configuration (YAML-based)​

Define common test capabilities in a single configuration file and apply them across all tests.

  • Update Appium version, device query, application, and cloud settings globally
  • Avoid modifying individual test scripts
  • Ensure consistency across test runs

Selective Test Execution​

Filter and run specific test classes or individual methods directly through configuration, avoiding the need to trigger entire suites for small changes. Run only the tests that matter.

  • Execute specific test classes or methods
  • Exclude unnecessary tests
  • Optimize execution time for targeted validation

Automated Retry Mechanism​

Automatically rerun failed tests a specified number of times to account for flaky tests, network glitches, or environmental issues.

  • Configure retry attempts using a single property
  • Reduce noise from flaky failures (e.g., network issues)
  • Improve test reliability in CI pipelines

Build & Run Tracking​

Assign unique Build IDs and Run IDs to executions, enabling granular filtering and reporting within the Digital.ai Cloud Reporter. Track and filter test executions across environments.

  • Build ID: Groups tests under a specific CI/CD run
  • Run ID: Identifies individual Test Orchestrator executions

This enables:

  • Filtering test results by build
  • Debugging failures across executions
  • Connecting CI tools (e.g., TeamCity) with test results

Enhanced Logging​

Test Orchestrator generates detailed execution logs:

  • Test execution flow
  • Retry attempts and outcomes
  • Failure reasons

Logs can be shared for faster debugging and support.

Prerequisites​

  • Java Version: Java 17 or 21 (Support for Java 25 is in progress).
  • Build Tool: Gradle or Maven.
  • Frameworks: JUnit 5 (Jupiter Engine) or TestNG (v7+).
  • Digital.ai Account: A valid access key and URL for a Digital.ai Testing cloud environment.
  • Appium Client Version - 8,9,10.
note

The Appium Client version is managed within your automation project dependencies (e.g., build.gradle) and is not configured via the Digital.ai YAML file.

How It Works​

The Test Orchestrator operates as a JVM agent. When you trigger test execution via Gradle or Maven, it instruments the test runner at runtime. It intercepts test initialization, applies the capabilities defined in your YAML config file, and manages execution based on your retry and selection criteria.

  1. Add the Test Orchestrator JAR to your project
  2. Configure your YAML file with required properties
  3. Update your Gradle test task:
    • Add the path to the Test Orchestrator JAR
    • Add path to YAML config file
    • (Optional) Set Build ID for tracking
  4. Run your tests using your CI tool or local

Test Orchestrator End-to-End Workflow​

You can follow this workflow to configure and run Test Orchestrator in your automation project.

Step 1: Obtain the Test Orchestrator JAR​

The Test Orchestrator is currently distributed directly by the Digital.ai Testing team. To request the agent for your organization:

  • Contact your Digital.ai representative or open a request via the Digital.ai Support Portal.
  • Provide your specific project requirements, including the framework (JUnit/TestNG) and Java version in use.
  • The team will provide the smart-agent.jar file and the necessary documentation for your specific integration environment.

The distributed JAR retains the smart-agent.jar filename; the documentation rename does not change the artifact name.

info

Reach out to the Digital.ai Testing team to obtain the most current version of the Test Orchestrator. They will ensure you have the appropriate JAR file and verify compatibility with your existing testing environment.

Step 2: Add the Test Orchestrator to Your Project​

Copy the smart-agent.jar file into your project directory.

Recommended structure:

project/
├── lib/
│   └── smart-agent.jar
├── src/
│   └── main/
│       └── java/
└── build.gradle

Step 3: Add the Test Orchestrator Configuration File​

Test Orchestrator uses a YAML configuration file (config.yml) to control execution behavior such as capabilities, filtering, and retry logic.

File Placement Add the configuration file in the same directory as the JAR:

project/
├── test-orchestrator/
│   ├── smart-agent.jar
│   └── config.yml

Minimal Example Configuration​

cloud:
  url: "https://cloud.digital.ai"
  accessKey: "<your-access-key>"

app:
  ios:
    app: "cloud:com.experitest.ExperiBank"
    bundleID: "com.experitest.ExperiBank"

run:
  maxRetryAttempts: 2
  testSelection:
    - com.example.iOSNativeLoginTest#login

This configuration supplies the required cloud connection details, defines the iOS application under test, retries failed tests up to two times, and selects a specific test method. Replace the example URL, access key, app identifiers, and test name with values from your environment.

Step 4: Download Sample Configuration​

For a complete configuration with advanced options, download the sample config.yml file.

The sample file includes:

  • iOS and Android configurations
  • Capability examples
  • Retry logic
  • Test filtering
  • Execution metadata
info

Incorrect YAML formatting (indentation or spacing issues) may cause execution failures. Validate your YAML before running tests.

Step 5: Update Your Gradle Test Task​

Modify your Gradle test task to include the Test Orchestrator as a JVM argument.

JUnit example (Gradle Groovy DSL):

test {
    jvmArgs(
      "-javaagent:<path-to-smart-agent.jar>=" +
        "<path-to-config.yml>",
            "-Dbuild.id=1234"
    )
    useJUnitPlatform()
}

TestNG example (Gradle Groovy DSL):

test {
    jvmArgs(
      "-javaagent:<path-to-smart-agent.jar>=" +
        "<path-to-config.yml>",
            "-Dbuild.id=1234"
    )
    useTestNG()
}

Step 6: Run Your Tests​

You can now run your tests using your CI tool (e.g., TeamCity) or locally. The Test Orchestrator will manage the execution based on your configuration, applying capabilities, handling retries, and generating logs.

mvn test
# or
./gradlew test

Test Orchestrator will:

  • Inject capabilities
  • Apply filtering rules
  • Retry failed tests
  • Attach execution metadata

Step 7: View Results​

After execution, view your test results in the Digital.ai Testing cloud reporter. Use the Build ID and Run ID to filter and analyze your test runs effectively.

Last updated on