Performance Pipeline

Introduction

You can measure your application's performance (memory usage, CPU usage, battery consumption, network bandwidth usage or duration) and how this has changed over previous versions. For a given transaction name and measure you define an accepted change (in "comparison target") and if the measure has changed to be equal or below that value it is acceptable ("ok") otherwise (obviously) not acceptable ("fail"). For example, maximum memory usage should not vary, for the latest version over past versions, or more than 5% for the "Inquiry" transaction:

Latest Version	Previous Version	Change (Last Over Previous)
100M	90M	(100 - 90) / 90 = 11%

Input

Filter: application name and device OS.
Base version: The version you want to compare to older versions; can be specific version string ("7.0") or "latest".
How many previous versions will be used for the comparison?
List of accepted change % per transaction name and measure (comparison targets).

Example:

Filter:
Application name: "Banking Self-Care"
Device OS: "Android"
Base version: "7.0"
Previous versions: 2
Comparison targets:

Transaction name Measure Accepted change %
Inquiry Max Memory 10%
Inquiry Max CPU 5%
Transfer Bytes uploaded 15%
Transfer Bytes downloaded 20%

Transaction name	Measure	Accepted change %
Inquiry	Max Memory	10%
Inquiry	Max CPU	5%
Transfer	Bytes uploaded	15%
Transfer	Bytes downloaded	20%

Output

For each comparison target (transaction/measure/maximum accepted change) a row will be returned with the following properties:

Transaction name
Measure name
Base value: the average of values for the measure for the base version.
Base count: how many transactions' measures are found for the base version.
Previous value: the average of average by version (see example below) of measured values of the 2 previous versions.
Previous count: how many transactions' measures are found for the 2 previous versions.
Previous key count: how many different versions found.
Accepted change %: same as in the request.
Actual change %: change of the "Base value" over "Previous value".
Status: "ok" in case "Actual change %" <= "Accepted change %" else "fail".

Example:

You have the values of Max Memory of different transaction tests for "Banking Self-Care" on "Android" OS for "Inquiry" transaction:

Version	Max Memory
7.0	110
7.0	105
7.0	107
7.0	99
6.9	80
6.9	79
6.9	88
6.9	95
6.9	88
6.9	120
6.8	125
6.8	110
6.7	50
6.7	65

The request asked for base version "7.0", the base value is the average of those values is 105.25.

Requested previous versions are 2 which are "6.9" and "6.8", the average of those values is 91.67 and 117.5 respectively, the final average is 104.58.

Actual change: ((105.25 - 98.125) / 98.125) * 100 = 0.64

Resulting row for first acceptance criteria will be:

Transaction name: "Inquiry"
Measure name: "memMax"
Base value: 105.25
Base count: 4
Previous value: 104.58
Previous count: 8
Previous key count: 2
Accepted change %: 10.9
Actual change %: 0.64
Status: ok

Endpoint

POST /api/transactions/compare?token=<accessKey>

Because the feature is meant to be used in an automated pipeline, you must provide the accessKey. Remember to select the desired project before getting the access key.

When the Cloud is operating in single-port mode, you must add the "/reporter" context to the request, for example, if Cloud's URL is www.mycloud.com the URL for the request would be www.mycloud.com/reporter/API/transactions/compare.

Request Body Object

{
	"filter": [ ...<filter expression>... ],
	"baseKey": <string>,
	"baseKeyValue": <string value>,
	"compareCount": <integer>,
	"comparisonTargets": [
		{
			"name": <string>,
			"measure": <string>,
			"acceptedChange": <numeric>
		}, ...
	]
}

Property

Notes

filter

Simple comparison expression is a 3 element array composed of property name followed by the comparison operator ("=", ">", ">=", "<>", "<", "<=", "startswith", "endswidth", "contains") then a value, example:

[ "appName", "=", "Bank" ]

In case you have more than one comparison expression, you must include each simple comparison expression array as elements of enclosing array and include the logical operator ("and" / "or") between those two, example:

[ [ "appName", "=", "Bank" ], "and", [ "deviceOs", "=", "Android"] ]

Currently accepted left side for comparison expressions are "appName" and "deviceOs".

baseKey

Transaction's property to be used to compute average values to be compared. As today the only supported base key is appVersion.

baseKeyValue

The base key used to compute measures value which will be compared against "older" values. We only accept appVersion as baseKey, so a simpler description is "base version to be compared against previous versions". he constant string "latest" means the maximum base key ("version") found.

compareCount

How many older base key values ("versions") before base key value will be used to compute measured values to be compared against the base key value ("base version").

comparisonTargets

List of comparisons to be performed:


name	Transaction name.
measure	Transaction’s measure property name to be computed then compared: cpuAvg cpuMax memAvg memMax batteryAvg batteryMax totalUploadedBytes totalDownloadedBytes duration speedIndex
acceptedChange	Maximum accepted change.

Examples:

{
	"filter": [ "appName", "=", "com.experitest.ExperiBank" ],
	"baseKey": "appVersion"
	"baseKeyValue": "7.0.7",
	"compareCount": 5,
	"comparisonTargets": [
		{
		"name": "EriBank Transaction Withdrawn",
		"measure": "maxMem",
		"acceptedChange": 5.0
		},
		{
		"name": "EriBank Transaction Deposit",
		"measure": "maxCpu",
		"acceptedChange": 15.0
		}
	]
}

{
	"filter": [
		[ "deviceOs", "=", "Android" ],
		"and",
		[ "appName", "=", "com.experitest.ExperiBank" ]
	],
	"baseKey": "appVersion"
	"baseKeyValue": "latest",
	"compareCount": 10,
	"comparisonTargets": [
		{
		"name": "Inquiry",
		"measure": "maxMem",
		"acceptedChange": 20.0
		}
	]
}

Response object

The response is an array of objects corresponding one element for each comparisonTarget in the request:

[
	{
		"name": <string>,
		"measure": "<string>",
		"baseValue": <number>,
		"baseCount": <integer>,
		"prevValue": <number>,
		"prevCount": <integer>,
		"prevKeyCount": <integer>,
		"acceptedChange": <number>,
		"actualChange": <number>,
		"status": <string>
		"reason": <string>,
		"link": <string>
	} ...
]

Property

Notes

name

Transaction name.

measure

Compared measure.

baseValue

The value computed for base key value ("version")

baseCount

How many non-null measure values found for the base key value.

prevValue

The value computed for previous base key values found ("previous versions").

prevCount

How many non-null measure values found for previous base key values ("previous versions").

prevKeyCount

How many distinct previous key values were found. For example, for the run, we are comparing "Inquiry" and "Deposit" transactions for the version "7.0" against previous 2 versions: "6.9" and "6.8":


Version	Transaction	Measure
7.0	Inquiry	100
6.9	Inquiry	92
6.9	Inquiry	90
6.8	Inquiry	87
7.0	Deposit	100
6.8	Deposit	95

Response contains:

{ "name: "Inquiry", … "prevKeyCount": 2, ... },

{ "name: "Deposit", … "prevKeyCount": 1, ... }, ...

acceptedChange

Same as request.

actualChange

Computed change:

((baseValue - prevValue) / prevValue) * 100

status

"ok" or "fail".

reason

Reason of "fail", can be one of the following lists:

"Change above accepted value."
"Value for [baseKey] = [base key values]] not found" (example: "Value for appVersion = [ 7.0, 6.9 ] not found".

link

Link to transaction’s page with JSON url encoded "options" containing:

Project name
Group by key name
Filter containing request filter plus previous base key values found ("previous versions")

Example ("options" not encoded for clarity):

https://experitest.com/reporter/reporter/transactions?options={"projectName":"Default","groupBy":"appVersion","filter":[["appName","=","Bank"],"and",["appVersion","in",["6.9","6.8"]],"and",["name","=","Inquiry"]]}

Worth mentioning that the generated link is aware of Reporter running behind Cloud in "single port" mode.

Java Example

Stand Alone Reporter

package com.experitests.manager.examples;

import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
import org.junit.Assert;
import org.junit.Test;

public class TransPerfExample1 {
    String baseUrl = "http://cloud/reporter/api/transactions/compare";
    String token = "xxxxx...xxxxx";

    @Test
    public void test1() throws Exception {
        String request = "{\n" +
                "\t\"filter\": [\"appName\", \"=\", \"Bank\"],\n" +
                "\t\"baseKey\": \"appVersion\",\n" +
                "\t\"baseKeyValue\": \"7.0\",\n" +
                "\t\"compareCount\": 1,\n" +
                "\t\"comparisonTargets\": [\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"cpuMax\", \"acceptedChange\": 10.0 },\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"memMax\", \"acceptedChange\": 10.0 },\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"batteryMax\", \"acceptedChange\": 10.0 },\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"totalDownloadedBytes\", \"acceptedChange\": 10.0 }\n" +
                "\t]\n" +
                "}";
        HttpResponse<String> response = Unirest.post(baseUrl + "?token=" + token)
                .header("Content-Type", "application/json")
                .body(request)
                .asString();
        int status = response.getStatus();
        Assert.assertEquals(200, status);
        if (status == 200) {
            System.out.println(response.getBody());
        }
    }
}

Cloud in single port:

(note the "/reporter" in baseUrl)

package com.experitests.manager.examples;

import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
import org.junit.Assert;
import org.junit.Test;

public class TransPerfExample2 {
    String baseUrl = "http://cloud/reporter/api/transactions/compare";
    String token = "xxxxx...xxxxx";

    @Test
    public void test1() throws Exception {
        String request = "{\n" +
                "\t\"filter\": [\"appName\", \"=\", \"Bank\"],\n" +
                "\t\"baseKey\": \"appVersion\",\n" +
                "\t\"baseKeyValue\": \"7.0\",\n" +
                "\t\"compareCount\": 1,\n" +
                "\t\"comparisonTargets\": [\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"cpuMax\", \"acceptedChange\": 10.0 },\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"memMax\", \"acceptedChange\": 10.0 },\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"batteryMax\", \"acceptedChange\": 10.0 },\n" +
                "\t\t{ \"name\": \"Inquiry\", \"measure\": \"totalDownloadedBytes\", \"acceptedChange\": 10.0 }\n" +
                "\t]\n" +
                "}";
        HttpResponse<String> response = Unirest.post(baseUrl + "?token=" + token)
                .header("Content-Type", "application/json")
                .body(request)
                .asString();
        int status = response.getStatus();
        Assert.assertEquals(200, status);
        if (status == 200) {
            System.out.println(response.getBody());
        }
    }
}

Version	Max Memory
7.0	110
7.0	105
7.0	107
7.0	99
6.9	80
6.9	79
6.9	88
6.9	95
6.9	88
6.9	120
6.8	125
6.8	110
6.7	50
6.7	65

Version	Max Memory
7.0	110
7.0	105
7.0	107
7.0	99
6.9	80
6.9	79
6.9	88
6.9	95
6.9	88
6.9	120
6.8	125
6.8	110
6.7	50
6.7	65

Introduction​

Input​

Endpoint​

Request Body Object​

Response object​

Java Example​

Stand Alone Reporter​

Cloud in single port:​