Skip to main content

The reporter can send test data to an ElasticSearch instance for further analysis and reporting.

Requirements

Integration between Reporter and ElasticSearch has these requirements:

  • Elasticsearch supported version: 7.5 through 7.117 and 8.0 through 8.6
  • Either user/password or API key:
    • user/password: Must be associated with the role with the following privileges:
      • index name "*", privileges: "view_index_metadata"
      • index name "reporter_tests_*", privileges: "create_index", "read", "write"
    • API key: Must be created with the following privileges:
      • index name "*", privileges: "view_index_metadata"
      • index name "reporter_tests_*", privileges: "create_index", "read", "write"

Configuration

Elasticsearch integration is configured with the following application.properties entries:

KeyDescription
elastic.urlURL of ElasticSearch instance required.
elastic.usernameElasticsearch's user name, required unless an API key is used.
elastic.passwordElasticsearch's password, required unless an API key is used.
elastic.apikeyAPI Key.
elastic.tests-sender.cronSee "CRON expression" section in Cron.
elastic.tests-sender.async-countSize of dedicated asynchronous executor's thread pool called elasticTestsAsyncExecutor. The default is 2.
elastic.tests-sender.bulk-sizeHow many tests' operations to send per bulk.
elastic.tests-sender.bulk-countHow many bulks to send per run.
elastic.tests-sender.enable-initialInitial enable state of sender. The default is false. Note: After configuring ElasticSearch in reporter, set this to true.
cust.idCustomer ID. This is needed to distinguish customers' indexes when they share an Elasticsearch instance.
elastic.index_nameIf present Reporter will send tests in the mode index per environmen" (see below).
elastic.envEnvironment’s name, used for "index per environment" to feed test’s document "env" property. Must be present if elastic_index is configured.

Control and Monitoring Rest Interfaces

EndpointMethodPurpose
/api/elastic/tests/stateGETShows the current state of Elasticsearch tests' sender



* Enabled: y/n

* Bulk size.

* Bulk count.

* The last time was enabled or disabled.

* Active (running) senders.

* Run count since last enable.

* The operation sent since the last enable.

* Failed operations since last enable.

* Total bulks sent.

* Total time spent sending bulks.

* Average time per bulk.
/api/elastic/tests/disableGETDisable the sender.



If is disabled scheduled task does nothing, just emits a log warning Senders not enabled, nothing to do.
/api/elastic/tests/enableGETEnable the sender.
/api/elastic/tests/clearGETClear counters.

Set the Project sendTestsToElastic Property

The reporter sends the project's tests when sendTestsToElastic is set to true. To change this, issue a PUT to {reporterUrl}/api/projects with the following JSON payload:

{ "id": <project's ID>, "sendTestsToElastic":  <true | false> }

This requires credentials via HTTP BASIC AUTHENTICATION or using token (bearer) authentication, and the user must be a Cloud Administrator.

Indexes 

Index Per Project or Per Environment​

The reporter can create an index per Reporter’s project or one index for the environment. In this case all the project’s tests are sent to the same index. Reporter determines which index is used based on the configuration key elastic.index_name. If it is present, only one index is used for the Reporter’s instance. If you use an index per environment, ensure that the user can create indexes. Consult the ElasticSearch documentation.

ElasticsSearch Index Creation​

Index Per Project ​

The reporter creates one index for each project whose sendTestsToElastic property is set to true.

Big customers may have their own private Elasticsearch instance but small customers may not. They may be sharing an ElasticSearch instance so this is the index name structure used is:

reporter_tests_${<cust.id>}_${<project.id>}

  • cust.id: Customer ID assigned by Continuous Testing Cloud. This is always an alphanumeric lowercase string (no special chars). This must be set in the application.properties.
  • project.id: ID of the project the tests belong to.

Storage settings such as shards and replicas are the responsibility of the ElasticSearch administrator.

Elasticsearch Index Template​

Before Reporter can create and send the test to index, ElasticSearch's admin must create the template named reporter_tests_template.

Search Template

PUT _index_template/reporter_tests_template
{
   "index_patterns":[
      "reporter_tests_*"
   ],
   "template": {
     "settings": {
        "number_of_shards": 1
      },
     "mappings":{
        "_source": {
          "enabled": true
        },
        "properties":{
           "id":{
              "type":"long"
           },
           "name":{
              "type":"text"
           },
           "startTime":{
              "type":"date"
           },
           "duration":{
              "type":"long"
           },
           "status":{
              "type":"keyword"
           },
           "success":{
              "type":"boolean"
           },
           "date":{
              "type":"date"
           },
           "hasReport":{
              "type":"boolean"
           }
        }
     }
   }
}
Last updated on