Skip to main content
Version: Release 24.3

Troubleshooting Runners

The logging feature in Release Runner is the best solution to troubleshoot and check for the state of runner lifecycle or the state of execution.

Log Levels and Types​

Log level and log type can be changed using configuration or environment variables.

To change the log level, use the RELEASE_RUNNER_LOG_LEVEL environment variable and set any of the following values:

  • 0: DEBUG level
  • 1: INFO level
  • 2: WARNING level
  • 3: ERROR level
  • 4: FATAL level
  • 5: PANIC level
  • 6: No logging
  • 7: Disable logging completely

Logs can be in JSON or standard console log format.

To switch log type, use the RELEASE_RUNNER_JSON_LOGGING environment variable and set any of the following values:

  • set to true to use JSON logging
  • set to false for console logs format
note

You can switch on JSON logging also using profile json-logging.

Execution logs are passed on to Release, and they are available in the Log console on task view for Task executions. By default, script executions logs are visible in the Runner logs as JSON entries for the execution.

To change this behavior, use the RELEASE_RUNNER_LOGGING_SCRIPT_LOG environment variable and set any of the following values:

  • set to SILENT if you don't want to log the script execution logs in Runner logs
  • set to LOG if you want to keep the default behavior and log the script execution logs in Runner logs

Troubleshooting​

The following are some of the troubleshooting scenarios:

Runner container is alive but not in ready state​

Wrong Release URL Setting​

Behaviour: Runner container is alive but doesn't enter ready state. Logs are showing messages similar to:

2023-09-22T11:03:51+02:00 ERR service/release/runner-registration.go:63 > Registration failed, got error from server: 'Post "http://uknown.host.acme/api/v1/runners-registration/register": dial tcp 127.0.0.1:3516: connect: connection refused'

Solution: Check if the set Release URL is correct and reachable from the container. Fix the URL string or re-configure your network and set correct URL in RELEASE_RUNNER_RELEASE_URL environment variable.

Wrong Release PAT Setting​

Behaviour: Runner container is alive but doesn't enter ready state. Logs are showing messages like:

2023-09-22T10:56:59+02:00 INF service/runner/runner-registration.go:27 > Registered existing runner with runner ID: [186d2384-641c-4021-8d54-f7ec7c9086e3].
2023-09-22T10:56:59+02:00 INF service/retry.go:59 > Retrying after error: HTTP status was [401], response body was {
"message":"Unauthorized",
"url":"/api/v1/runners-registration/register",
"status":"401"
}

Solution: Check if the set Release PAT is correct, expired, or missing. Fix the PAT string or generate a new one and set it in the RELEASE_RUNNER_TOKEN environment variable.

Runner is Not Processing New Jobs​

Behaviour: Logs show messages of 503 service unavailable and log messages that health is failing:

2023-09-22T11:12:33+02:00 WRN service/release/health.go:54 > Error received on health check: Get "http://localhost:5516/api/v1/runners/Configuration/Custom/Configuration186d2384641c40218d54f7ec7c9086e3/health?requestId=7e633cbf-451d-44fb-94aa-636a4439bed1": dial tcp 127.0.0.1:5516: connect: connection refused

Solution: You lost connection to Release because the Release service shutdown. Check the Release service and wait for Runner to re-establish the connection. Once Release starts, the logs will display messages as follows:

2023-09-22T11:13:24+02:00 WRN service/release/client-release.go:136 > Server health check returned response [STARTING] for requestID [d8c60154-e753-46fd-9eb8-9a29e07e453b]
2023-09-22T11:13:24+02:00 WRN service/release/health.go:54 > Error received on health check: health response state is [STARTING]
2023-09-22T11:13:27+02:00 INF service/release/client-release.go:168 > Trying long poll Job Reserve, with requestId [6e4c6425-0fa4-4b8e-8d87-223ca7396c5e].
2023-09-22T11:13:27+02:00 INF service/release/sse-client.go:17 > Registering SSE connector.
2023-09-22T11:13:28+02:00 INF service/release/sse-commands.go:51 > Processing SSE command: EnableRunner.
2023-09-22T11:13:28+02:00 INF service/release/router.go:78 > Runner enabled triggering reserve.
2023-09-22T11:13:28+02:00 INF service/release/sse-commands.go:93 > Confirming EnableCommand