Troubleshooting Release Remote Runner
The logging feature in Release Remote Runner is the best solution to troubleshoot and check for the state of runner lifecycle or the state of execution.
Log Levels and Types
Log level and log type can be changed using configuration or environment variables.
To change the log level, use the RELEASE_RUNNER_LOG_LEVEL
environment variable and set any of the following values:
0
: DEBUG level1
: INFO level2
: WARNING level3
: ERROR level4
: FATAL level5
: PANIC level6
: No logging7
: Disable logging completely
Logs can be in JSON or standard console log format.
To switch log type, use the RELEASE_RUNNER_JSON_LOGGING
environment variable and set any of the following values:
- set to
true
to use JSON logging - set to
false
for console logs format
You can switch on JSON logging also using profile json-logging
.
Execution logs are passed on to Release, and they are available in the Log console on task view for Task executions. By default, script executions logs are visible in the Runner logs as JSON entries for the execution.
To change this behavior, use the RELEASE_RUNNER_LOGGING_SCRIPT_LOG
environment variable and set any of the following values:
- set to
SILENT
if you don't want to log the script execution logs in Runner logs - set to
LOG
if you want to keep the default behavior and log the script execution logs in Runner logs
Troubleshooting
The following are some of the troubleshooting scenarios:
Runner container is alive but not in ready state
Wrong Release URL Setting
Behaviour: Runner container is alive but doesn't enter ready state. Logs are showing messages similar to:
2023-09-22T11:03:51+02:00 ERR service/release/runner-registration.go:63 > Registration failed, got error from server: 'Post "http://uknown.host.acme/api/v1/runners-registration/register": dial tcp 127.0.0.1:3516: connect: connection refused'
Solution: Check if the set Release URL is correct and reachable from the container. Fix the URL string or re-configure your network and set correct URL in RELEASE_RUNNER_RELEASE_URL
environment variable.
Wrong Release PAT Setting
Behaviour: Runner container is alive but doesn't enter ready state. Logs are showing messages like:
2023-09-22T10:56:59+02:00 INF service/runner/runner-registration.go:27 > Registered existing runner with runner ID: [186d2384-641c-4021-8d54-f7ec7c9086e3].
2023-09-22T10:56:59+02:00 INF service/retry.go:59 > Retrying after error: HTTP status was [401], response body was {
"message":"Unauthorized",
"url":"/api/v1/runners-registration/register",
"status":"401"
}
Solution: Check if the set Release PAT is correct, expired, or missing. Fix the PAT string or generate a new one and set it in the RELEASE_RUNNER_TOKEN
environment variable.
Runner is Not Processing New Jobs
Behaviour: Logs show messages of 503 service unavailable and log messages that health is failing:
2023-09-22T11:12:33+02:00 WRN service/release/health.go:54 > Error received on health check: Get "http://localhost:5516/api/v1/runners/Configuration/Custom/Configuration186d2384641c40218d54f7ec7c9086e3/health?requestId=7e633cbf-451d-44fb-94aa-636a4439bed1": dial tcp 127.0.0.1:5516: connect: connection refused
Solution: You lost connection to Release because the Release service shutdown. Check the Release service and wait for Runner to re-establish the connection. Once Release starts, the logs will display messages as follows:
2023-09-22T11:13:24+02:00 WRN service/release/client-release.go:136 > Server health check returned response [STARTING] for requestID [d8c60154-e753-46fd-9eb8-9a29e07e453b]
2023-09-22T11:13:24+02:00 WRN service/release/health.go:54 > Error received on health check: health response state is [STARTING]
2023-09-22T11:13:27+02:00 INF service/release/client-release.go:168 > Trying long poll Job Reserve, with requestId [6e4c6425-0fa4-4b8e-8d87-223ca7396c5e].
2023-09-22T11:13:27+02:00 INF service/release/sse-client.go:17 > Registering SSE connector.
2023-09-22T11:13:28+02:00 INF service/release/sse-commands.go:51 > Processing SSE command: EnableRunner.
2023-09-22T11:13:28+02:00 INF service/release/router.go:78 > Runner enabled triggering reserve.
2023-09-22T11:13:28+02:00 INF service/release/sse-commands.go:93 > Confirming EnableCommand