Version: Release 24.3

Manage Plugins Offline in Kubernetes Environment

This guide explains how to manage Digital.ai Release plugins in offline mode when working with a Release cluster created using the Operator-based installer. You'll learn how to manage plugins when the Release UI is unavailable or not functioning properly.

This process involves:

Creating a temporary pod named dai-xlr-plugin-management
Stopping all other Release pods while keeping the temporary pod running
Accessing the temporary pod to manage plugins using the Plugin Manager CLI
Restarting the Release pods
Deleting the temporary pod after completion

This approach is also a way to manage plugins if GUI or xl plugin is not working, for example to delete the installed plugins.

note

This guide uses the default namespace digitalai for examples. If you installed Release in a custom namespace, use your namespace instead.

Verify the PVC name on your current namespace (it depends on the CR name):

kubectl get pvc -n digitalai

Example Output
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                                   AGE
dai-xlr-digitalai-release   Bound    pvc-1ace4aaf-d5e2-4ad6-a913-a39924310ea6   1Gi        RWO            vp-azure-aks-test-cluster-file-storage-class   150m
data-dai-xlr-postgresql-0   Bound    pvc-d761fcee-787c-4393-b346-6c3e79e06112   1Gi        RWO            vp-azure-aks-test-cluster-disk-storage-class   150m
data-dai-xlr-rabbitmq-0     Bound    pvc-dc39f7dd-d05a-4f87-acaf-89473a791765   1Gi        RWO            vp-azure-aks-test-cluster-file-storage-class   150m

Suppose the release PVC name is dai-xlr-digitalai-release.

Create a pod-dai-xlr-plugin-management.yaml file and add the following code to the file:

cat << EOF >> pod-dai-xlr-plugin-management.yaml
apiVersion: v1
kind: Pod
metadata:
  name: dai-xlr-plugin-management
spec:
  restartPolicy: Never
  securityContext:
    runAsUser: 10001
    runAsGroup: 0
    fsGroup: 10001
  containers:
    - name: sleeper
      command: [ "/bin/sh" ]
      # don't run Release, but prepare container conf and start sleep
      args: [ "-c", "grep -vwE exec.*XLReleaseBootstrapper bin/run.sh > bin/run.sh; ./bin/run-in-container.sh; sleep 1d;" ]
      image: xebialabs/xl-release:24.3.0
      imagePullPolicy: Always
      resources:
        requests:
          memory: "1Gi"
        limits:
          memory: "1Gi"
EOF

note

Replace the image version (24.3.0) with the correct version of the product you are using.

Extract the current Release StatefulSet environment configuration:

kubectl -n $NAMESPACE get sts $STS_NAME -o yaml \
  | yq '.spec.template.spec.containers[0].env' > sts-env.yaml

Merge the Release StatefulSet environment configuration with the base pod file:

yq '.spec.containers[0].env = load("sts-env.yaml")' \
pod-dai-xlr-plugin-management.yaml > pod-dai-xlr-plugin-management-patched.yaml

Apply the patched pod file to create the temporary pod:

kubectl apply -n $NAMESPACE -f pod-dai-xlr-plugin-management-patched.yaml

Stop the Release pods.
- The Release pods must be stopped before running the plugin manager script to ensure safe and consistent plugin management. To do this, set the number of replicas 0.
```
kubectl get digitalaireleases.xlr.digital.ai -n digitalai
NAME      AGE
dai-xlr   179m

kubectl patch digitalaireleases.xlr.digital.ai dai-xlr -n digitalai \
--type=merge \
--patch '{"spec":{"replicaCount":0}}'
```
- Restart the Release stateful set, the name depends on the CR name (or you can wait few seconds, the update will be automatic after earlier change):
```
kubectl rollout restart sts dai-xlr-digitalai-release -n digitalai
```
Wait until all the Release pods terminate.

Log on (SSH) to the newly created pod.

kubectl exec -it dai-xlr-plugin-management -n digitalai -- bash

Add or remove the plugins using the Plugin Manager CLI. Go to bin directory after SSH login:
```
cd /opt/xebialabs/xl-release-server/bin
```
See Plugin Manager CLI to know more about how to add or remove plugins.

For example, the following commands are to delete the xlr-svn-plugin plugin.
```
bash-4.2$ ./plugin-manager-cli.sh -list
...

bash-4.2$ ./plugin-manager-cli.sh -delete xlr-svn-plugin

xlr-svn-plugin deleted from database
Please verify and delete plugin file in other cluster members' plugins directory if needed
```
Exit the SSH shell using the exit command.

Restart the Release pods.

Set the number of replicas back to the required number (2 replicas in this example)

kubectl get digitalaireleases.xlr.digital.ai -n digitalai

Output
NAME      AGE
dai-xlr   179m

kubectl patch digitalaireleases.xlr.digital.ai dai-xlr -n digitalai \
--type=merge \
--patch '{"spec":{"replicaCount":2}}'

Restart the Release stateful set, the name depends on the CR name (or you can wait few seconds, the update will be automatic after previous change):

kubectl rollout restart sts dai-xlr-digitalai-release -n digitalai

Delete the temporary pod—dai-xlr-plugin-management.

kubectl delete pod dai-xlr-plugin-management -n digitalai --force=true

A Script to Automate All the Above Steps

Replace the environment variables with relevant values of your Kubernetes environment.

Here is an example that shows how this could be done with kubectl as the bash script.

note

Stop all the pods before you run the script.

SOURCE_PLUGIN_DIR=/tmp
PLUGIN_NAME=xlr-hashicorp-vault-plugin
PLUGIN_VERSION=22.3.0-704.113
SOURCE_PLUGIN_FILE=$PLUGIN_NAME-$PLUGIN_VERSION.jar
RELEASE_STS=dai-xlr-digitalai-release
RELEASE_CR=dai-xlr
NAMESPACE=digitalai
REPLICA_COUNT=$(kubectl get digitalaireleases.xlr.digital.ai dai-xlr -n digitalai -o 'jsonpath={.spec.replicaCount}')

kubectl apply -f pod-dai-xlr-plugin-management.yaml -n digitalai
kubectl patch digitalaireleases.xlr.digital.ai $RELEASE_CR -n $NAMESPACE \
      --type=merge \
      --patch '{"spec":{"replicaCount":0}}'
sleep 30; kubectl rollout status sts $RELEASE_STS -n $NAMESPACE --timeout=300s
kubectl wait --for condition=Ready --timeout=60s pod dai-xlr-plugin-management -n $NAMESPACE

kubectl cp $SOURCE_PLUGIN_DIR/$SOURCE_PLUGIN_FILE $NAMESPACE/dai-xlr-plugin-management:$SOURCE_PLUGIN_DIR/  
kubectl exec dai-xlr-plugin-management -n $NAMESPACE -- /opt/xebialabs/xl-release-server/bin/plugin-manager-cli.sh -add $SOURCE_PLUGIN_DIR/$SOURCE_PLUGIN_FILE

kubectl patch digitalaireleases.xlr.digital.ai $RELEASE_CR -n $NAMESPACE \
      --type=merge \
      --patch "{\"spec\":{\"replicaCount\":$REPLICA_COUNT }}"
kubectl delete -f pod-dai-xlr-plugin-management.yaml -n $NAMESPACE
sleep 30; kubectl rollout status sts $RELEASE_STS -n $NAMESPACE --timeout=300s

A Script to Automate All the Above Steps​

A Script to Automate All the Above Steps