Troubleshoot Install or Upgrade on Kubernetes
Problem: The product is deployed with a wrong property (wrong version, host, etc.)
You can fix a wrong property using one of the following methods:
1. With xl kube install and generated yaml files
It is suggested first to clean existing deployments:
xl kube clean
For more information, see xl kube clean Command Reference.
Fix the wrong value using the generated yaml file (for example, Release image version from CR yaml file) and reinstall with:
xl kube install --files <reference to the generated yaml files>
For more information see xl kube install Command Reference.
2. With xl kube install and generated answer file
If the product is deployed with a wrong property because of a mistake you made while answering questions from xl kube install options reference, it might be the simplest way to change that value directly in the answer file.
It is suggested first to clean the existing deployment:
xl kube clean
For more information, see xl kube clean Command Reference.
Let's say you entered wrong Keystore Passphrase, you can update its value in answer file (the key is KeystorePassphrase).
Install the product using the updated answer file. New yaml files will be generated based on that.
xl kube install --answers <path to the generated answer files>
For more information see xl kube install Command Reference.
3. With kubectl
-
To edit Digital.ai Release's custom resource (CR), use:
kubectl edit digitalaireleases.xlr.digital.ai dai-xlr -n digitalai
Save the updated custom resource file.
Problem: Entered wrong answer during the install/upgrade with --dry-run flag active
Solutions:
If flag --dry-run is used with xl kube install/upgrade command
With --dry-run flag nothing will be deployed, but yaml files and answers will be generated and kept on the filesystem.
Change wrong values from the generated file as explained in the last section, and reinstall the product with xl kube install --files <reference to the generated yaml files>.
Another option is to run xl kube install --answers <path to the answers file> with generated and corrected answers file.
As a result of this command, new yaml files will be generated and the product will be installed based on the updated answer file.
Problem: Pod(s) are not running
Solution: Use the xl kube check. command to check everything related to the operator and to store the resources describe details, yaml and logs. Investigate the collected data or use --zip-files flag to zip all collected files and results of the check and send it to your support team.
Problem: I upgraded the product but I need to fix a few wrong values for a few properties (wrong version, image tag, and so on)
Solution: You can fix your product (Release) via one of the following two ways: use the generated answer files or use the generated yaml files.
-
Using the generated answers file:
Yaml files contain more parameters than the answers file and therefore might be difficult to edit. Answer files might be the better option if you just want to edit what you answered during the previous upgrade.
Once you edit the answers file, repeat the upgrade (
xl kube upgrade) with the updated answers file.xl kube upgrade --answers ./answers.yamlThis will generate new yaml file from the xl-op-blueprints.
-
Using the generated YAML file
If you would like to repeat upgrade with already generated yaml files, which you might have updated to fix some upgrade issues, run the
xl kube install --filescommand with a reference to the generated YAML files from the upgrade you want to repeat.xl kube install --files 20260414-150930
Digital.ai Release: PostgreSQL Pod is Not Starting
Problem
-
After installation on the cluster, it is possible to have problems with starting PostgreSQL successfully.
-
The status of the PostgreSQL pod is
CrashLoopBackOff. -
In the container log looks like the following:
postgresql 09:41:26.12 │postgresql 09:41:26.12 Welcome to the Bitnami postgresql container │postgresql 09:41:26.13 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql │postgresql 09:41:26.13 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues │postgresql 09:41:26.13 │postgresql 09:41:26.15 INFO ==> ** Starting PostgreSQL setup ** │postgresql 09:41:26.17 INFO ==> Validating settings in POSTGRESQL_* env vars.. │postgresql 09:41:26.18 INFO ==> Loading custom pre-init scripts... │postgresql 09:41:26.18 INFO ==> Initializing PostgreSQL database... │postgresql 09:41:26.21 INFO ==> pg_hba.conf file not detected. Generating it... │postgresql 09:41:26.22 INFO ==> Generating local authentication configuration
This could mean that this is a problem with volume permissions on the mounted file system for the PostgreSQL volume.
Solution
If you are on AWS EKS with EFS storage class, see the next troubleshooting section.
For other clusters try to enable the following property on the CR:
spec.postgresql.volumePermissions.enabled: true
That option will try to fix volume permissions. To apply the CR change use:
$ kubectl patch -n $NAMESPACE --type=merge digitalaireleases.xlr.digital.ai $CR_NAME -p '{"spec":{"postgresql":{"volumePermissions":{"enabled":"true"}}}}'
Replace the $NAMESPACE with the selected namespace where the Release installation will be done.
Replace the $CR_NAME with the correct Release CR name, it is possible to check for the CR name with:
$ kubectl get -n $NAMESPACE digitalaireleases.xlr.digital.ai`
NAME AGE
dai-xlr 13h
Later, delete the existing PVC and postgres pod with:
$ kubectl delete -n $NAMESPACE --wait=false pvc data-dai-xlr-postgresql-0
$ kubectl delete -n $NAMESPACE pod dai-xlr-postgresql-0
Correct names of the resources depend on the CR name.
Digital.ai Release: PostgreSQL Pod is Not Starting on AWS EKS with EFS Storage Class
Problem
-
After installation on AWS EKS cluster it is possible to have problems with starting successfully PostgreSQL.
-
The status of the pod is
CrashLoopBackOff. -
In the container log looks like the following:
postgresql 09:41:26.12 │postgresql 09:41:26.12 Welcome to the Bitnami postgresql container │postgresql 09:41:26.13 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql │postgresql 09:41:26.13 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues │postgresql 09:41:26.13 │postgresql 09:41:26.15 INFO ==> ** Starting PostgreSQL setup ** │postgresql 09:41:26.17 INFO ==> Validating settings in POSTGRESQL_* env vars.. │postgresql 09:41:26.18 INFO ==> Loading custom pre-init scripts... │postgresql 09:41:26.18 INFO ==> Initializing PostgreSQL database... │postgresql 09:41:26.21 INFO ==> pg_hba.conf file not detected. Generating it... │postgresql 09:41:26.22 INFO ==> Generating local authentication configuration
This could mean that this is a problem with volume permissions on the mounted file system for the PostgreSQL volume.
Solution
One way to solve the problem is to use with PostgreSQL other storage class, for example gp2.
If you are required to use AWS EKS based storage class, you can do the following (example is based on efs.csi.aws.com provisioner):
-
Create new AWS EKS storage class with differently setup permissions. Create following file
efs-postgresql-sc.yaml:kind: StorageClassapiVersion: storage.k8s.io/v1metadata:name: efs-postgresqlprovisioner: efs.csi.aws.comparameters:basePath: /dynamic_provisioningprovisioningMode: efs-apfileSystemId: fs-XXXXXXXXdirectoryPerms: "700"gid: "1001"uid: "1001"
Replace fs-XXXXXXXX with correct EFS file system ID.
-
Apply the defined storage class on your cluster.
kubectl apply -f efs-postgresql-sc.yaml -
Repeat installation of the Digital.ai Release with using PostgreSQL storage class name the one you just created:
efs-postgresql-sc.
OpenShift: PostgreSQL (CNPG) operator pod blocked by Security Context Constraints
Symptom: On OpenShift, the CloudNativePG PostgreSQL controller pod (cnpg-controller-manager-...) never starts and the ReplicaSet reports an error similar to:
Error creating: pods "cnpg-controller-manager-..." is forbidden: unable to validate against any
security context constraint: ... runAsUser: Invalid value: 10001: must be in the ranges:
[1000860000, 1000869999] ...
Cause: The CNPG operator pod requests a fixed runAsUser that falls outside the namespace's allowed UID range, and no Security Context Constraint (SCC) grants it.
Resolution: The installer does not grant this SCC for you, so you must grant it yourself. The anyuid SCC permits the fixed UID that the CNPG operator pod requests, but it is not bound to the operator's service account by default.
-
Identify the service account that the CNPG controller uses:
oc get deployment cnpg-controller-manager -n <namespace> \-o jsonpath='{.spec.template.spec.serviceAccountName}' -
Grant the
anyuidSCC to that service account:oc adm policy add-scc-to-user anyuid -z <cnpg-serviceaccount> -n <namespace> -
Delete the failing ReplicaSet's pods so the deployment recreates them under the new SCC:
oc rollout restart deployment/cnpg-controller-manager -n <namespace>
To avoid the failure on a new installation, grant the SCC before you run xl kube install.
xl kube clean fails with "no such file or directory"
If xl kube clean on a Bitnami-alternatives installation fails at the Deleting Operator resources step with an error similar to:
Error while getting files from .../kubernetes/template: lstat .../kubernetes/template: no such file or directory
the clean process referenced a kubernetes/template directory that was not generated. This is fixed in 26.3. The clean process now generates or gracefully skips the missing template directory and completes successfully.
xl kube clean does not remove an ingress controller installed by 25.3.x
After upgrading from 25.3.x to 26.1 or later, xl kube clean may leave the old IngressNginx Controller resources orphaned because earlier cleanup only recognized the 26.1+ naming (dai-ingress-nginx-controller) and not the 25.3.x naming (ingress-nginx-controller). This is fixed in 26.3. Cleanup now detects both naming conventions and prompts to remove the old resources.