Install or Upgrade Digital.ai Release on an Air-Gapped Kubernetes Cluster

In this article, you will learn how to install or upgrade Release on an air-gapped Kubernetes cluster, providing a step-by-step guide for those who need to install or upgrade Release on a Kubernetes cluster that is disconnected from the internet.

Prerequisites

• Set up your Kubernetes cluster
• XL CLI 22.3.5 or later
• kubectl (Version 23.1.x and above)
• Release Operator 22.3.1 or later

Install Release on an Air-gapped Cluster Environment

Here's a detailed step-by-step.

Step 1 — Download Matching Version of Blueprints

1. Run the following command:

xl kube help

It returns the following message:

Install, upgrade or clean Digital.ai Deploy or Digital.ai Release on a Kubernetes cluster using operator technology.

Installation blueprint files are used from https://dist.xebialabs.com/public/xl-op-blueprints/23.1.x/.

You need to have kubectl installed and configured for the target Kubernetes cluster.

Once you run the command, xl kube will search for blueprints from this location.

Note: 23.1.x denotes the appropriate version of xl-op-blueprints pointed by xl. You will use the specific version for your case.

2. Download all the files from that location and place them in the server directory where you will execute the xl kube command.

Tip: Download the zipped version of the xl-op-blueprints from this location

3. Unzip it to the server where you will execute the xl kube command.

Step 2 — Set up Image Registry Accessible to Kubernetes Cluster

When you run a Kubernetes cluster in an air-gapped environment, it's impossible to download images from public registries such as docker.io, gcr.io, or quay.io. Therefore, you need to set up an image registry also called as custom image registry. This should be either public (not password protected) or private (password protected), that is accessible to the Kubernetes cluster. This can be done by creating a private image repository on your cloud provider or a local image registry.

Prerequisite Images

Based on your installation plan, push the images to your image repository.

Following is the list of the images that you will need for version 23.1.x:

• docker.io/xebialabs/xl-release:23.1.x
• docker.io/xebialabs/tiny-tools:22.2.0
• docker.io/xebialabs/release-operator:23.1.x
• gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0

For nginx

• docker.io/bitnami/nginx:1.21.3-debian-10-r48
• docker.io/bitnami/nginx-ingress-controller:1.0.4-debian-10-r13

For haproxy

quay.io/jcmoraisjr/haproxy-ingress:v0.13.4

For embedded keycloak

docker.io/jboss/keycloak:17.0.1

For embedded postgresql

docker.io/bitnami/postgresql:11.13.0-debian-10-r73

For embedded rabbitmq

• docker.io/bitnami/rabbitmq:3.9.8-debian-10-r6
• docker.io/bitnami/bitnami-shell:10-debian-10-r233

Step 3 — Push Image to Internally Accessible Docker Registry

To make the required images available in an air-gapped Kubernetes cluster, you'll need to pull them first on a Bastion instance with public internet connectivity, tag them, and push them to the previously created Docker image registry that's internally accessible. Once this is done, your Kubernetes cluster can pull the required images from the Docker registry.

For example: In the Docker registry (myrepo_host), run the following command to push the Docker image (xl-release) to the repository (myrepo):

docker pull docker.io/xebialabs/xl-release:22.3.1
docker tag docker.io/xebialabs/xl-release:22.3.1 myrepo_host/myrepo/xl-release:22.3.1
docker push myrepo_host/myrepo/xl-release:22.3.1

Note: Before pushing a Docker image to a registry, it's important to ensure that you're logged in to the Docker registry.

Step 4 — Create a Secret for Private Image Registry with Password Protection

Note: It's required only when you use a private image registry.

Run the following command:

kubectl create secret docker-registry regcred \
  --docker-server=myrepo_host \
  --docker-username=<reg-user> \
  --docker-password=<reg-password> \
  -n digitalai

In this example, a secret called regcred is created, which can be used to pull images from a custom private image registry.

Step 5 — Use xl kube install to Install Using Custom Docker Image Registry Option

When using custom Docker registry, the operator image will be in this format: myrepo_host/myrepo/release-operator:image_tag

The following is an example of the installation on Minikube with a local Docker registry running at localhost:5000.

• Registry name is localhost:5000
• Repository name is myrepo
• Operator image is localhost:5000/myrepo/release-operator:23.1.x

Note: Ensure to override the default answers and specify in this format. Additionally, you will use the actual image tag version that you want to use in place of 23.1.x.

❯ xl kube install -l ./xl-op-blueprints
? Following kubectl context will be used during execution: `minikube`? Yes
? Select the Kubernetes setup where the Digital.ai Devops Platform will be installed, updated or cleaned: PlainK8s [Plain multi-node K8s cluster]
? Do you want to use an custom Kubernetes namespace (current default is 'digitalai'): No
? Do you want to create custom Kubernetes namespace digitalai, it does not exist: Yes
? Product server you want to perform install for: dai-release [Digital.ai Release]
? Select type of image registry: public [Custom Public Registry (Uses a specific custom registry)]
? Enter the custom docker image registry name: localhost:5000
? Enter the repository name (eg: <repositoryName> from <repositoryName>/<imageName>:<tagName>): myrepo
? Enter the image name (eg: <imageName> from <repositoryName>/<imageName>:<tagName>): xl-release
? Enter the image tag (eg: <tagName> from <repositoryName>/<imageName>:<tagName>): 23.1.x
? Enter PVC size for Release (Gi): 1
? Select between supported Access Modes: ReadWriteMany [ReadWriteMany]
? Select between supported ingress types: nginx [NGINX]
? Do you want to enable an TLS/SSL configuration (if yes, requires existing TLS secret in the namespace): No
? Provide DNS name for accessing UI of the server: test.com
? Provide administrator password: OJnEi1BMBRuDm3ny
? Type of the OIDC configuration: no-oidc [No OIDC Configuration]
? Enter the operator image to use (eg: <imageRegistryName>/<repositoryName>/<imageName>:<tagName>): localhost:5000/myrepo/release-operator:23.1.x
? Select source of the license: file [Path to the license file (the file can be in clean text or base64 encoded)]
? Provide license file for the server: ./xl-release-license.lic
? Select source of the repository keystore: generate [Generate the repository keystore during installation (you need to have keytool utility installed in your path)]
? Provide keystore passphrase: ANk9nAJnmQjLmFPy
? Provide storage class for the server: standard
? Do you want to install a new PostgreSQL on the cluster: Yes
? Provide Storage Class to be defined for PostgreSQL: standard
? Provide PVC size for PostgreSQL (Gi): 1
? Do you want to install a new RabbitMQ on the cluster: Yes
? Replica count to be defined for RabbitMQ: 1
? Storage Class to be defined for RabbitMQ: standard
? Provide PVC size for RabbitMQ (Gi): 1

...

? Do you want to proceed to the deployment with these values? Yes
For current process files will be generated in the: digitalai/dai-release/digitalai/20221019-234412/kubernetes
Generated answers file successfully: digitalai/generated_answers_dai-release_digitalai_install-20221019-234412.yaml
Starting install processing.
...

Once the installation is completed successfully, you will see the operator and other resources pods up and running.

Upgrade Release on an Air-gapped Cluster Environment

Note: The steps to upgrade Release is similar to the installation steps. You must ensure that the already installed cluster resources are overwritten or upgraded with the newly supplied values.

Run the following command to upgrade Release:

xl kube upgrade

Use xl kube upgrade to Upgrade Using Custom Docker Image Registry Option

The following is an example of running xl kube upgrade command to upgrade Release.

❯ xl kube upgrade -l ./xl-op-blueprints
...
? Select type of image registry: public [Custom Public Registry (Uses a specific custom registry)]
? Enter the custom docker image registry name: localhost:5000
? Enter the repository name (eg: <repositoryName> from <repositoryName>/<imageName>:<tagName>): myrepo
...
? Enter the operator image to use (eg: <imageRegistryName>/<repositoryName>/<imageName>:<tagName>): localhost:5000/myrepo/release-operator:22.3.1
...
? Edit list of custom resource keys that will migrate to the new Release CR: 
...
? Do you want to proceed to the deployment with these values? Yes
For current process files will be generated in the: digitalai/dai-release/digitalai/20221020-004412/kubernetes
Generated answers file successfully: digitalai/generated_answers_dai-release_digitalai_upgrade-20221020-004412.yaml
Starting upgrade processing.
...

Important: During the upgrade, for the prompt Edit list of custom resource keys that will migrate to the new Release CR:, you must add the following keys:

For Default Image Registry

.spec.TinyToolsImageRepository
.spec.nginx-ingress-controller.defaultBackend.image.registry
.spec.nginx-ingress-controller.defaultBackend.image.repository
.spec.nginx-ingress-controller.image.registry
.spec.nginx-ingress-controller.image.repository
.spec.haproxy-ingress.controller.image.repository
.spec.keycloak.image.repository
.spec.keycloak.postgresql.image.registry
.spec.keycloak.postgresql.image.repository
.spec.postgresql.image.registry
.spec.postgresql.image.repository
.spec.rabbitmq.image.registry
.spec.rabbitmq.image.repository
.spec.rabbitmq.volumePermissions.image.registry
.spec.rabbitmq.volumePermissions.image.repository

For Public Custom Docker Registry

.spec.TinyToolsImageRepository
.spec.nginx-ingress-controller.defaultBackend.image.repository
.spec.nginx-ingress-controller.image.repository
.spec.nginx-ingress-controller.global.imageRegistry
.spec.haproxy-ingress.controller.image.repository
.spec.keycloak.image.repository
.spec.keycloak.postgresql.image.registry
.spec.keycloak.postgresql.image.repository
.spec.postgresql.image.repository
.spec.postgresql.global.imageRegistry
.spec.rabbitmq.image.repository
.spec.rabbitmq.global.imageRegistry
.spec.rabbitmq.volumePermissions.image.repository

For Private Custom Docker Registry

.spec.TinyToolsImageRepository
.spec.nginx-ingress-controller.defaultBackend.image.repository
.spec.nginx-ingress-controller.image.repository
.spec.nginx-ingress-controller.global.imageRegistry
.spec.haproxy-ingress.controller.image.repository
.spec.keycloak.image.repository
.spec.keycloak.postgresql.image.registry
.spec.keycloak.postgresql.image.repository
.spec.postgresql.image.repository
.spec.postgresql.global.imageRegistry
.spec.rabbitmq.image.repository
.spec.rabbitmq.global.imageRegistry
.spec.rabbitmq.volumePermissions.image.repository
.spec.nginx-ingress-controller.global.imagePullSecrets
.spec.keycloak.imagePullSecrets
.spec.keycloak.postgresql.imagePullSecrets
.spec.postgresql.global.imagePullSecrets
.spec.rabbitmq.global.imagePullSecrets