EKS Anywhere, Part-1 Dell EMC Unity-XT CSI 2.2.0

This article is part of the EKS Anywhere series EKS Anywhere., extending the Hybrid cloud momentum

In this article, we are going to explore the implementation, integration and testing of Dell Unity-XT CSI with EKS Anywhere.

Why., motivation for this article: Firstly…most of the public docs for EKS Anywhere leverages examples around the default VMware’s CNS (Cloud Native Storage) CSI. Second, the CSI documentation often refers to descriptive technicalities with lesser focus on an end-to-end validated scenario.

In this article, we will observe the simplicity of Dell EMC CSI installation with EKS Anywhere that can further enhance the performance and other related attributes of persistent workloads.

As a context, one can observe DellEMC’s CSI coverage CSI Drivers | Dell Technologies. In addition to the CSI implementations, DellEMC has also innovated and instrumented CSM (Container Storage Modules), which further heighten the range of experiences by adding authentication, authorization, observability, replication, etc.

As the context for CSI is now set, let’s begin our implementation use-case

Goals:

• Implement CSI drivers for Unity-XT on EKS Anywhere cluster
• Implement snapshotting capabilities via external-snapshotter
• Deploy a persistent MySQL workload with a web-frontend
• Test various use-cases around persistence, snapshotting (backup & restore)

As one can see from the below kubectl output for my EKS Anywhere workload cluster, AWS EKS Anywhere ships with the standard storage class which is mapped to VMware’s CNS CSI. This default CSI is also covered in the other article

 

 

kubectl get storageclassNAME                 PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGEstandard (default)   csi.vsphere.vmware.com   Delete          Immediate           false                  7d4h

 

 

In the use-case example for this article, we will use Dell EMC Unity XT as the target NFS platform for our MySQL persistent workload in EKS Anywhere workload cluster. The persistent volume claims will leverage the CSI drivers of Dell EMC CSI version 2.2.0 to create respective volumes

Pre-requisites:

1. A working EKS Anywhere cluster either standalone or with a dedicated management cluster. It does not matter which model you use to deploy the cluster., at the end of the day we need a working EKS Anywhere cluster. Also, you can name the EKS Anywhere cluster as testunitycsicluster01 just to be consistent with the below implementation example
2. All file and template references used in this article indicate the paths setup in relation to the same git cloned repository that was setup during the EKS Anywhere administrative machine installation
3. At-least one static IP address from the same range of EKS Anywhere cluster network. This will be used for the adminer web application that is exposed via the MetalLB based load-balanced service
4. A Unity XT array configured with storage pool, nas server, that will be used for persistent storage. Also note the Unique ArrayId, IP address, username, and password that will be used to integrate the CSI drivers.

Note: As per the cluster creation process you should have ear-marked a set of static IP addresses (for control plane and for load balancer services). We will need those static IP addresses in this configuration. I am using a single static IP 172.24.165.26 to 172.24.165.26 (yours could/will be different) to expose the adminer web application via MetalLB load-balancer

Prepare the EKS Anywhere cluster environment for Unity XT CSI 2.2.0 installation

Logon the EKS Administrative machine and follow the below steps

 

 

Target your EKS Anywhere workload cluster from the EKS Anywhere Administrative machine (This example assumes the cluster name is testunitycsicluster01)export CLUSTER_NAME=testunitycsicluster01cd $HOMEexport KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfigCreate an implementation of Kubernetes External Snapshotter, which is now official GAcd $HOMEgit clone https://github.com/kubernetes-csi/external-snapshotter/cd ./external-snapshottergit checkout release-5.0Now, we can deploy the external-snapshotter objectskubectl kustomize client/config/crd | kubectl create -f -customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io createdcustomresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io createdcustomresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io createdkubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -serviceaccount/snapshot-controller createdrole.rbac.authorization.k8s.io/snapshot-controller-leaderelection createdclusterrole.rbac.authorization.k8s.io/snapshot-controller-runner createdrolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection createdclusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role createddeployment.apps/snapshot-controller createdAt this stage, we can verify if the external snapshotter is and up and runningkubectl get pods -n kube-system -l app=snapshot-controllerNAME                                   READY   STATUS    RESTARTS   AGEsnapshot-controller-75fd799dc8-7hq5f   1/1     Running   0          7m18ssnapshot-controller-75fd799dc8-x9kkj   1/1     Running   0          7m18sCreate a new namespace for our Unity CSIkubectl create namespace unityNext we will download the required files for the Dell EMC Unity XT CSI implementationcd $HOMEgit clone -b v2.2.0 https://github.com/dell/csi-unity.gitNavigate to the installer directorycd $HOME/csi-unity/dell-csi-helm-installerCreate a file named emptysecret.yaml with the below content in this directoryapiVersion: v1kind: Secretmetadata:name: unity-certs-0namespace: unitytype: Opaquedata:cert-0: ""Next issue the commandkubectl create -f emptysecret.yamlsecret/unity-certs-0 createdCreate another file called secret.yaml in this directory and paste the below replacing your Unity XT arrayId, username, password and endpoint IP address.storageArrayList:- arrayId: "aaaaaaaaaaaaa"username: "bbbbbbbbbbbb"password: "cccccccccccc"endpoint: "https://172.24.165.5/"skipCertificateValidation: trueisDefault: trueNext create a secret from the above create filekubectl create secret generic unity-creds -n unity --from-file=config=secret.yamlsecret/unity-creds createdWe will now copy the sample values.yaml into our CSI installer directorycd $HOME/csi-unity/dell-csi-helm-installercp $HOME/csi-unity/helm/csi-unity/values.yaml myvalues.yamlNext we will modify the myvalues.yaml to insert some unique values so that the volumes can be rendered with relevant references of cluster name, etc., such that the volumes can be easily identifiedcd $HOME/csi-unity/dell-csi-helm-installersed -i "s/csivol/$CLUSTER_NAME-vol/g" myvalues.yamlsed -i "s/csi-snap/$CLUSTER_NAME-snap/g" myvalues.yamlThe next steps will ensure that the kubernetes versions are correctly referenced in the Helm chartCopy the value of Kubernetes version installed on the EKS Anywhere cluster nodes. As you can see that in my case the version (highlighted below) is v1.21.9-eks-c9274eakubectl get nodesNAME                                          STATUS   ROLES                  AGE    VERSIONtestunitycsicluster01-2fp59                   Ready    control-plane,master   114m   v1.21.9-eks-c9274eatestunitycsicluster01-7r2cg                   Ready    control-plane,master   112m   v1.21.9-eks-c9274eatestunitycsicluster01-md-0-59f9568584-pklvw   Ready112m v1.21.9-eks-c9274ea testunitycsicluster01-md-0-59f9568584-sfhf9 Ready112m v1.21.9-eks-c9274ea Edit the Helm chart for the Unity XT CSI installation cd $HOME/csi-unity/helm/csi-unity ls -al total 24 drwxrwxr-x 3 ubuntu ubuntu 4096 May 19 12:59 . drwxrwxr-x 3 ubuntu ubuntu 4096 May 19 12:59 .. -rw-rw-r-- 1 ubuntu ubuntu 619 May 19 12:59 Chart.yaml drwxrwxr-x 2 ubuntu ubuntu 4096 May 19 12:59 templates -rw-rw-r-- 1 ubuntu ubuntu 6912 May 19 12:59 values.yaml As you can see there is a file named Chart.yaml. Edit this file in your favorite editor and changed the highlighted kubeVersion with the kubernetes version noted for your EKS Anywhere cluster ORIGINAL FILE CONTENT name: csi-unity version: 2.2.0 appVersion: 2.2.0 kubeVersion: ">= 1.21.0 < 1.24.0" # If you are using a complex K8s version like "v1.21.3-mirantis-1", use this kubeVersion check instead # WARNING: this version of the check will allow the use of alpha and beta versions, which is NOT SUPPORTED # kubeVersion: ">= 1.21.0-0 < 1.24.0-0" description: | Unity CSI (Container Storage Interface) driver Kubernetes integration. This chart includes everything required to provision via CSI as well as a Unity StorageClass. keywords: - csi - storage sources: - https://github.com/dell/csi-unity maintainers: - name: DellEMC CHANGED FILE CONTENT AFTER EDITING AND SAVING THE ABOVE FILE name: csi-unity version: 2.2.0 appVersion: 2.2.0 kubeVersion: "v1.21.9-eks-c9274ea" # If you are using a complex K8s version like "v1.21.3-mirantis-1", use this kubeVersion check instead # WARNING: this version of the check will allow the use of alpha and beta versions, which is NOT SUPPORTED # kubeVersion: ">= 1.21.0-0 < 1.24.0-0" description: | Unity CSI (Container Storage Interface) driver Kubernetes integration. This chart includes everything required to provision via CSI as well as a Unity StorageClass. keywords: - csi - storage sources: - https://github.com/dell/csi-unity maintainers: - name: DellEMC

 

 

Step-3 Installing Unity XT CSI 2.2.0

Before starting the installation, here is a quick snapshot of how my Unity-XT hosts dashboard looks like. As you can see there is no host that is present for our EKS Anywhere cluster testunitycsicluster01. What we can expect is that once the CSI is installed on the EKS Anywhere cluster, the worker nodes will automatically register with the target Unity-XT system

At this stage, all our pre-requisites are met and if no errors/mistakes were made in the above., then we can issue the below command to install the CSI drivers.

Note: The installation script would take a couple of minutes to finish everything that’s under the hood., so be patient

 

 

cd $HOME/csi-unity/dell-csi-helm-installer./csi-install.sh --namespace unity --values ./myvalues.yaml --skip-verify

 

 

Output log: The below log indicates that our installation went through smoothly (after being patient for 3–5 minutes)

 

 

./csi-install.sh --namespace unity --values ./myvalues.yaml --skip-verifyWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.WARNING: version difference between client (1.24) and server (1.21) exceeds the supported minor version skew of +/-1WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.WARNING: version difference between client (1.24) and server (1.21) exceeds the supported minor version skew of +/-1------------------------------------------------------> Installing CSI Driver: csi-unity on 1.21------------------------------------------------------------------------------------------------------------> Checking to see if CSI Driver is already installed------------------------------------------------------Skipping verification at user request||- Installing Driver                                                Success||--> Waiting for Deployment unity-controller to be ready          Success||--> Waiting for DaemonSet unity-node to be ready                 Success------------------------------------------------------> Operation complete------------------------------------------------------

 

 

Once the CSI installation is successful, you can view the worker nodes automatically registered in the Unity-XT console’s host section. As you can observe the below snapshot, the worker nodes starting with prefix of testunitycsicluster01 are seen in the dashboard. Also note that since our use-case is NFS, we have not configured any iSCSI in the Kubernetes nodes., so zero initiators are seen against the host entries

Step-4 Observe the functional components of Unity XT CSI 2.2.0 installation

As you can see from the below output, there are 2 controller and 2 node pods that are in RUNNING status. The count of controller and node pods is configurable in the myvalues.yaml

 

 

kubectl get pods -n unityNAME                                READY   STATUS    RESTARTS   AGEunity-controller-7df987bf94-tbl8k   5/5     Running   0          29sunity-controller-7df987bf94-zb8fb   5/5     Running   0          29sunity-node-bm6gh                    2/2     Running   1          29sunity-node-dvfvd                    2/2     Running   1          29s

 

 

Another thing to note would be that the controller pod has 5 containers (attacher, provisioner, snapshotter, resizer, driver) and node pod has 2 containers (driver, registrar) running that combined together create the necessary routines for Unity-XT CSI

You can issue the commands to get logs from individual containers

 

 

Example for controller podkubectl logs unity-controller-7df987bf94-tbl8k -n unity -c container-name-of-any-of-the-5-containersExample for node podkubectl logs unity-node-bm6gh -n unity -c container-name-of-any-of-the-2-containers

 

 

One of the interesting logs from the driver container of the node pod tells us how the routines are performed to register the worker nodes on the Unity-XT system

 

 

kubectl logs unity-node-bm6gh -n unity -c drivertime="2022-05-19T15:34:02Z" level=info msg="configured csi-unity.dellemc.com" ArrayId=virt2148drw2v6 Endpoint="https://172.24.165.5/" IsDefault=true SkipCertificateValidation=true password="*******" username=admin time="2022-05-19T15:34:02Z" level=info runid=node-0 msg="Starting goroutine to add Node information to storage array" func="github.com/dell/csi-unity/service.(*service).syncNodeInfoRoutine()" file="/go/src/csi-unity/service/node.go:1835"

 

 

OK, we are all done with implementing the Unity-XT CSI on our EKS Anywhere cluster. With this step completed you can move over to the Part-2 where we will deploy and test use-cases leveraging the Unity-XT CSI.

cheers,

Ambar Hassani

#iwork4dell