EKS Anywhere, customize ubuntu OVA for specific requirements

This article is part of the EKS Anywhere series EKS Anywhere., extending the Hybrid cloud momentum 

MOTIVATION FOR THIS ARTICLE

In the previous article of this saga series, we have created baseline ubuntu templates for specific Kubernetes versions using the image-builder process. Now, there are umpteen number of reasons as to why one would like to further customize the ubuntu template. As in the case of this saga series, we will have two specific use-cases which will require customizations on the ubuntu OS template

• Use-case-1: iSCSI-based CSI drivers for Dell PowerStore persistent storage 
• Use-case-2: SSO authentication for our clusters using KeyCloak OIDC with self-signed SSL certificates 

WHAT'S THE CHALLENGE?

The official documentation on EKS Anywhere website did not work for me and appears to be flawed. While that said, I have raised a defect with AWS on their GitHub project Customize Ubuntu OVA workflow is broken after Self creation of Ubuntu images · Issue #3840 · aws/eks-anywhere (github.com)

Alternatively, this article proof-points the exact steps to be executed while building customized ubuntu templates.

IMPORTANT NOTE: Although this procedure has been documented for customizing our ubuntu template created earlier to support Kubernetes version 1.21, the same can be followed to customize the other templates created for higher Kubernetes version by providing the respective OS template name

HOW DO WE START

The below video narrates the exact steps to create the customizations in the ubuntu OS template that we created in the previous article.

You should already have a baseline ubuntu OS template created in the previous article, e.g., ubuntu-2004-kube-v1.21 or similar based on the Kubernetes version.

Also, ensure that the KeyCloak server is already created as per this article such that we can retrieve the self-signed certificate.

STEP-1 SSH into EKS Anywhere administrative machine as image-builder user

Once logged in as an image-builder user and use the below export statements (change the data center name for your vSphere)

Please change the template name to the one that needs to be customized

 

 

 

 

export vsphere_datacenter=IAC-SSCexport OLD_TEMPLATE_NAME=ubuntu-2004-kube-v1.21export NEW_TEMPLATE_NAME=ubuntu-2004-kube-v1.21-customNote that in case of Kubernetes version 1.22, the export statements will change to ubuntu-2004-kube-v1.22 and ubuntu-2004-kube-v1.22-custom

 

 

 

 

Note that govc is already installed in the image-builder user profile along with the vsphere connection attributes. In a nutshell, you can simply execute the below command after changing the Storage and Network attributes as per your environment.

 

 

 

 

govc vm.clone -on=false -vm=/$vsphere_datacenter/vm/Templates/${OLD_TEMPLATE_NAME} -folder=/$vsphere_datacenter/vm/Templates -ds=CommonDS -net=iac_pg ${NEW_TEMPLATE_NAME}

 

 

 

 

 Create a SSH key to support a temporary user named custom-image-builder via cloud-init

 

 

 

 

cd $HOMErm -rf .ssh/custom-image-builder*ssh-keygen -t rsa -f ~/.ssh/custom-image-builder -C "custom-image-builder"

 

 

 

 

We will now create two files, metadata.yaml and userdata.yaml to support the cloud-init method for ubuntu customization. These should be created in the home directory itself of the image-builder user

metadata.yaml

 

 

 

 

cd $HOMEcat > $HOME/metadata.yaml <

 

 

 

 

 

userdata.yaml

 

 

 

 

cd $HOME cat > $HOME/userdata.yaml < 

 

 

 

 

Run the below command to output the SSH public key created in the above step

 

 

 

 

sshpublickey=$(cat .ssh/custom-image-builder.pub) echo $sshpublickey

 

 

 

 

Take the entire output of the above command, i.e., value of the public key and replace the static term named sshpublickey_value in the userdata.yaml file

Once the userdata.yaml has the correct public key value, execute the below command to set the metadata and userdata for the cloned virtual machine

 

 

 

 

cd $HOME export METADATA=$(gzip -c9 /dev/null || base64; }) \ USERDATA=$(gzip -c9 /dev/null || base64; }) govc vm.change -vm "${NEW_TEMPLATE_NAME}" \ -e guestinfo.metadata="${METADATA}" \ -e guestinfo.metadata.encoding="gzip+base64" \ -e guestinfo.userdata="${USERDATA}" \ -e guestinfo.userdata.encoding="gzip+base64" 

 

 

 

 

Power on the cloned virtual machine and get it's IP address

 

 

 

 

govc vm.power -on ${NEW_TEMPLATE_NAME} govc vm.info ${NEW_TEMPLATE_NAME} Note: the IP address will take some time to appear for the second command

 

 

 

 

Once we have the IP address, SSH into the cloned virtual machine using the private key and custom-image-builder as the username. 

 

 

 

 

cd $HOME ssh -i .ssh/custom-image-builder custom-image-builder@ 

 

 

 

 

 

STEP-2 Let the customizations BEGIN

Now that we are SSH'd into the cloned virtual machine successfully, we can start deploying additionally packages or any other required alterations such as ssl-certs, etc.

In this case, we will install iSCSI and tree packages and also deploy the KeyCloak self-signed certificate in the trust store

Install iSCSI and tree packages

 

 

 

 

sudo apt update -y sudo apt install open-iscsi tree -y sudo systemctl enable --now iscsid

 

 

 

 

My KeyCloak server is already deployed as per this article and running at the FQDN keycloak.thecloudgarage.com. We will scan the certificate and create a .crt file to be placed in the trust store of the cloned virtual machine

 

 

 

 

export fqdnOfKeycloakServer=keycloak.thecloudgarage.com sudo echo -n | openssl s_client -connect $fqdnOfKeycloakServer:443 -servername $fqdnOfKeycloakServer \ | openssl x509 > $HOME/$fqdnOfKeycloakServer.crt cat $HOME/$fqdnOfKeycloakServer.crt > $HOME/keycloak.crt sudo cp $HOME/keycloak.crt /usr/local/share/ca-certificates && sudo update-ca-certificates

 

 

 

Next and most importantly, we need to ensure that the virtual machine template does not create duplicate iqn for iSCSI client. This is a trickier part. So as a workaround, we will develop a service that will render an iqn unique to the hostname whenever eks anywhere creates a cluster node with this template. The procedure cannot be documented in-line within this block due to code insert restrictions. Please complete the same using this link

With the above executed, the required customizations are complete. One can, however, extend other customizations required in individual setups

 

STEP-3 CREATE THE TEMPLATE METHODICALLY

If this is not done properly, the customized ubuntu template will not work and EKS Anywhere cluster creation will fail. Follow the exact steps given below

Execute the commands as noted in this gist. Unable to place the commands in this blog post due to certain technical errors related to this blogging platform. So, I have placed them in the gist URL

createandtagcustomtemplate (github.com)

Once you have executed the commands, we will EXIT the SSH session from the cloned virtual machine and we should be back onto the EKS Anywhere administrative machine as an image-builder user

Execute the below commands on the EKS Anywhere administrative machine while logged in as an image-builder user

 

 

 

 

govc vm.power -off ${NEW_TEMPLATE_NAME} govc snapshot.create -vm ${NEW_TEMPLATE_NAME} root govc vm.markastemplate ${NEW_TEMPLATE_NAME} govc tags.attached.ls -r /$vsphere_datacenter/vm/Templates/${OLD_TEMPLATE_NAME}

 

 

 

 

The last command in the above block will output two tags as seen in the below example

 

 

 

 

Please change the Kubernetes version to v1.22 or v1.23 based on the input template being customized govc tags.attached.ls -r /IAC-SSC/vm/Templates/ubuntu-2004-kube-v1.21 Output will look similar to the below tags. Note the tag value for EKS Distro release will change for other Kubernetes versions os:ubuntu eksdRelease:kubernetes-1-21-eks-18

 

 

 

 

We will now attach these two tags to the newly created virtual machine template

 

 

 

 

#Insert the correct tag value from the above output govc tags.attach eksdRelease:kubernetes-1-21-eks-18 /$vsphere_datacenter/vm/Templates/${NEW_TEMPLATE_NAME} govc tags.attach os:ubuntu /$vsphere_datacenter/vm/Templates/${NEW_TEMPLATE_NAME}

 

 

 

 

We will now ensure that the previously created template named ubuntu-2004-kube-v1.21 is deprecated but retained in the Templates folder for backup purposes and the newly created custom template assumes the same name of ubuntu-2004-kube-v1.21

 

 

 

 

Insert a date time for keeping track of old deprecated templates govc object.rename /$vsphere_datacenter/vm/Templates/${OLD_TEMPLATE_NAME} ${OLD_TEMPLATE_NAME}-deprecated- govc object.rename /$vsphere_datacenter/vm/Templates/${NEW_TEMPLATE_NAME} ${OLD_TEMPLATE_NAME} govc session.logout 

 

 

 

 

Now we are all set to create the EKS Anywhere cluster with the customized ubuntu OS template.

Any further customizations once the new template is created have to undergo the same procedure again.

cheers

Ambar Hassani

#iwork4dell