Exploring Isilon as storage for rancher/K8S

Steffen,

I don't mind helping out. I'll work up some details in the morning. Of note, EMC is releasing a newer version of the isilon CSI soon. So some of the steps I use may not apply with the newer update. Main thing to start with is adding the feature gates to your cluster. But I am assuming you are using custom clusters. If you are not, then I won't be much help. I am only using custom clusters in our environment. Essentially that is RKE clusters.

I posted a reply with how to do this. It seems to have disappeared, or maybe it went to moderation. I'll repost if I don't see it later today.

Hi,

Let me try to answer your concerns. First, you are right we (Dell EMC) don't officially support RancherOS but we do support the kubernetes version under and we have customers using the CSI driver with Rancher.

To your questions :

1) If I were you, I would choose the CSI driver option since it comes with Snapshot support, AZ selection, Volume expansion in the next release, etc.

2) If your question is should I use helm or operator for the deployment, I would say it depends. With helm, it is easier to tweak the installation since it is "just" a template ; the current version of the installer supports only helm v2, helm v3 is on the roadmap. On the other hand, the installation with the operator is easier and doesn't require any extra modules like tiller, but like I said, if you need to tweak things you might be blocked.

In the post you refer, indeed with PostgreSQL, we had to mount the volume with the subPath option (https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath) to circumvent rights issue. But to be honest, I'm not sure this is related to Rancher itself.

HTH

Hi Steffen,

Thanks for the post.
While we wait for @cocampbe to repost I thought that I would take a stab at #1 and #2.

You certainly can use the basic nfs-client-provisioner with the Isilon array but as you mentioned you have to manually create the export on the Isilon for it to use (or automate that in some other way)

If you are expected to need a lot of somewhat temporary directories/exports on the Isilon for your K8S pods then using the CSI driver may be best way to go.  It has the added benefit of automativally creating the directory and export on the Isilon automation when requested during pod deployment  (or simple standalone PVC creation).  The retention policy can be set as well to delete/remove the export and filesystem when the pod goes away.  From a scale up/scale down perspective this is pretty useful.

Thanks
Bryan

I was using the nfs-provisioner, but we went with the isilon CSI to for the quotas. If you use the nfs provisioner, there are no quotas. In my experience, new developers on k8s tend not to clean up and they tend not to think about space. Also, they may see that the pvc is set for a set amount, but there is nothing enforcing that amout.

Just for the sake of sharing knowledge. This applies if you leave the reclaim policy as Retain. I found that if you have orphaned pv's, ie, no longer in use. It's best to change the policy to delete for the PV.

kubectl patch pv PV_NAME -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'

 If you do this, the CSI will take care of cleaning up the export, quota, etc. If you just delete the PV, you then have to do manual cleanup on the isilon, YUCK!

Hi, thanks everyone for the quick responses. I am particularly happy about the insights when the CSI driver is required, and when simple NFS might be sufficient. As a side node, after setting up the nfs-client-provider with the simple Linux server as StorageClass, I never touched the NFS server afterwards for volume creation etc, so it might not be so much more work with "creating the directory and export on the Isilon automation when requested during pod deployment" using the nfs-client-provisioner.

I will talk to our storage admins if it makes sense to try the CSI on a testing environment. Here @cocampbe has a fork (and gist) to work with helm3, while @Flo_csI recommends the Operator based option which is easier but less flexible. That'll also give me reason to read current K8S docs to learn what Operators are in first place In any case, the new CSI version might be available by the time I get to this.

Thanks again everyone, and I hope the thread also helps newcomers in the future. Case closed for now. Steffen

Ah, another very good insight. I might revise my current opinion and bite the bullet.

I couldn't resist and just tried out of curiosity how the rancher app catalogue copes with the forked repo. The rancher app catalogue is supposedly helm-compatible, and you point it to a git repo to install helm charts. Since the last Rancher upgrade to 2.4.x I can specify whether a chart is helm2 / helm3 flavour. It seems that it is a bit more restrictive than real helm, and complains about naming, I guess not liking "2" at the beginning and/or "+" in the name. Certainly Rancher is not supported, so I just mention that as feedback and not as an issue.

Thanks, Steffen

Error in chart(s): failed 2vols+restore template name csi-isilon-forked-inofficially-2vols+restore: [a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')];template name csi-isilon-forked-inofficially-2vols+restore: [a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')];

I don't use the rancher catalog except for some testing. I don't think many of us using it find it quite useful. What I am bout to mention is not related to the CSI. But if you have helm charts, or other code available in a git repo, you might look into fluxcd, or maybe even argo. I haven't used argo, but fluxcd is really cool. It's a gitops operator. SO you define your yaml and put it in the repo. Flux then syncs it up every 5 mins, by default. It's nice because you get IaC, an audit trail, etc. And you know who did what. With plain k8s, it's hard to know who made a change when they use kubectl. 

There is another project called tanka by grafana. Seems really interesting. I haven't looked into it yet. But seems like a way to get away from all the yaml. Din't get me wrong, i like yaml. Use it all the time with ansible. But the yaml for k8s can get confusing.

Well, they updated the isilon CSI. It now uses helm 3. I am using 1.17 so I had to copy k8s-1.16-values.yaml to k8s-1.17-values.yaml. I also hado to change to change the verify.kubernetes to accommodate rancher, as they use the term controlplane and not master. Deployment went well. I do have an issue with a few pods. But I think that is related to maintenance on our isilon.