CSI with CentOS 7.6, Kubernetes v1.14.1 and VxFlex OS 3.0

TLDR; this works! 

Challenges faced:

• Corporate firewalls blocking yum, curl, wget, etc with various certificate related errors.
  • Solution: copy your company certs (e.g. companyroot.cer and companySSLDecryptionCAv2.cer into /etc/pki/ca-trust/source/anchors and run "update-ca-trust extract"
    
    
• Docker version 18.09 failing with: 
  "error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.google-containers.appspot.com/containers/images/sha256:cfaa4ad74c379e428b953c9aa9962e25a6de470a38b3b62ea2feaef1bfb30e0d: remote error: tls: handshake failure, error: exit status 1"
  • Solution: Issue is due to 18.09 requiring TLS 1.2 which fails behind many corporate firewalls. Back rev to an earlier version such as 18.6.3.
    
    
• Feature gates causing services to fail to start. 
  • Solution: Instead of putting all feature-gates on a single line, put them down as one line each. e.g.:
    • - --feature-gates=VolumeSnapshotDataSource=true
      - --feature-gates=KubeletPluginsWatcher=true
      - --feature-gates=CSINodeInfo=true
      - --feature-gates=CSIDriverRegistry=true
      - --feature-gates=BlockVolume=true
      - --feature-gates=CSIBlockVolume=true
      
      
• rbac-config.yaml --- error: error parsing rbac-config.yaml: error converting YAML to JSON: yaml: line 6: did not find expected '-' indicator
  • Solution: Retry it by copying and pasting it in notepad before inserting into rbac-config.yaml

• Running sh install.vxflexos -- Error: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout
  • Solution: This one tore a lot of hair out - issue was due to Flannel pod network add-on, installed Romana instead:  " kubectl apply -f https://raw.githubusercontent.com/romana/romana/master/containerize/specs/romana-kubeadm.yml" - after this it immediately worked.

\\

I hope this helps save someone else a lot of time in the future. From start to finish of this Kubernetes deployment, it probably took roughly 18 hours due to these hurdles along the way. Next time it should only take a few minutes.