Powerscale CSI driver initialization fails with "Authorization required"

Hi,

We have a strange authorisation issue with Powerscale CSI driver that I am not sure where it comes from.

So, we have the following setup:

Isilon - OneFS 9.2.0

Separate access zone for CSI - zone-CSI (for security reasons don't want to allow System AZ access for CSI). AZ zone-CSI has base directory set to /ifs/CSI. Also a separate IP pool pool-CSI is created for AZ zone-CSI with range 192.168.10.51-53. Pool-CSI is in different groupnest and subnet than System zone

.
In zone-CSI created custom Role role-CSI with following privileges as per CSI documentation:
ISI_PRIV_LOGIN_PAPI Read Only
ISI_PRIV_NFS Read Write
ISI_PRIV_IFS_RESTORE Read Only
ISI_PRIV_NS_IFS_ACCESS Read Only
ISI_PRIV_IFS_BACKUP Read Only

ISI_PRIV_QUOTA and ISI_PRIV_SNAPSHOT privileges are not set since zRBAC in 9.2.0 does not allow to add them. However, for the time being we don't plan to set quotas and snapshots in Kubernetes so these 2 privileges should not be an issue.

In the AD authentication provider for zone-CSI created an user account isicsi and added it to the role-CSI to get listed above privileges.

Our aim with that setup is to separate isicsi account in its own AZ and to prevent it to have privileges in all other access zones which will be the case if it is defined in system zone.

We use Kubernetes v1.19.6
CSI driver version 1.6 with following parameters (parameters with default values are not listed):

myvalues.yaml
# "isiPort" defines the HTTPs port number of the PowerScale OneFS API server
isiPort: "8080"

# The name of the access zone a volume can be created in
isiAccessZone: "zone-CSI"

# "volumeNamePrefix" defines a string prepended to each volume created by the CSI driver.
volumeNamePrefix: k8s

# The default base path for the volumes to be created, this will be used if a storage class does not have the IsiPath parameter specified
# Ensure that this path exists on PowerScale.
isiPath: "/ifs/CSI"

secret.yaml
isilonClusters:
- clusterName: "Isilon"             # logical name of PowerScale Cluster
username: "isicsi"                    # username for connecting to PowerScale OneFS API server
password: "xxxxxxx"                # password for connecting to PowerScale OneFS API server
endpoint: "192.168.10.51"       # HTTPS endpoint of the PowerScale OneFS API server
isDefault: true                          # default cluster (would be used by storage classes without ClusterName parameter)
skipCertificateValidation: true  # indicates if client side validation of server's SSL certificate can be skipped
isiPath: "/ifs/CSI"                      # base path for the volume(directory) to be created on PowerScale

Endpoint 192.168.10.51 is different than the endpoint for system zone. But as per Powerscale documentation the privilege ISI_PRIV_LOGIN_PAPI gives option to login or call API on the specific access zone with zone aware RBAC/.

 

With above setup when we try to initialize CSI driver Isilon returns 401 error msg="init client failed for isilon cluster 'Isilon': 'Authorization required'".
At the same time when we try to login to Isilon with browser on same endpoint 192.168.10.51:8080 with same user isicsi it is successful and it shows management GUI interface with appropriate permissions defined by the role role-CSI (i.e. almost everything is disabled but full permission to create, delete and manage NFS exports).

Also from K8s linux box:

doing curl -kv https://192.168.10.51:8080 -uisicsi is successful

doing curl -kv https://192.168.10.51:8080/platform/latest/ -uisicsi fails with Authorization required just the same as CSI driver initialisation

So could anyone help me to understand what could be wrong and why API calls from CSI driver fail with "Authorization required"?

I found the following article, but don't think it is relevant in the case with CSI:
https://www.dell.com/community/Isilon/API-Auth-Changes-with-Recent-Security-Patches-anti-CSRF-token/td-p/7170612

Any support will be much appreciated.