PowerVault ME5: How to use CLI to troubleshoot SupportAssist connectivity

• Opening an SSH session to the PowerVault ME5 .
• Review system health, network and SupportAssist configuration.
• Review alerts and event logs for information relating to SupportAssist .
• Check SupportAssist connection to Dell EMC secure remote servers .
• How to check CloudIQ is enabled .
• What do the endpoint values mean .
• Contacting Dell technical support for assistance .

To enable SupportAssist on PowerVault ME5 the following prerequisites are required.
 

1. Access Key and PIN must be generated to enable key exchange between SupportAssist on the ME5 array and Dell EMC secure remote server.  See KB 000198043: ME5: How to generate an Access Key and Pin to enable SupportAssist
2. The PowerVault ME5 must be able to resolve DNS host names.
3. SupportAssist direct connections require an Internet connection to be able to establish a session to external hosts esrs3-core.emc.com and esrs3-coredr.emc.com on outbound TCP port 443.
4. SupportAssist gateway connections require Secure Connect Gateway - Virtual Edition v5.x on an internal management network. See KB 000200129: PowerVault ME5: How to configure SupportAssist to Connect via Gateway Server

Opening an SSH session  

Open a secure shell (SSH) session using a terminal application such as PuTTY and capture the output to text file for later review If you cannot install a terminal application, Microsoft Windows 10 and Windows Server 2019 and later, come with a command-line SSH client, example:  (C:\> ssh manage@<controller.management.IP.address>). Unlike terminal applications you cannot capture the output to text file for later review.

Review system health, network, and SupportAssist configuration

The following commands can be used to learn about the system state and network configuration settings that are needed to ensure SupportAssist is functioning. More information about the command-line interface (CLI) can be found in the Dell PowerVault ME5 Series Storage System CLI Reference Guide Execute each of these commands in turn.  

• show service-tag-info

• Record the Dell service tag to ensure you are working on the intended PowerVault ME5 array, and where needed, that the access key and PIN two factor authentication that is used to establish secure key exchange between the ME5 and the Dell EMC secure remote server is being generated and applied based on this service tag.

• show system

• Check PowerVault ME5 system health status. If health is degraded, this may need to be addressed before resolving the SupportAssist connection issue. If there is a hardware fault contact your local Dell technical support.
• For dual controller module systems the Other MC Status should be Operational.

• show network-parameters and ping

• The controller module management ports health must be OK. Link Speed should be 100mbps or 1000mbps.
• It must be possible to ping the Gateway address from the CLI.
• Dual controller systems should have both management ports that are configured and connected to the management network.

• show dns-parameters

• Each controller module must be configured to resolve DNS names using a DNS server. Ensure that the Name Servers are configured correctly. Usually, where ICMP response is not blocked, it should be possible to get a ping response from the DNS server to confirm network connectivity between the PowerVault ME5 array and DNS server.
• DNS is needed to resolve hosts esrs3-core.emc.com and esrs3-coredr.emc.com used in direct connections.
• Note: Neither of these hosts respond to ping requests.

• show support-assist

• Is the SupportAssist State set to Enabled?
• What is the current Operation Mode?
• What is the connection Preference? Direct or Gateway.
• Is a proxy server being needed for Internet connections in your organisation?

• show support-assist-telemetry-status

• If SupportAssist has not been configured or no connection has been attempted, Latest Upload Status for each category is blank, and Latest Upload Time displays N/A.
• Messages that the payload has been Sent successfully to Dell Service Enabler means the SupportAssist service is running, however, to determine if the payload was successfully sent to the Dell EMC backend you must examine the ME5 event logs for Event 655 to determine if the telemetry payload reached its destination. 

Review alerts and event logs for information relating to SupportAssist.

PowerVault ME5 introduces alerts, a storage enclosure health and notification system that is designed to identify actionable conditions and promote best practices.
 

• show alerts

Check the alert status for any events that are related to SupportAssist that are unresolved. Filter using the following command: show alerts unresolved

• show events

Example: 

In this example the alert flags Lost connectivity to the Dell SupportAssist backend, the event log indicates that SupportAssist is not registered with the Dell backend. An access key and PIN needs to be generated for this service tag and used to enable secure key exchange to authenticate the device.

# show alerts unresolved

10 System                           WARNING       No       Yes          Lost connectivity to the Dell SupportAssist backend.

- Check Alerts for any network related problems.
- Contact local administrator about recent network changes.
- If network configuration is correct and issue persists contact Dell support.

# show events last 20

A6552      2022-08-21 06:32:59  362   INFORMATIONAL  Scheduler: SupportAssist is not registered with the Dell backend. (task: CHKeepAlive, task type: CHKeepAlive)
A6553      2022-08-21 06:43:00  362   INFORMATIONAL  Scheduler: SupportAssist is not registered with the Dell backend. (task: CHCloudIQStats, task type: CHCloudIQStats)
A6554      2022-08-21 06:58:00  362   INFORMATIONAL  Scheduler: SupportAssist is not registered with the Dell backend. (task: CHCloudIQStats, task type: CHCloudIQStats)
A6555      2022-08-21 07:13:00  362   INFORMATIONAL  Scheduler: SupportAssist is not registered with the Dell backend. (task: CHCloudIQStats, task type: CHCloudIQStats)
A6556      2022-08-21 07:28:00  362   INFORMATIONAL  Scheduler: SupportAssist is not registered with the Dell backend. (task: CHCloudIQStats, task type: CHCloudIQStats)
A6557      2022-08-21 07:32:59  362   INFORMATIONAL  Scheduler: SupportAssist is not registered with the Dell backend. (task: CHCloudIQConfig, task type: CHCloudIQConfig)

From PowerVault Manager you my see a scenario such as described in KB 000200734: ME5: Alert: Lost connectivity to the Dell SupportAssist backend and Event: Scheduler - SupportAssist is not registered with the Dell backend

Check SupportAssist connection to Dell EMC secure remote servers.

• Determine what type of connection is being used (Direct or gateway?)
• Determine if you can connect to a site on the Internet on port 443. This is to test if external connections to the Internet are blocked by firewall or no Internet connection. 
• Is a proxy server used by this organisation?  If so these details may need to be entered using the command:  set support-assist-proxy

This is an example of a direct connection test where the access key and PIN authentication have not been entered,

check support-assist-connection mode direct
SupportAssist Connection
------------------------
Connection State: Not Connected

  Endpoint
  --------
  Mode: direct
  Endpoint: https://esrs3-core.emc.com
  Status: fail
  HTTP Status: 0
  Message: 
  Status Detail: SSLError
  Proxy Type: none

  Endpoint
  --------
  Mode: direct
  Endpoint: https://esrs3-coredr.emc.com
  Status: fail
  HTTP Status: 0
  Message: 
  Status Detail: SSLError
  Proxy Type: none

In the above example, a direct test connection fails on both Endpoints.

• The Connection State reports Not Connected.
• Status is fail,
• Status Detail on both is SSLError.

The reason for the SSLError on both endpoints indicates that key exchange cannot happen between the ME5 and Dell secure remote server.

It is possible Status Detail: SSLError only occurs on one endpoint and not the other. If so key exchange has been enabled and there may be a problem with one of the Dell EMC hosts.
 

How to resolve the SSLError condition when it occurs on both endpoints simultaneously. 

To resolve the SSLError when it occurs on both endpoints you must generate an access key and PIN using this service tag. Follow the instructions in KB 000198043:ME5: How to generate an Access Key and Pin to enable SupportAssist

To enable SupportAssist using CLI, generate the access key and PIN, then use the following commands substituting the correct values for the access key and PIN:

# set support-assist enabled 
To use SupportAssist you must run "show support-assist eula" and read the EULA.
Do you accept the EULA? (y/n) y

# set support-assist-authentication access-key ABCD1234 pin 1234

A successful direct connection test looks like this. There will be a successful connection to at least one endpoint. 

# check support-assist-connection mode direct
SupportAssist Connection
------------------------
Connection State: Connected

  Endpoint
  --------
  Mode: direct
  Endpoint: https://esrs3-core.emc.com
  Status: success
  HTTP Status: 200
  Message: OK
  Status Detail: Success
  Proxy Type: none

  Endpoint
  --------
  Mode: direct
  Endpoint: https://esrs3-coredr.emc.com
  Status: success
  HTTP Status: 200
  Message: OK
  Status Detail: Success
  Proxy Type: none


Success: Command completed successfully. 

How to use the check support-assist command to check if port 443 is open

A useful test to determine if outbound traffic is allowed from the PowerVault ME5 on port 443 is to use mode gateway to test an Internet connect.   

• If this is successful, it indicates that the network connection between the PowerVault ME5, and Internet is Ok.
• If this fails work with your network administrator to enable outbound TCP connections on port 443. Some organisation may need to specify a proxy server.

# check support-assist-connection mode gateway uri https://www.dell.com:443 proxy false
SupportAssist Connection
------------------------
Connection State: Connected

  Endpoint
  --------
  Mode: gateway
  Endpoint: https://www.dell.com:443
  Status: success
  HTTP Status: 200
  Message: OK
  Status Detail: Success
  Proxy Type: none


Success: Command completed successfully. (2022-08-27 04:13:58)

How to test the connection between the PowerVault ME5 and Dell Secure Connect Gateway Virtual edition

SupportAssist on PowerVault ME5 can be configured to use a direct connection or a gateway connection to proxy the connections using Secure Connect Gateway virtual edition (SCG/VE).  For more see KB 000200129: PowerVault ME5: How to configure SupportAssist to Connect via Gateway Server

To configure SupportAssist preferred connection to use the gateway use the following commands:

# set support-assist-connection connection-preference gateway
# set support-assist-connection gateway-urls https://r.s.x.y:9443

Then test the connection to the gateway appliance on port 9443 from the PowerVault ME5. Substitute the r.s.x.y with the correct IP address or DNS hostname for the SCG/VE appliance.
 

# check support-assist-connection mode gateway
SupportAssist Connection
------------------------
Connection State: Connected

  Endpoint
  --------
  Mode: gateway
  Endpoint: https://r.s.x.y:9443
  Status: success
  HTTP Status: 200
  Message: OK
  Status Detail: Success
  Proxy Type: none


Success: Command completed successfully. (2022-08-30 19:41:59)

How to check CloudIQ is enabled.

Dell CloudIQ is an online software as a service portal that is free to access for customers with PowerVault ME arrays that have an active ProSupport or ProSupport+ contract. To send telemetry data from the PowerVault ME5 array, SupportAssist and CloudIQ support must be enabled. 

To onboard a PowerVault ME5 array in CloudIQ see KB 000022224: CloudIQ: General procedures to onboard PowerVault ME Series arrays to CloudIQ

To determine if CloudIQ is enabled on your PowerVault ME5 use the command:  show support-assist 

• SupportAssist State must be set as Enabled
• CloudIQ must be set as Enabled 

To enable SupportAssist to send CloudIQ telemetry data use the command: set support-assist cloud-iq enabled

# show support-assist
SupportAssist
-------------
SupportAssist State: Enabled
Operation Mode: Normal
Auto-case Creation: Disabled
CloudIQ: Enabled                     <-- CloudIQ telemetry data can be sent from this system
Connection Preference: Direct

  Gateway Endpoints
  -----------------
  Gateway 1: 
  Gateway 2:
  Gateway 3:


  Proxy Information
  -----------------
  Proxy State: Disabled
  Protocol: HTTP
  Host: 
  Ports: 
  User Name: 


Success: Command completed successfully. (2022-08-27 04:01:40)

Check support-assist endpoint responses

Contacting Dell technical support for assistance.

If you require assistance from Dell technical support to resolve getting SupportAssist, or CloudIQ enabled.
 

1. Gather the support bundle from the PowerVault ME5. See KB 000196715: ME5: How to collect PowerVault support logs
2. If possible, save the text output from the SSH session showing the CLI output. 
3. Contact your local Dell technical support for assistance.