PowerVault ME4 - How to troubleshoot CloudIQ upload failures

What is APEX AIOps infrastructure observability i.e. CloudIQ ?   

May 2024: Dell rebranded CloudIQ as APEX AIOps infrastructure observability.  The term may be used interchangeably, this article will continue to use the original reference CloudIQ since PowerVault ME4 documentation and command line interface continue to use this. 

• CloudIQ is a no-cost cloud-native application that proactively monitors and measures the overall health of your storage systems through intelligent, comprehensive, and predictive analytics.
  
  

• CloudIQ leverages the SaaS deployment model to add new features and functionality, pushing the upgrades to the cloud and exposing these new features to users regularly with no downtime.
  
  

• CloudIQ supports PowerEdge servers, PowerVault ME Series arrays and more Dell infrastructure products. For more information read the whitepaper Dell APEX AIOps Infrastructure Observability: A Detailed Review.
  
  

• CloudIQ also provides a Mobile App for iOS and Android allowing monitoring from anywhere.
  
  

• Users must have a valid Dell customer business support account by registering using a proper corporate email address.  For more see How to Register for Access to Dell Technologies Online Support or Upgrade an Existing Account
  
  

• A paid (active) ProSupport or ProSupport Plus agreement and an accepted End User License Agreement (EULA) for each asset to be viewed within CloudIQ is required.
  
  

• PowerVault ME4 series controllers should be firmware G280 or later.
  
  

• SupportAssist must be enabled in PowerVault Manager or CLI to send data from the asset to CloudIQ hosted on Dell infrastructure.
  
  

• Users may access the CloudIQ user interface at https://cloudiq.dell.com or https://apex-aiops.dell.com/ with their Dell support username (email) and password.
  
  

• If you you would like to use this free service to monitor your PowerVault ME array, follow the instructions outlined in APEX AIOps Observability: General Procedures to Onboard PowerVault ME Series Arrays

Verifying connectivity from the PowerVault ME4 array.

Open an SSH session with administrator-level credentials to the ME4 controller A management IP address. You can use a terminal application such as PuTTY.

The following commands can be used to examine the settings and show if the ME4 is sending data:   

show network-parameters
show dns-management-hostname
show dns-parameters
show support-assist
show cloud-iq 

What should I observe from this output?

• The management interfaces on both controllers are configured, enabled, and healthy. By default SupportAssist data is sent through controller A management interface unless controller A is offline.

• The hostname must be unique on each controller.

• Domain Name Server (DNS) IP addresses must be configured on each controller. Sometimes DNS settings may not get set after a controller is replaced causing SupportAssist to fail due to being unable to resolve the dell.com DNS names.

• SupportAssist must show enabled and whether your organization uses a direct TCP connection or a proxy server. SupportAssist establishes an outbound TCP connection on port 443 to any of the following addresses.

• CloudIQ must be enabled.

Stopping services on PowerVault ME4

If the settings are correct and no telemetry data is being sent, carry out the following commands. These commands are not production impacting and do not require a maintenance window.

You must wait a minute for the management controller (MC) services to complete restart.

set cloud-iq off
set support-assist off
restart mc both

If you are using SupportAssist with a proxy, ping your DNS servers and the proxy server to confirm these are accessible from ME management interface on controller A (and B)

The Dell endpoint no longer responds to ICMP requests, therefore use Windows PowerShell on a local desktop to test if a TCP connection can be opened to the Dell endpoint on port 443 from your network.    Linux users can use nc -zv stor.g3.ph.dell.com 443 to test if a TCP connection can be established on port 443 

PS C:\WINDOWS\system32> Test-NetConnection stor.g3.ph.dell.com -Port 443
ComputerName     : stor.g3.ph.dell.com
RemoteAddress    : 143.166.135.19
RemotePort       : 443
InterfaceAlias   : Ethernet 3
SourceAddress    : 10.0.0.50
TcpTestSucceeded : True

Restarting service on PowerVault ME series

Once MC services have restarted and ping response is verified, restart SupportAssist and CloudIQ services in the following order:   

set support-assist on
set cloud-iq on

If you do not onboard your PowerVault ME on CloudIQ, you can disable sending telemetry by turning off the service. to disable run the following command:  set cloud-iq off

Confirm connectivity

Run the following commands to confirm connectivity and send a support bundle to Dell.

check support-assist
show support-assist
show cloud-iq
send support-assist-logs

When CloudIQ is enabled,  PowerVault SupportAssist service sends array performance IO telemetry at 15 minute intervals and system configuration telemetry at 60 minute intervals.  It may take an hour before you observe regular updates when you first onboard the PowerVault ME4.   Login to https://cloudiq.dell.com or https://apex-aiops.dell.com/ or use the CloudIQ smartphone app to view your onboarded Dell infrastructure product status.

If the status of the PowerVault ME4 is still not being updated within CloudIQ, contact Dell technical support for further assistance.  Include a support bundle if support assist logs are not being sent. For instructions, see Dell KB article: ME4: How to Collect support logs on ME4 models