PowerStore: Health Check shows all green when there are warnings

In all current versions of PowerStoreOS, upgrades and health checks are not always clear as seen from PowerStore Manager (UI).

When you run a PUHC (pre-upgrade health check) or an upgrade on a PowerStore, the process will check the health of the system. While the PUHC works, it doesn't always report problems in a way that's easy to notice.

Here is a view of the Settings > Upgrades screen. The % Complete / Status column shows a green check mark, which seems to indicate everything is fine.

Example from the Job Details:

You can see all the steps taken by the health check and everything displays a green check.
To see the full details, highlight one of the steps, for example Health check - node A:

Now you can see that the system has two warnings.

Any warnings should be addressed before continuing with the upgrade. 

In this example, there is a cyc_net error and a Host IO load error. 

• One error is simple. We should wait for a period of lower IO (or maintenance window) before performing the upgrade.
• The second error requires troubleshooting.

 

PUHC check from the Service Console

Example from the same PUHC as seen from the command line:

Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Checking Active Imports..................................... OK  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Check : Checking network connection configuration  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Warning : Appliance nodes are not connected to the same switches (247390384816168)  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Action : Please make sure the nodes in the appliance are connected to the same ToR switches  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Check : Checking network connection configuration  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Warning : Appliance nodes are not connected to the same switches (247390384816168)  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Action : Please make sure the nodes in the appliance are connected to the same ToR switches  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Check : Checking if system cpu usage is normal  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Warning : CPU utilization level too high (247390385274892)  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Action : Reduce system load prior to NDU  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Check : Checking for Mapper status  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Warning : Host IO load on Node A is higher than recommended for running an upgrade (247390384816199)  Jun 16 18:18:47 xxxxxxx-A cyc_puhc[62722]: Action : Reduce host IO load and retry

 

Unlike the UI, the warnings and network errors show up in detail at the command line.
The system will allow the user to upgrade with warning level errors (but not error or critical level messages), whether these warnings are seen or not. 

The green check mark is used to indicate that the step itself completed running. This was found to be confusing and will be fixed in a future release.

Fix

The fix for this issue will be included in a future version of PowerStoreOS.
 

Workarounds

Method 1

Ensure that you click all the way down to Health check - node A and Health check - node B on all appliances in the cluster. Take this step before performing any upgrades. 
 

Method 2

Observe the PUHC from the command line. 
The following command will allow you to watch the health check live for any problems. Anything that is observed should be addressed. 

svc_journalctl -t cyc_puhc -f

Choose one of the following commands to look at the last PUHC performed.

svc_journalctl -t cyc_puhc -r

 -- or --

svc_journalctl -t cyc_puhc -rg "Warning"

 
Note: The version of the command with "Warning" as a search term creates a much shorter CLI output but has the potential of missing an "information" level message.

The CLI also shows more details about any errors encountered.

• If no warnings are observed, then you can proceed.
• If warnings are observed, please address them before proceeding.