This article summarizes the common PowerPath problems and solutions on the Unix platform.

Q1:  How to investigate path dead in PowerPath?

A1: Verify with the “powermt display dev=all” command to check the status of all the paths. If the logs show a dead path that means PowerPath cannot use this path to send I/O requests therefore set it to dead. Generally it is caused by the fiber channel issues (most of the time), LUN operations(removing the LUN from the host at the storage side) or some other software issues. If there is no operation on the LUN or the host, we need to collaborate with the switch vendor to check the switch status, such as the zoning and the port.

After the physical paths being recovered, PowerPath will test the dead path periodically. If the path passes a test, PowerPath will restore it to service and resumes sending I/O to it, or customers restore it manually by the “powermt restore” command, or the “powermt check” and “powermt config” commands.

Q2:  What to do if PowerPath cannot see the LUNs or LUNs are visible but the paths are missing?

A2: Since PowerPath is running on the HBA, it will see the LUNs and the paths only if the HBA can see them. If PowerPath cannot see the LUNs or the LUNs are visible but the paths are missing, firstly you should check if the HBA can see all the LUNs and the paths, each OS vendor provides some commands for inspecting, such as ‘fdisk -l’, ‘lsscsi’ of Linux, ‘format’ of Solaris, ‘ioscan’ of HP-UX, ‘lsdev’ of AIX. EMC provides a software inquiry (inq) which can be executed on all Unix platforms. Customers can obtain it from the “tools/bin” catalogue of emcgrab, or from the FTP:ftp://ftp.emc.com/pub/symm3000/inquiry/.

After adding the executable privilege it can be executed to check the LUN status of the host(by adding -hba/-et/-btl/-sym_wwn/-clar_wwn, type –h or –help for detail information).

If the host cannot see the LUNs, you need to consider if some errors occur during the assigning or scanning (such as cfgmgr of AIX and ioscan of HP-UX) process, Besides, for Solaris with a non-leadville driver, we still need to check the configuration of the persistent-binding.

After the HBA can see all the LUNs and the paths, usually you only need to use “powermt config” to place the newly discovered devices under PowerPath control.

If the HBA could see all the LUNs and the paths, however PowerPath cannot see them, you need to check the following scenarios:

1.     Whether the storage is under PowerPath control(using the “powermt display options” command and see if the status is ‘managed’), whether the LUN has been unmanaged manually before(using “powermt display unmanaged”), and whether the “powermt config” command has been issued after the LUN was assigned.

2.     For CLARiiON/VNX, whether the failover mode is properly configured. The problem that the host can see the LUNs while PowerPath cannot usually is caused by the incorrect failover mode, especially in AIX (In AIX, failover mode need to be set to 3 or 4 for different scenarios). Then you need to modify the failover mode value referring to the document and reboot the host(For failover mode configuration please refer to emc99467：https://support.emc.com/kb/31521).

3.     Whether there are errors occurring during PowerPath configuration, such as AIX ODM package missing or installation errors, Solaris not disabling native MPxIO, etc. Please refer to the relevant document (eg ODM package installation readme file and PowerPath installation guide).

4.     Whether an incorrect PowerPath package has been installed. Before installation please verify that the OS is supported by PowerPath, then download the package and install it according to the document requirement.

If all the above scenarios have been verified but the issues still exist, you need to call EMC 800 to open a case for further analysis.

Q3:  What to do if PowerPath fails during launch or load and the relevant command cannot be issued?

A3: For the Linux platform, if on the initial PowerPath installation, before rebooting the host you need to run “/etc/init.d/PowerPath” to start PowerPath and load the relevant modules and functions (after reboot they will be loaded automatically), or the PowerPath command cannot be issued. On other Unix platforms, PowerPath will be loaded automatically into the kernel. If the relevant command cannot be issued, usually you need to verify the compatibility of the system, to see whether the system is supported by PowerPath, whether the correct version of PowerPath has been installed and any errors occur during the installation. If all of the above has been verified, you could try to uninstall and reinstall PowerPath. If the issues still not resolved, you need to call EMC 800 to open a case for further analysis.

Q4:  PowerPath pseudo names are not persistent across reboot, or not synchronized between two cluster nodes?

A4: After each PowerPath configuration change, run a “powermt save” command to be sure that the changes are saved (usually save to /etc/powermt_custom or/etc/powermt_custome.xml). During reboot PowerPath will automatically execute a “powermt config” to configure PowerPath and a “powermt load” to load the configuration file which was saved before. If the customers change the configuration without save, it is likely to lead to PowerPath loading the old configuration file so that the pseudo device names will change ( sometimes the problems occurre in some specific versions of the powerpath, then the customers need to create a case). After the PowerPath configuration changes, you should run a “powermt save” to save the configuration.

Usually in the cluster environment, customers assign the same LUNs to each node. PowerPath arranged the same pseudo device name for each LUN. In some cases, for example, a LUN was assigned to a node and then was removed with the presence of residual PowerPath device file, which will cause PowerPath pseudo device names different on each host. The “emcpadm” command can be used to rename the pseudo devices, export and import pseudo device name and logical unit data in XML format. The usage of this command can be referred from “emcpadm help” and “man emcpadm”.

Q5:  What to do if I see PowerPath commands show error message, policy displays BasicFailover, UNLIC paths, asb:iopf?

A5: Normally if PowerPath is configured with the correct license, all the paths should display active/active, policy also should be automatically configured to SymmOpt/CLAROpt/Adaptive according to the storage models. If PowerPath shows paths as unlic or Policy as BasicFailover in “powermt display dev=all”, it is probably caused by the incorrectly configured license or that the license is added after the LUN being managed by PowerPath.

The Basic Failover policy is used for one HBA zoned to two ports on the backend storage. If one of the backend ports fails, the I/O traffic will be sent down the other path. There is no load balancing with this configuration. This is a valid policy if there is no License Key. Unlic indicates that unlicensed PowerPath is running (no license key has been installed). In this scenario, all paths are marked unlicensed except one path to each SP. Unlicensed paths cannot become candidates for path failover.

Use the powemmt check_registration command to check the state of the PowerPath license on a host, use the emcpreg command to add/delete the license. If the license is correct but there are still part of the paths are unlic, it may because the paths are recognized before the license is installed, use the command “powermt set policy=co|so|ad（co=CLAROpt, so=SymmOpt, ad=Adaptive）”, so the policy will be changed while the corresponding path will be active.

The setting asb:iopf stands for -Auto standby intermittent I/O failure algorithm. This feature, the intermittent I/O failure algorithm, means the path will not be used unless all other active paths are dead. asb path mode settings do not persist across reboots. Once the customers found that path is set to asb:iopf, they should consider to check the SAN environment to see if this link is stable. Use the “powermt” command to turn off / on the autostandby functions or modify its threshold, or they can also manually set a asb:iopf path to active, please refer to the powerpath document: https://support.emc.com/kb/16228.

                                                                                                                                                                                                            Author: Jiawen Zhang

iEMC APJ

Please click here for all contents shared by us.