Service being upgraded will fail on an SDS node upgrade. The SDS node will have CloudLink agent upgraded, but will still be in maintenance mode.
CloudLink 7.0, 7.02, 7.1.0, 7.1.1 does not add a required sleep timer in /opt/emc/extra/pre_run.sh . If it is added manually, PowerFlex Manager 3.8 removes the sleep timer when doing a service upgrade on SDS nodes. Without the sleep timer, the PowerFlex service will start before CloudLink has unlocked the drives. This causes some or all the drives to be in an error state.
This will be fixed in PowerFlex Manager 3.8.3. There is a PowerFlex Manager 3.8.2 patch to ensure that the sleep timer is present.
For PowerFlex Manager 3.8.0 and 3.8.1, use the following method to perform service upgrades that have CloudLink encryption.
Upgrade CloudLink resource with RCM to desired release.
1. Log in to PowerFlex Manager.
2. Go to the Resources tab and select All Resources, and then select a CloudLink Center to upgrade. You only need to upgrade a single node of a CloudLink Center cluster. The remaining nodes are upgraded automatically.
3. Click Update Resources.
4. Choose Allow PowerFlex Manager to perform firmware and software updates now or Schedule firmware and software updates.
5. Click Apply and click Yes to confirm.
Auto upgrade all the agents.
- Login to CloudLink Center.
- The SDS nodes will each have an alarm indicating that the CloudLink agent has not been upgraded yet.
- Set the Machine Agent Upgrade policy to Auto for the machine group that contains the SDS nodes that are being upgraded (This will trigger an upgrade of the CloudLink agent on all SDS nodes):
- Agents > Machine Groups > Action > Modify Group > Machine Agent Upgrade
- Wait several minutes for CloudLink to upgrade all the CloudLink agents on the SDS nodes to the new version. Ensure CloudLink agent alarms on the SDS nodes have cleared.
Update the /opt/emc/extra/pre_run.sh
- On each SDS node that was upgraded, check the /opt/emc/extra/pre_run.sh
- If it does not exist using 'vi' create the file with all the commands shown below.
- If it does exist, using 'vi', add the “sleep 60” timer as shown below.
/opt/emc/extra/pre_run.sh:
#!/bin/bash -f
if [ -f /sbin/svm ]; then
echo svm is installed $(date) >> /var/log/svm-sds
/sbin/svmd -l /var/log/svmd.log -p /var/run/svmd.pid &
end=$((SECONDS+300))
sleep 60
while [ $SECONDS -lt $end ]; do
/sbin/svm unlocked > /dev/null && break
sleep 5
done
fi
4. Change the permissions on the pre_run.sh file “chmod 755 /opt/emc/extra/pre_run.sh”
Reboot nodes and clear errors (optional):
1. Put an SDS node in protected maintenance mode.
2. Record drive letters in use (eg. sda-sdk or sdb – sdl)
3. Reboot node.
4. Once reboot is complete, ensure that there are no PowerFlex errors in PowerFlex GUI. If there are, then clear the errors.
5. Check /opt/emc/extra/pre_run.sh still contains the “sleep 60” timer. If not, add the "sleep 60" timer and repeat from step 3.
6. Check the drive letter order. If drives have reordered then reset drives via scli commands:
scli --query_all_sds
scli --query_sds -sds_id <sds_id from previous command eg. d565c31e00000003>
scli --update_sds_original_paths -sds_id <sds_id eg.d565c31e00000003>
7. If there were PowerFlex errors on the first reboot, the repeat steps 3-6.
8. Take node out of protected maintenance mode
9. Repeat steps 1-8 for all the other SDS nodes to confirm that pre_run.sh is setup correctly.
Start service upgrade
1. Inventory all nodes in PFxM and confirm that CLoudLink is compliant with the new RCM on all SDS nodes.
2. Perform the RCM upgrade on the service.
For PowerFlex Manager 3.8.2, apply the attached patch prior to service upgrades that have CloudLink encryption:
Prior to starting the RCM upgrade, apply the PowerFlex Manager 3.8.2 patch:
- Take a snapshot of the PowerFlex Manager appliance. (remove the snapshot when completed).
- Download the zip file attached to this KB and unzip.
- Copy the Dell-VxFM-asm-deployer-3.8.2-52.noarch.rpm file to the /var/tmp directory on the PowerFlex Manager appliance.
- ssh to PowerFlex Manager appliance.
- Ensure no jobs are running on PFxM.
- Run the following commands on the PowerFlex Manager appliance to apply the patch:
- cd /var/tmp
- sudo rpm -Fvh Dell-VxFM-asm-deployer-3.8.2-52.noarch.rpm
- sudo shutdown -r now
The RCM upgrade can now be performed and there should not be any SDS errors on the encrypted drives when the nodes are rebooted.