During a PowerStore upgrade task, the upgrade manager service fails with the following error:
Upgrade failed on appliance xxxx with the following message: Error occurred while sending command with the following message: [{"severity":"Error","message_l10n":"NAS NDU Startup operation failed with ERROR : Connection refused: /0:0:0:0:0:0:0:1:18080. Check the connectivity with NAS and try again."}] (0xE04030010003)
Note:
- PowerStoreOS 2.0.0 has resolved the issue that is mentioned. For more information, see release notes.
- Error code "0xE04030010003" is a generic error code which means "Upgrade failed on appliance {0} with the following message: {1}" and is not related to the issue mentioned in this KB.
- This issue is not relevant to "Block only" systems (which are not expected to have "sdnas_container"). It is relevant only to "Unified" (Block + File) systems.
The root cause in this case is due to some internal dependencies for NAS failing to correctly startup.
As a result of this issue, the system is unable to start the NAS services following the failover, causing the upgrade to stop.
The NAS server may also become inaccessible due to its failover operation that is performed during the upgrade process.
The following entries confirming this behavior are found in the journal logs:
Jan 24 00:28:50.209542 PowerStore-A control-path[2639]: 2021-01-24 00:28:50.209 [] [ERROR] [com.emc.bedrock.core.LocalizableMessage|managementdb-0] Invalid resource: 'com.emc.cyclone.messages.sdnas.SdnasCommands: nasNdu.upgrade - A1,b'
Jan 24 00:28:50.209692 PowerStore-A control-path[2639]: 2021-01-24 00:28:50.209 [] [ERROR] [com.emc.bedrock.core.LocalizableMessage|managementdb-0] Invalid resource: 'com.emc.cyclone.messages.sdnas.SdnasCommands: nasNdu.upgrade - A1,b'
Jan 24 00:28:50.213658 PowerStore-A control-path[2639]: 2021-01-24 00:28:50.213 [] [INFO] [com.emc.bedrock.command.CommandServiceVerticle|vert.x-eventloop-thread-3] [eafc5a28-83d6-448e-b059-55aa5c6c263d] About to process Command com.emc.cyclone.contexts.sdnas.domain.ndu.commands.NasNduUpgradeCommand parent id: 98e3f493-a890-48e4-92d5-902dd1ba7e26, instance id: b
Jan 24 00:28:50.217942 PowerStore-A control-path[2639]: 2021-01-24 00:28:50.217 [NDU_NAS] [INFO] [com.emc.cyclone.contexts.sdnas.application.ndu.NduOperationAS|vert.x-eventloop-thread-3] NASNDU_FLOW: prepare_SDNAS_for_upgrade on A1:b started
Jan 24 00:28:50.296345 CKM00202900964-A control-path[2639]: 2021-01-24 00:28:50.296 [NDU_NAS] [INFO] [com.emc.cyclone.contexts.sdnas.application.ndu.NduOperationAS|vert.x-eventloop-thread-3] loadNodeUpgradeConfigTask : Successed to create payload for ansible
Jan 24 00:28:50.299116 PowerStore-A control-path[2639]: 2021-01-24 00:28:50.299 [NDU_NAS] [ERROR] [com.emc.cyclone.contexts.sdnas.application.ndu.NduOperationAS|vert.x-eventloop-thread-3] SDNAS: Node Upgrade Failed. Error in call to ansible REST Call.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /0:0:0:0:0:0:0:1:18080
Jan 24 00:28:50.299330 PowerStore-A control-path[2639]: 2021-01-24 00:28:50.299 [] [ERROR] [com.emc.bedrock.core.LocalizableMessage|vert.x-eventloop-thread-3] Invalid resource: 'com.emc.cyclone.messages.sdnas.Errors: nodeupgrade.ansible.sdnas.error - SDNAS: Node Upgrade Failed. Error in call to ansible REST Call.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /0:0:0:0:0:0:0:1:18080'
A manual startup of the affected services is required. Contact Dell Technical Support or your Service Provider for further assistance and quote this article number.