RAID 5 array shown as foreign during expansion rebuild

Question

Dear Dell Community,

I'm currently facing an issue with my RAID 5 array on an R7515 server with a PERC H730P Mini RAID controller. I'd like to share the details of my problem and seek assistance from the community.

Initially, I had a RAID 5 array consisting of three 14TB disks. To expand the storage, I inserted two new disks of the same type, and the RAID controller started the reconstruction process. During the reconstruction, which reached around 23% progress after 2 days, a power outage occurred. I gracefully shut down the machine before the UPS ran out of power. Unfortunately, the hydro repair took longer than expected, and the server eventually lost power completely.

Upon restarting the server, the RAID controller recognized all five disks as having a foreign RAID configuration. When trying to import this configuration, it displayed an error stating that "the configuration is not complete." The disks themselves are relatively new and working normally. I've tried various troubleshooting steps, including using the perccli tool to analyze the RAID configurations. Interestingly, the perccli tool shows that the two newly inserted disks have a RAID 5 mark and the original 25.465TB size, but the initial three disks are recognized as standalone disks without any RAID mark. Perccli also reported that there are no dirty data in the cache.

All disks have remained in their original position and have not been moved or touched ever since.

I'm wondering if there's any way to recover the RAID array and the valuable data stored on it. I'd appreciate any suggestions, guidance, or insights on this issue from the community. Thank you in advance for your assistance.

DELL-Chris H · Accepted Answer

While the retag would be the route I would look into, I am not sure if the other drives changed the structure of the VD yet. If you are certain that the data is copied and protected, we can try with the original drives to see if we can recover it to then retry the expansion. Again this is going to risk the data, so I wanted to let you know.
Retagging is the process of deleting the Virtual Disk, and then recreating it exactly as it was (raid level, stripe size, etc), but without initializing the virtual disk, as initializing would format the drives deleting the data. So essentially you would delete it, recreate it with the original drives without initializing the VD, boot to OS and verify data, then retry the expansion with the other drives. Now you may likely need to format the new expansion drives, to clear the VD flags from it before trying to expand.

DELL-Chris H · Answer

JeremyLLL,

Will you confirm a couple things for me?

Is the Virtual Disk bootable to the OS, and is the data intact?

Is there a complete backup of the data?

I ask as anything we do can put that data at risk, so if there is no backup I would look into a 3rd party data recovery.

Let me know what you see.

JeremyLLL · Answer

Thank you so much for your prompt reply! I would like to double-check the "retag" procedure with you. Since the issue occurred during the expansion from 3 disks to 5, your suggestion is to use the original 3 disks for the retag, right? That means A. I should remove the newly added disks 4 and 5 B. clear the foreign configuration on the first 3 disks C. recreate the RAID 5 array D. and recreate the virtual disk without reinitializing it. Then, I should attempt to boot into ESXi. Please kindly correct me if I have misunderstood any part of this process.

By the way, I am currently backing up these disks. Each backup takes about a day, and I am now working on the third one. I will try the retag and update you on the situation as soon as all the images are created, which should be in two or three days. Thank you again for your kind support.

JeremyLLL · Answer

Thank you soooo much!!! Will try it as soon as the back up finished. Have a nice day!!!

JeremyLLL · Answer

Hi Chris,

I'm very happy and excited to see your reply! Since the disaster, I've been extremely careful in every step I've taken to ensure the data remains intact. Here's a summary of what I've done so far:

I checked the controller's status in iDRAC and tried to import the foreign configuration with all five disks mounted, the first four disks mounted, and the first three disks mounted, but all attempts failed.
I used the perccli tool to “show” detailed information about the array status.
I did not clear the configuration or change any parameters using perccli, as I believe it would be too risky, especially given that the rebuild process involves expanding the array from three to five disks.
I came across the suggestion to "retag" the array on the forum, but I haven't tried this yet
Finally I shut down the server completely, removed each disk, and inserted them one by one into another r7525 server with an HBA controller and enough space. I then used the 'dd' command under Linux to create image files of each disk to ensure nothing worse would happen to the data.

To answer your questions:

The Virtual Disk is not bootable to the OS.

the RAID array is used purely for data storage and is formatted as VMFS6. It is shared by several virtual machines, which access the data stored on the array.

I would greatly appreciate any suggestions or guidance you can provide to help me recover the data.

Thank you for your assistance!

DELL-Chris H · Answer

Everything looks correct except C, which should be "Delete the Virtual disk".

C and D of your steps were somewhat redundant, but C would represent where you would delete the existing Virtual Disk, then followed by the steps in D.

DELL-Chris H · Answer

You too, let me know what you see. Keep in mind that this may not work and you have to restore, as it dies in process, I just want you to be prepared for the possibility.

JeremyLLL · Answer

Good morning Chris, I just retagged the 3 original disks, and now ESXi can recognize the VMFS filesystem for this virtual disk. I can see all the partition parameters through the 'partedUtil getptbl' command. However, it is unable to mount it back as a datastore. I am not sure if this is due to data loss, but I will investigate further under VMware topic later. I noticed through iDRAC that a 'Background Initialization' process is currently underway, even though I am certain I chose 'No' for the initialization option during the retag process. For now, I will wait for this process to complete and see if I can mount the disk back as a datastore after that. I am hoping for the best outcome.

PowerEdge HDD/SCSI/RAID

RAID 5 array shown as foreign during expansion rebuild

Was this post helpful?