Skip to main content
Start a Conversation

Solved!

Go to Solution

RealBOFH

1 Rookie

 • 

7 Posts

59

July 5th, 2024 10:12

PERC h755 disk order in Linux

Dear fellow Dell users,

[tl;dr]

At this moment, my question is: How can I make sure the Virtual Disks are shown to the OS in the order they're created?
Or, is there any other way to get the correct partitions on the desired arrays?

Background

I have to install many servers. Any manual action in the installation process should be automated. We have different types of servers, from different manufacturers. Some servers have different types of disks in them and we use the RAID controllers to create arrays (Dell, for example, calls them "Virtual Disks", other manufacturers refer to them as "Logical disks", etc. I call them arrays). Our relational database machines are designed to have three arrays:

  • OS+binlogs (SSD storage ~900G)
  • DB data (SSD storage ~3,5T)
  • Backups (HDD storage ~4T)

Before you ask: the two SSD storages have different kinds of SSD's under them. One is more suitable for write intensive workloads and one more for read intensive workloads.
We use Redfish to create those RAID arrays. I will leave out the details of how we do that, but that works.
The RAID arrays are created in the order as listed above.

The problem

Even though we create the arrays in the order we like them, they're not configured in the desired order. And that order is not predictable. So, if we use /dev/sda, /dev/sdb and /dev/sdc, we may end up with our OS+binlogs volume on the spinning backup disks, while our DB data volume is written on the 900G SSD's and the backups on the SSD's intended for data. (or any variant you can think of).

What I tried to circumvent this:

  • The RAID controller assigns ID's to the arrays. For a moment we thought these affected how the controller presented the arrays to the driver. The ID's were assigned in decreasing order, so I created the arrays in opposite order. No luck, obviously.
  • Dell seems to support a Firmware Device Order option (which is about the only thing you cannot enable using RedFish!) which promises to address exactly this problem. We tried and proved it's not working.
  • I tried to add some code to the early_command script we have. That code could use the /dev/block/sd?/queue/rotational and /dev/block/sd?/size information to determine which disk is for what. I reasoned: the smallest SSD is always for the OS, the biggest for DB data and the rotating one for backups. Sadly, despite writing a beautiful script, the megaraid_sas driver isn't loaded yet so /dev/block is still empty at this stage. I then tried to download a megaraid_sas.ko file to load but that fails as well. (I made sure it's the same kernel as busybox is running on then, but this is were I have not enough knowledge of the process to fix the load issue).

Since I'm stuck at this on the provisioning part, I'm trying to fix this in the Debian Installer at the moment. So, I will post a detailed version of this in the Debian user forum. The reason I ask here is I still have a vague hope that there is some way we can force a RAID controller to show the arrays to the OS in the order they're created. So, I'm betting on two horses: Fix it from the RAID controller and if that doesn't work: Fix it in the early_command stage of the Debian Installer....

Below some images

DISK ID Linux path
01-OSlogs 239 /dev/sda
02-Data 238 /dev/sdc
03-Backups 237 /dev/sdb

RealBOFH

1 Rookie

 • 

7 Posts

July 9th, 2024 10:58

I worked around this problem in the Debian installer by using `partman/early_command` instead of `preseed/early_command`, however, I'm still curious to the answer to my original question (How can I make sure the Virtual Disks are shown to the OS in the order they're created?).

DELL-Erman O

Moderator

 • 

2.4K Posts

July 5th, 2024 14:40

Hello,

 

As I understood, you want the disks to show up in a specific order (OS, data, backups) but the controller is like nah, I got my own system. This throws a fit in your server setup because the disks are all jumbled up. According to images it likely shows block device names that might be assigned to VDs in an unpredictable order. 

You may creating the disks and give them clear labels (OS, data, backup). This way, even if the order is messed up, you can easily tell which disk is which. And you can check if there recently came up PERC firmware. Since you mentioned using an early command script, make sure it runs at the right point during the boot process. You might also need to manually load the megaraid_sas driver before your script kicks in. 

Let's see how our community contributes to it!

 

 

RealBOFH

1 Rookie

 • 

7 Posts

July 5th, 2024 15:02

@DELL-Erman O​ Thanks for your answer.

You may creating the disks and give them clear labels (OS, data, backup).

I created the Virtual Disks with clear labels, as you can see in the screenshot I attached to the question. However, I never found that name in the block device information in Linux. So, I still can't tell which disk is which. Should I find a way to read the labels in the early command stage, that would absolutely help me fixing this problem.

Since you mentioned using an early command script, make sure it runs at the right point during the boot process. You might also need to manually load the megaraid_sas driver before your script kicks in.

Yes, nice that you bring that up. So, the early command script is a script that runs in the very early stage of a Debian installer. At this moment I fail to load the megaraid_sas driver in it, I don't yet know why. I asked the Debian Community how to load the driver at that point.

I have a reason why I posted these questions in both this community forum and the Debian forum: From Dell, I would like to learn if there is a way to make the Virtual Disks order predictable. Sadly, the FDO option (see note in my opening post about this) doesn't fix the problem.

From the Debian folks I hope to learn how I can fix this in the early command stage of the Debian Installer.

pjwelsh

1 Rookie

 • 

49 Posts

July 9th, 2024 16:49

@RealBOFH​ Sorry, but the general answer will be you won't *always* see them in the order created. The order is derived by many factors and is a major reason to use UUID or labels instead. This is a very long standing situation.

pjwelsh

1 Rookie

 • 

49 Posts

July 9th, 2024 16:56

I should have noted that the command "blkid" or "lsblk -f" are a couple of several ways to to see disk labels (if any).

RealBOFH

1 Rookie

 • 

7 Posts

July 10th, 2024 07:17

Thanks @pjwelsh for your explanation.

major reason to use UUID or labels instead

We could use UUID's in the fstab to mount filesystems (we don't, we use the LVM names), but there is no way to relate the label that we give to a Virtual Disk to the UUID. Same for labels, by the way.

the command "blkid" or "lsblk -f" are a couple of several ways to to see disk labels (if any).

You probably saw the screenshot (of the Virtual Disks in iDRAC webUI) in my opening post? So, I have three Virtual Disks: '01-OSlogs', '02-Data' and '03-Backups'.
Apart from the fact that busybox doesn't have `lsblk` (I'm not sure about `blkid`) build in, those tools do not show the labels we gave them on the controller. Here you see the output of those commands after I installed the server that I also took the screenshot of:

$ sudo blkid
/dev/sdb1: UUID="EkfpPE-0XKo-KfId-nY16-IAPz-6Yzu-suOv77" TYPE="LVM2_member" PARTUUID="581549d6-2788-4e2b-ae55-e0e4800cfa42"
/dev/sdc2: UUID="7E2E-48FB" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="14c0e461-dcee-46e4-8b12-0fc78283375d"
/dev/sdc3: UUID="MFBDEx-kvez-JXsr-sk7t-WYpg-KPgx-uosJ99" TYPE="LVM2_member" PARTUUID="84dfc45d-77f2-4c1f-b596-3ae2968bc50d"
/dev/sdc1: UUID="2FC9-B084" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="393c7bab-3cc9-46c8-8430-7b5c6a7c56ec"
/dev/sda1: UUID="K5Bc4U-EVIE-g7pG-oK06-qUnQ-lPxK-qD8Ytu" TYPE="LVM2_member" PARTUUID="9cd18dc5-b9c1-44e8-b0e3-29c925e45187"
/dev/mapper/vg_system-swap: UUID="1d393c59-fd68-48e2-8fa0-9eca84498291" TYPE="swap"
/dev/mapper/vg_system-mysql_log: LABEL="mysql_log" UUID="a446cfb7-baea-4c6a-9e06-ee55c16b7642" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/vg_system-root: LABEL="root" UUID="649fdf92-2bb8-4ef0-866b-643d6c20273b" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/vg_data-mysql_data: LABEL="mysql_data" UUID="e717d4cb-319a-4f74-a89e-29d55a53b3ce" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/vg_backup-mysql_backup: LABEL="mysql_backup" UUID="0323dcee-3ea1-4a60-a3a2-6fcc34727d43" BLOCK_SIZE="512" TYPE="xfs"
$ lsblk -f
NAME                       FSTYPE      FSVER    LABEL        UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                                                                
└─sda1                     LVM2_member LVM2 001              K5Bc4U-EVIE-g7pG-oK06-qUnQ-lPxK-qD8Ytu                
  └─vg_backup-mysql_backup xfs                  mysql_backup 0323dcee-3ea1-4a60-a3a2-6fcc34727d43      4.3T     1% /srv/backup
sdb                                                                                                                
└─sdb1                     LVM2_member LVM2 001              EkfpPE-0XKo-KfId-nY16-IAPz-6Yzu-suOv77                
  ├─vg_system-swap         swap        1                     1d393c59-fd68-48e2-8fa0-9eca84498291                  [SWAP]
  ├─vg_system-root         ext4        1.0      root         649fdf92-2bb8-4ef0-866b-643d6c20273b       31G    12% /
  └─vg_system-mysql_log    xfs                  mysql_log    a446cfb7-baea-4c6a-9e06-ee55c16b7642    845.4G     1% /srv/mysqllog
sdc                                                                                                                
├─sdc1                     vfat        FAT32                 2FC9-B084                                             
├─sdc2                     vfat        FAT32                 7E2E-48FB                               234.4M     2% /boot/efi
└─sdc3                     LVM2_member LVM2 001              MFBDEx-kvez-JXsr-sk7t-WYpg-KPgx-uosJ99                
  └─vg_data-mysql_data     xfs                  mysql_data   e717d4cb-319a-4f74-a89e-29d55a53b3ce      3.4T     1% /srv/mysql

Anyway, as I wrote yesterday, I was able to work around this in a script that I run during the `partman/early_command` stage of the Debian Installer, so for me this problem is solved. Well, not solved, but worked around :).


Thanks to everyone who read the question and special thanks to those who thought with me and answered!

(edited)

View More

View All

No Events found!

Top