Dell OpenManage error /tmp/secupd on CentOS 6 after upgrade

Hello,
as a further information, I can see two times the `/tmp/SECUPD' file system mounted.
I can unmount them and the error goes. Anyway if I reboot the server the two file systems and the errors come back again.

Andrea

[root@mywdb9 ~]# df
df: `/tmp/SECUPD': Input/output error
df: `/tmp/SECUPD': Input/output error
Filesystem            1K-blocks      Used  Available Use% Mounted on
/dev/mapper/vglocal00-root00
                       82438832  43800572   34443960  56% /
tmpfs                 198429444         0  198429444   0% /dev/shm
/dev/sda1                243823     34432     196591  15% /boot
/dev/mapper/vglocal00-data00
                     1667338816 640548764 1026790052  39% /data
/dev/mapper/vglocal00-tmp00
                        1998672     18412    1875404   1% /tmp

[root@mywdb9 ~]# mount
/dev/mapper/vglocal00-root00 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vglocal00-data00 on /data type xfs (rw,noatime)
/dev/mapper/vglocal00-tmp00 on /tmp type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sdb on /tmp/SECUPD type vfat (rw,shortname=mixed)
/dev/sdc on /tmp/SECUPD type vfat (rw,shortname=mixed)

[root@mywdb9 ~]# umount /tmp/SECUPD

[root@mywdb9 ~]# umount /tmp/SECUPD

[root@mywdb9 ~]# df
Filesystem            1K-blocks      Used  Available Use% Mounted on
/dev/mapper/vglocal00-root00
                       82438832  43395464   34849068  56% /
tmpfs                 198429444         4  198429440   1% /dev/shm
/dev/sda1                243823     34432     196591  15% /boot
/dev/mapper/vglocal00-data00
                     1667338816 640548764 1026790052  39% /data
/dev/mapper/vglocal00-tmp00
                        1998672     18408    1875408   1% /tmp

That's not the OpenManage temp drive, it's the iDRAC USB virtual drive, aka Virtual Media, used for OS installs. You place an CD/USB pen/disk on your machine, and the iDRAC presents that to the local hw as a boot drive. It's handy, but is also a know source of some issues...

I've also updated all firmware on a R310 today (CentOS 6.8 and OMSA 8.40) and saw that error,  together with a few FAT read errors in the console... in my case, I really have an USB external disk there, so all the probing of DSU may have triggered something.

Anyway, a know workaround for that is to go to the iDRAC config, and disable Virtual Media. I haven't got around to do that on that machine, maybe tomorrow...

Hello,

thanks for the reply.
Ok, the correct definition is the iDRAC USB virtual drive (Virtual Media). However this virtual drive is "placed" by or because the Dell OMSA.
I rebooted the server many times and only now just after the OMSA upgrade I see these errors.
We do not have any physical device attached.
We will test the workaround that you suggested and we will evaluate if we can apply it, if it will work. Anyway we would like to have a permanent solution with the OMSA software, because it seems an installation bug/issue.

Thanks,
Andrea

Hi!

I also have the problem. After mounting /tmp as LVM I noticed this Error. I just wanted to mention, that I checked the iDRAC virtual media, and I don't have a valid license for that. Assuming, i don't think that this is causing the error. 

I also get following error:
sd 11:0:0:0: [sdc] Assuming drive cache: write through in dmesg

and this in /var/log/messages

-> FAT-fs (sdc): Directory bread(block XX) failed
-> systemd-readahead[990]: open(/opt/dell/invcol.wTwUoYaSZI/NIC_QLogic_Firmware/x86_64/rhel-6/libdfc.so.11.0.232) failed: Too many levels of symbolic links

If I uninstall OMSA the errors are gone - I think we are all have the same issue. I run centos7 with OMSA 8.4.

PS: I don't want to take over this Thread, I just wanted to leave some information I collected.  

BG
Cookie

Hi!

I'm also experiencing the same issue.

Running Centos7 with OMSA 8.4 on a R730 set up today.

dmesg shows:

--> FAT-fs (sdc): Directory bread(block XX)

--> systemd-readahead[992]: open(/opt/dell/invcol.B7goibexhN/NIC_QLogic_Firmware/x86_64/rhel-6/libdfc.so.11.0.232) failed: Too many levels of symbolic links

--> sd 11:0:0:0: Attached scsi generic sg2 type 0
sd 11:0:0:0: [sdb] 2112 512-byte logical blocks: (1.08 MB/1.03 MiB)
sd 11:0:0:0: [sdb] Write Protect is off
sd 11:0:0:0: [sdb] Mode Sense: 23 00 00 00
sd 11:0:0:0: [sdb] No Caching mode page found
sd 11:0:0:0: [sdb] Assuming drive cache: write through

Sometimes after a reboot, in df it shows /dev/sdc mounted as /tmp/SECUPD but most of the time I get "df: `/tmp/SECUPD': Input/output error".

When i uninstall OMSA and reboot. No error where thrown to dmesg ..

PS: I just wanted to leave my information here, maybe we are all having the same issue. 

BG
Cookie

Do you have tmp mounted with the nodev option ?

On my case, it seems that it's that what is causing the issue.

I uptaded two R320 machines (with iDRAC7 Basic), the first one had the same errors, so on the second one, I've remounted /tmp as dev/exec, and I haven't seen any erros/warnings

Our hardening scripts add that to the mount options, for instance this our standard (on CentOS 6)

# mount  | egrep tmp
tmpfs on /dev/shm type tmpfs (rw,noexec,nosuid,nodev)

And after I've remount tmpfs:

# mount | egrep tmp
tmpfs on /dev/shm type tmpfs (rw,nosuid)

In my case, the first machine (the one with nodev/noexec) still has the /tmp/SECUPD mount points, but the second one doesn't, so I',m guessing this is just temporary, I've also used DSU to upgrade them to the latest firmware....

I'm going to reboot them latter today. see what happens after the reboot

For me, it seems that uninstalling the iDrac OMSA Packages fixed the problem. I didn't need to configure iDRAC. 

yum remove srvadmin-idra* -y

After reboot, I don't see any error messages or that /tmp/SECUPD is mounted.

Maybe you are in the same situation as me.

BG
Cookie

I've just rebooted them, and I can confirm that, even with a basic iDRAC license, which does not allow the use of Virtual Media, I've got the same error on both...

I don't have time for more tests today, but I will uninstall the idrac RPM's and check if that also works for me... 

I just made another reboot. Now the /tmp/SECUPD mount is there again. Thought I figured out, what the problem is ...

EDIT 11/11/16 11:10:
I have two server in the same setup. I forgot on one to do something .. the Problem doesn't accure atm.

HOW TO:

• (optional) - iDrac Settings - Virtual Media - USB Controller Management (or something like that) set to -- iDrac only --> i don't know if this is also needed
• yum remove srvadmin-idra* -y --> uninstalling 
• mv /opt/dell/srvadmin/sbin/lx64/invcol /opt/dell/srvadmin/sbin/lx64/invcol.bak
• mv /opt/dell/srvadmin/sbin/lx32/invcol /opt/dell/srvadmin/sbin/lx32/invcol.bak
• mv /opt/dell/srvadmin/sbin/invcol /opt/dell/srvadmin/sbin/invcol.bak (i don't exactly know what this is, i just found this on a DELL Support Page, if the Device is removed but still showing up - Maybe someone of you guys know?)
• REBOOT
• check /var/log/messages

My Selinux is also disabled, I got some line in dmesg where Selinux check the SBD/SDC drive .. 

Please advice me, if it also helps you guys. At the moment are both servers "fixed" .. I rebooted several times to test if it was luck, but i seems the be fixed.

BG,
Cookie

Create a file /etc/modprobe.d/disableusb.conf  (can be named anything) with this content:

install usb-storage /bin/true

Note: This will disable a user's ability to mount usb storage, but it fixes the issue, and it's more secure.

root can still insmod usb-storage if needed.

This is occurring on a number of my servers... the setting of usb-storage /bin/true to disable usb storage appears to prevent the issue... but if you want to leave usb-storage enabled, I found the issue to be some stale mount information where /tmp/SECUPD still thinks it's mounted possibly several times.  

I am able to fix the issue manually without rebooting by running "umount /tmp/SECUPD" once for every instance of the error... so in the below example, df reported the /tmp/SECUPD io error twice... I had to umount /tmp/SECUPD twice.

Originally prints twice with a single df -h command (sending standard out to /dev/null for clarity)

[root@nodename ~]# df -h > /dev/null
df: `/tmp/SECUPD': Input/output error
df: `/tmp/SECUPD': Input/output error

Now unmounting /tmp/SECUPD once... removes one of the above errors.

[root@nodename ~]# umount /tmp/SECUPD

[root@nodename ~]# df -h > /dev/null
df: `/tmp/SECUPD': Input/output error

Now unmounting /tmp/SECUPD again... removes the last remaining error.

[root@nodename ~]# umount /tmp/SECUPD
[root@nodename ~]# df -h > /dev/null
[root@nodename ~]#

I have the same errors on CentOS 7.9 :

# df >/dev/null
df: ‘/tmp/SECUPD’: Input/output error
# mount | grep /tmp
/dev/sdr on /tmp/SECUPD type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/sds on /tmp/SECUPD type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
# ls /dev/sdr /dev/sds
ls: cannot access /dev/sdr: No such file or directory
ls: cannot access /dev/sds: No such file or directory
# 

Is there a solution that does not involve uninstalling OMSA or disabling the "usb-storage" module ?

If I umount /tmp/SECUPD manually twice, will this issue come back after reboot ?

Hello,

I don't see other workaround, let see if other customer can suggest anything.

Thanks