------------------------------------ ERROR - Critical System Memory Event ------------------------------------ Node = Nodes Extra = {'Nodes': {'169.254.1.1': ['Memory #0x02 - Uncorrectable ECC (UnCorrectable ECC | DIMMB1) (06/10/2023 08:45:16)', 'Memory #0x03 - Uncorrectable ECC (UnCorrectable ECC | DIMMB1) (06/10/2023 08:45:16)', 'Memory Mmry ECC Sensor - Correctable ECC (11/26/2015 12:38:51)']}} RAP = RAP163 Solution = KB 215723 Timestamp = 2023-07-10_170539 PSNT = CKMXXXXXXXXXXX @ 4.8-92.0
Commande : (Commande à distance)
# sudo ipmitool -H <iDrac IP> -U root -P passwd -I lanplus sel elist
Commande : (nœud local)
# sudo ipmitool sel elist
Exemple :
admin@node1:~> sudo ipmitool -H 192.XXX.2XX.107 -U root -P passwd -I lanplus sel elist 1 | 12/04/2021 | 07:29:19 | Event Logging Disabled SEL | Log area reset/cleared | Asserted 2 | 12/29/2021 | 23:00:29 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 3 | 01/26/2022 | 11:44:08 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 4 | 08/03/2022 | 18:31:45 | Power Supply PS Redundancy | Redundancy Lost | Asserted 5 | 08/03/2022 | 18:31:48 | Power Supply Status | Power Supply AC lost | Asserted 6 | 08/03/2022 | 18:43:14 | Power Supply Status | Power Supply AC lost | Deasserted 7 | 08/03/2022 | 18:43:22 | Power Supply PS Redundancy | Fully Redundant | Asserted 8 | 08/03/2022 | 18:51:27 | Power Supply PS Redundancy | Redundancy Lost | Asserted 9 | 08/03/2022 | 18:51:27 | Power Supply Status | Power Supply AC lost | Asserted a | 08/03/2022 | 19:02:03 | Power Supply Status | Power Supply AC lost | Deasserted b | 08/03/2022 | 19:02:14 | Power Supply PS Redundancy | Fully Redundant | Asserted c | 01/19/2023 | 05:38:27 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted d | 02/06/2023 | 02:10:25 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted e | 03/02/2023 | 17:12:15 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted f | 05/09/2023 | 15:56:41 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 10 | 05/09/2023 | 17:16:16 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 11 | 05/09/2023 | 20:57:41 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 12 | 05/09/2023 | 20:59:25 | Unknown #0x2e | | Asserted 13 | 05/09/2023 | 20:59:25 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMB1) | Asserted 14 | 05/11/2023 | 05:43:34 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 15 | 06/10/2023 | 08:43:26 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 16 | 06/10/2023 | 08:45:16 | Unknown #0x2e | | Asserted 17 | 06/10/2023 | 08:45:16 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 18 | 06/10/2023 | 08:45:16 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMB1) | Asserted
Commande :
# sudo dmidecode -t memory | grep "Locator\|Size" | grep -v "Cache\|Volatile\|Cache\|Logical\|Bank"
Exemple :
admin@node1:~> sudo dmidecode -t memory | grep "Locator\|Size" | grep -v "Cache\|Volatile\|Cache\|Logical\|Bank" Size: No Module Installed <-- DIMM is missing Locator: A1 Size: 16384 MB Locator: A2 Size: No Module Installed Locator: A3 Size: No Module Installed Locator: A4 Size: No Module Installed Locator: A5 Size: No Module Installed Locator: A6 Size: No Module Installed Locator: A7 Size: No Module Installed Locator: A8 Size: 16384 MB Locator: B1 Size: 16384 MB Locator: B2 Size: No Module Installed Locator: B3 Size: No Module Installed Locator: B4
Exemple : effacement du journal des événements système (SEL) :
Interrogez l’iDRAC pour obtenir le journal des événements système et vérifiez que l’erreur est présente dans la sortie.
N’oubliez pas qu’avant d’effacer le journal des événements système, recherchez toute autre erreur à corriger. Enregistrez également le journal dans /var/log/hardware
Tel que décrit dans KB 49569
.
Dans cet exemple, 192.168.219.101 correspond à l’adresse IP de l’iDRAC du nœud 1 :
admin@provo~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel list 1 | 01/06/2022 | 04:34:58 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted 2 | 02/03/2022 | 17:15:21 | Physical Security #0x73 | General Chassis intrusion () | Asserted 3 | 02/03/2022 | 17:15:28 | Physical Security #0x73 | General Chassis intrusion () | Deasserted 4 | 08/18/2023 | 01:44:01 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted
Effacez le SEL :
admin@provo:~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel clear Clearing SEL. Please allow a few seconds to erase.
Vérifiez que la liste a été effacée :
admin@provo~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel list 1 | 08/30/2023 | 12:56:55 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted