------------------------------------
ERROR - Critical System Memory Event
------------------------------------
Node = Nodes
Extra = {'Nodes': {'169.254.1.1': ['Memory #0x02 - Uncorrectable ECC (UnCorrectable ECC | DIMMB1) (06/10/2023 08:45:16)', 'Memory #0x03 - Uncorrectable ECC (UnCorrectable ECC | DIMMB1) (06/10/2023 08:45:16)', 'Memory Mmry ECC Sensor - Correctable ECC (11/26/2015 12:38:51)']}}
RAP = RAP163
Solution = KB 215723
Timestamp = 2023-07-10_170539
PSNT = CKMXXXXXXXXXXX @ 4.8-92.0명령: (리모트 명령)
# sudo ipmitool -H <iDrac IP> -U root -P passwd -I lanplus sel elist
명령: (로컬 노드)
# sudo ipmitool sel elist
예:
admin@node1:~> sudo ipmitool -H 192.XXX.2XX.107 -U root -P passwd -I lanplus sel elist 1 | 12/04/2021 | 07:29:19 | Event Logging Disabled SEL | Log area reset/cleared | Asserted 2 | 12/29/2021 | 23:00:29 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 3 | 01/26/2022 | 11:44:08 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 4 | 08/03/2022 | 18:31:45 | Power Supply PS Redundancy | Redundancy Lost | Asserted 5 | 08/03/2022 | 18:31:48 | Power Supply Status | Power Supply AC lost | Asserted 6 | 08/03/2022 | 18:43:14 | Power Supply Status | Power Supply AC lost | Deasserted 7 | 08/03/2022 | 18:43:22 | Power Supply PS Redundancy | Fully Redundant | Asserted 8 | 08/03/2022 | 18:51:27 | Power Supply PS Redundancy | Redundancy Lost | Asserted 9 | 08/03/2022 | 18:51:27 | Power Supply Status | Power Supply AC lost | Asserted a | 08/03/2022 | 19:02:03 | Power Supply Status | Power Supply AC lost | Deasserted b | 08/03/2022 | 19:02:14 | Power Supply PS Redundancy | Fully Redundant | Asserted c | 01/19/2023 | 05:38:27 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted d | 02/06/2023 | 02:10:25 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted e | 03/02/2023 | 17:12:15 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted f | 05/09/2023 | 15:56:41 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 10 | 05/09/2023 | 17:16:16 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 11 | 05/09/2023 | 20:57:41 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 12 | 05/09/2023 | 20:59:25 | Unknown #0x2e | | Asserted 13 | 05/09/2023 | 20:59:25 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMB1) | Asserted 14 | 05/11/2023 | 05:43:34 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted 15 | 06/10/2023 | 08:43:26 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 16 | 06/10/2023 | 08:45:16 | Unknown #0x2e | | Asserted 17 | 06/10/2023 | 08:45:16 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted 18 | 06/10/2023 | 08:45:16 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMB1) | Asserted
명령:
# sudo dmidecode -t memory | grep "Locator\|Size" | grep -v "Cache\|Volatile\|Cache\|Logical\|Bank"
예:
admin@node1:~> sudo dmidecode -t memory | grep "Locator\|Size" | grep -v "Cache\|Volatile\|Cache\|Logical\|Bank"
Size: No Module Installed <-- DIMM is missing
Locator: A1
Size: 16384 MB
Locator: A2
Size: No Module Installed
Locator: A3
Size: No Module Installed
Locator: A4
Size: No Module Installed
Locator: A5
Size: No Module Installed
Locator: A6
Size: No Module Installed
Locator: A7
Size: No Module Installed
Locator: A8
Size: 16384 MB
Locator: B1
Size: 16384 MB
Locator: B2
Size: No Module Installed
Locator: B3
Size: No Module Installed
Locator: B4
예 - SEL(System Event Log) 지우기:
iDRAC를 쿼리하여 시스템 이벤트 로그를 가져오고 출력에 오류가 있는지 확인합니다.
SEL을 지우기 전에 해결해야 할 다른 오류가 있는지 확인하십시오. 또한 로그를 /var/log/hardware 에 설명된 대로 KB 49569명령을 입력합니다.
이 예에서 192.168.219.101은 노드 1의 iDRAC IP에 해당합니다.
admin@provo~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel list 1 | 01/06/2022 | 04:34:58 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted 2 | 02/03/2022 | 17:15:21 | Physical Security #0x73 | General Chassis intrusion () | Asserted 3 | 02/03/2022 | 17:15:28 | Physical Security #0x73 | General Chassis intrusion () | Deasserted 4 | 08/18/2023 | 01:44:01 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC | DIMMA1) | Asserted
SEL을 지웁니다.
admin@provo:~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel clear Clearing SEL. Please allow a few seconds to erase.
목록이 지워졌는지 확인합니다.
admin@provo~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel list 1 | 08/30/2023 | 12:56:55 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted