ECS：xDoctor：RAP163：嚴重系統記憶體事件

------------------------------------
ERROR - Critical System Memory Event
------------------------------------
Node      = Nodes
Extra     = {'Nodes': {'169.254.1.1': ['Memory #0x02 - Uncorrectable ECC (UnCorrectable ECC |  DIMMB1) (06/10/2023 08:45:16)', 'Memory #0x03 - Uncorrectable ECC (UnCorrectable ECC |  DIMMB1) (06/10/2023 08:45:16)', 'Memory Mmry ECC Sensor - Correctable ECC (11/26/2015 12:38:51)']}}
RAP       = RAP163
Solution  = KB 215723
Timestamp = 2023-07-10_170539
PSNT      = CKMXXXXXXXXXXX @ 4.8-92.0

1. 檢查 SEL 記錄，確認節點上有無法修正的錯誤。

命令：（遠端命令）

# sudo ipmitool -H <iDrac IP> -U root -P passwd -I lanplus sel elist

命令：（本機節點）

# sudo ipmitool sel elist

範例：

admin@node1:~> sudo ipmitool -H 192.XXX.2XX.107 -U root -P passwd -I lanplus sel elist
   1 | 12/04/2021 | 07:29:19 | Event Logging Disabled SEL | Log area reset/cleared | Asserted
   2 | 12/29/2021 | 23:00:29 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
   3 | 01/26/2022 | 11:44:08 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
   4 | 08/03/2022 | 18:31:45 | Power Supply PS Redundancy | Redundancy Lost | Asserted
   5 | 08/03/2022 | 18:31:48 | Power Supply Status | Power Supply AC lost | Asserted
   6 | 08/03/2022 | 18:43:14 | Power Supply Status | Power Supply AC lost | Deasserted
   7 | 08/03/2022 | 18:43:22 | Power Supply PS Redundancy | Fully Redundant | Asserted
   8 | 08/03/2022 | 18:51:27 | Power Supply PS Redundancy | Redundancy Lost | Asserted
   9 | 08/03/2022 | 18:51:27 | Power Supply Status | Power Supply AC lost | Asserted
   a | 08/03/2022 | 19:02:03 | Power Supply Status | Power Supply AC lost | Deasserted
   b | 08/03/2022 | 19:02:14 | Power Supply PS Redundancy | Fully Redundant | Asserted
   c | 01/19/2023 | 05:38:27 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
   d | 02/06/2023 | 02:10:25 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
   e | 03/02/2023 | 17:12:15 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
   f | 05/09/2023 | 15:56:41 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMA1) | Asserted
  10 | 05/09/2023 | 17:16:16 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
  11 | 05/09/2023 | 20:57:41 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMA1) | Asserted
  12 | 05/09/2023 | 20:59:25 | Unknown #0x2e |  | Asserted
  13 | 05/09/2023 | 20:59:25 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMB1) | Asserted
  14 | 05/11/2023 | 05:43:34 | Memory Mem ECC Warning | Transition to Critical from less severe | Asserted
  15 | 06/10/2023 | 08:43:26 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMA1) | Asserted
  16 | 06/10/2023 | 08:45:16 | Unknown #0x2e |  | Asserted
  17 | 06/10/2023 | 08:45:16 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMA1) | Asserted
  18 | 06/10/2023 | 08:45:16 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMB1) | Asserted

2. 確認是否因為事件而遺失 DIMM。

命令：

# sudo dmidecode -t memory | grep "Locator\|Size" | grep -v "Cache\|Volatile\|Cache\|Logical\|Bank"

範例：

admin@node1:~> sudo dmidecode -t memory | grep "Locator\|Size" | grep -v "Cache\|Volatile\|Cache\|Logical\|Bank"
        Size: No Module Installed <-- DIMM is missing
        Locator: A1
        Size: 16384 MB
        Locator: A2
        Size: No Module Installed
        Locator: A3
        Size: No Module Installed
        Locator: A4
        Size: No Module Installed
        Locator: A5
        Size: No Module Installed
        Locator: A6
        Size: No Module Installed
        Locator: A7
        Size: No Module Installed
        Locator: A8
        Size: 16384 MB
        Locator: B1
        Size: 16384 MB
        Locator: B2
        Size: No Module Installed
        Locator: B3
        Size: No Module Installed
        Locator: B4

範例 - 清除系統事件記錄 （SEL）：

查詢 iDRAC 以取得系統事件記錄，並確認輸出中有錯誤。 

請記住，在清除 SEL 之前，請先檢查是否有任何其他需要解決的錯誤。此外，請將記錄儲存至 /var/log/hardware 如中所述 KB 49569。

在此範例中，192.168.219.101 對應至節點 1 的 iDRAC IP：

admin@provo~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel list
   1 | 01/06/2022 | 04:34:58 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted
   2 | 02/03/2022 | 17:15:21 | Physical Security #0x73 | General Chassis intrusion () | Asserted
   3 | 02/03/2022 | 17:15:28 | Physical Security #0x73 | General Chassis intrusion () | Deasserted
   4 | 08/18/2023 | 01:44:01 | Memory #0x02 | Uncorrectable ECC (UnCorrectable ECC |  DIMMA1) | Asserted

  清除 SEL： 

admin@provo:~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel clear
Clearing SEL.  Please allow a few seconds to erase.

驗證清單是否已清除：

admin@provo~> ipmitool -I lanplus -H 192.168.219.101 -U root -P passwd sel list
   1 | 08/30/2023 | 12:56:55 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted