Debugging ESXi Machine Check Exception (MCE) PSOD

Description:

ESXi servers may stop with a purple diagnostic screen (PSOD) due to MCE (machine check exception) errors. MCE errors are due to hardware issues.

The MCE purple diagnostic screen displays multiple things. Out of these, the MCi_STATUS register is useful in debugging the problem.

Register Details:

• The MCi_STATUS register consists of information about the machine check error.
• The register is displayed next to the "S:" As shown in the example below:

Bits and their significance:

To debug the error, the low 16 bits of the MCi_STATUS register are important. These bits indicate a simple or compound error.

Option 1: Using Automatic Tool

• NOTE: You can now debug the error with the following tool that automates the steps below: 

  https://vmware-psod-reader.github.io/vmware-psod-reader/

Option 2: Using Manual Steps

1. Simple errors are easy to debug

   0000 0000 0000 0000 -- No Error reported to this bank of error-reporting registers.
   0000 0000 0000 0001 -- Unclassified - Error has not been classified.
   0000 0000 0000 0010 -- Parity error in internal microcode ROM.
   0000 0000 0000 0011 -- External error-BINIT# from another processor caused this processor MCE. Happens only if BINIT# observation enabled during power on.
   0000 0000 0000 0100 -- Functional redundancy check master/slave error.
   0000 0000 0000 0101 -- Internal parity error.
   0000 0000 0000 0110 -- SMM handler tried to execute outside the ranges specified by SMRR.
   0000 0100 0000 0000 -- Internal timer error.
   0000 1110 0000 1011 -- I/O error.
   0000 01xx xxxx xxxx -- Internal unclassified error. Atleast one X must be equal to 1.

2. Compound errors

   000F 0000 0000 11LL - Generic Cache Hierarchy error.
   000F 0000 0001 TTLL - {TT}TLB{LL}_ERR. TLB errors.
   000F 0000 1MMM CCCC - {MMM}_Channel{CCCC}_ERR - Memory controller errors.
   000F 0001 RRRR TTLL - {TT}CACHE{LL}_{RRRR}_ERR - Cache Hierarchy errors.
   000F 1PPT RRRR IILL - BUS{LL}_{PP}_{RRRR}_{II}_T_ERR - Bus and Interconnect errors.

   • F - Form flag

     0 – Normal Filtering
     1 – Corrected Filtering

     Filtering means that some or all the subsequent corrections to this entry in this structure are not posted.

   • TT - Applies to 2 and 4 above. 

     Indicates type of transaction:

     00 - Instruction
     01 - Data
     10 - Generic

   • LL - Applies to 1, 2, 4 and 5 above. 

     Indicates the level in the memory hierarchy where the error occurred

     00 - Level 0 - L0
     01 - Level 1 - L1
     10 - Level 2 - L2
     11 - Generic – LG (It is shown only when processor cannot determine the hierarchy level)

   • RRRR - Indicates the type of action associated with the error. Actions are:

     0000 - Generic Error - ERR
     0001 - Generic Read - RD
     0010 - Generic Write - WR
     0011 - Data Read - DRD
     0100 - Data Write - DWR
     0101 - Instruction Fetch - IRD
     0110 - Prefetch - PREFETCH
     0111 - Eviction - EVICT
     1000 - Snoop - SNOOP

   • PP (Participation) - Describes the role of the local processor in the error.

     00 - SRC - Local processor originated request
     01 - RES - Local processor responded to request
     10 - OBS - Local processor observed error as third party
     11 - Generic

   • T (Time-out) - 1 = Request timed out.
   • II (Memory or I/O)

     00 - M - Memory Access
     10 - IO - I/O
     01 - Reserved
     11 - Other transaction

   • Memory Controller errors - Defined by MMM and CCCC subfields above
   • MMM - Memory error

     000 - GEN - Generic undefined request
     001 - RD - Memory read error
     010 - WR - Memory write error
     011 - AC - Address/Command error
     100 - MS - Memory Scrubbing error
     101-111 - Reserved

   • CCCC - Channel with the error

     0000-1110 - CHN - Channel number
     1111 - Channel not specified

Here is an example of how to analyze an MCE purple diagnostic screenshot:

 

1. Note down the MCi_STATUS register value is 0xfe20004000011166.
2. Convert it to binary:

   1111 1110 0010 0000 0000 0000 0100 0000 0000 0000 0000 0001 0001 0001 0110 0110

3. Take the lower 16 bits of the MCi_STATUS register:

   0001 0001 0110 0110

   Compare it with the compound errors. In this case, this appears to be a cache hierarchy error (type 4).

4. Apply the values:

   F = 1
   RRRR = 0110 – Prefetch
   TT = 01 – Transaction type - Data
   LL = 10 – Level 2 cache

Conclusion:

• It looks like the purple diagnostic screen occurred while there was a prefetch operation on some data in the L2 cache of the processor.
• Thus, this could be a problem with the L2 cache on the processor.
• Therefore, we should replace the CPU first and check if the issue is resolved. In case the issue comes up again, we could replace the motherboard.