Dell PowerEdge R750 and R7525 servers that are configured with rear 4x2.5" hard drive backplane are equipped with three additional rear system fans (7,8,9). Under normal system operation, these fans may repeatedly report that they have been removed and reinserted into the server. These error events are recorded in both the System Event Logs (SEL) and iDRAC9 Lifecycle Logs.
Onsite end users can visually identify when the rear 4x2.5" hard drive backplane is present by checking the back of the server for the 2.5" drive slots in the upper left third of server. Seer the image below to identify the 4x2.5" backplane cage.
Remote end users can identify when the rear 4x2.5" hard drive backplane is present through the iDRAC9 web user interface. Go to
Storage >
Overview >
Enclosures. Expand available enclosure details and check for the Enclosure Location = Rear.
Systems exhibiting this issue may cycle through events indicating that the rear fans 7-9 are missing and approximately five seconds later are inserted. These FAN0029 and FAN0028 events are recorded in the SEL and Lifecycle Log like the examples below.
System event log example:
2022-02-15 01:32:10 3076 Fan 7A is either removed, incorrectly installed, or not present.
2022-02-15 01:32:10 3077 Fan 8A is either removed, incorrectly installed, or not present.
2022-02-15 01:32:10 3078 Fan 9A is either removed, incorrectly installed, or not present.
2022-02-15 01:32:15 3079 Fan 7A inserted.
2022-02-15 01:32:15 3080 Fan 8A inserted.
2022-02-15 01:32:15 3081 Fan 9A inserted.
Lifecycle log example:
2022-02-15 01:32:15 3324 FAN0029 Fan 7A is either removed, incorrectly installed, or not present.
2022-02-15 01:32:28 3325 FAN0029 Fan 8A is either removed, incorrectly installed, or not present.
2022-02-15 01:32:29 3326 FAN0029 Fan 9A is either removed, incorrectly installed, or not present.
2022-02-15 01:32:30 3327 FAN0028 Fan 7A is inserted.
2022-02-15 01:32:31 3328 FAN0028 Fan 8A is inserted.
2022-02-15 01:32:32 3329 FAN0028 Fan 9A is inserted.
Also, the lifecycle logs may contain Fan RPM events (FAN0000, FAN0005) and Fan Redundancy events (RDU0001, RDU0002).
2022-02-15 01:38:55 3413 RDU0001 The fans are redundant.
2022-02-15 01:38:54 3412 FAN0005 Fan 9A RPM is within range.
2022-02-15 01:38:54 3411 FAN0000 Fan 9A RPM is less than the lower warning threshold.
2022-02-15 01:38:53 3410 FAN0005 Fan 8A RPM is within range.
2022-02-15 01:38:52 3409 FAN0000 Fan 8A RPM is less than the lower warning threshold.
2022-02-15 01:38:52 3408 FAN0005 Fan 7A RPM is within range.
2022-02-15 01:38:51 3407 FAN0000 Fan 7A RPM is less than the lower warning threshold.
2022-02-15 01:38:51 3406 RDU0002 Fan redundancy is lost.
The iDRAC9 queries the system fans for RPM readings every five seconds. When these cycles of events occur in rapid succession, the iDRAC is misinterpreting some returned data bytes as errors leading to the false "removed" error messages. On the following query from the iDRAC, the fan and iDRAC correctly sync leading to the "inserted" messages within logs.
Dell iDRAC9 firmware version 5.10.10.52 (April 2022) addresses mishandling of these returned RPM responses. An upgrade to this version should address the issue.