Using Systemd for Automated System Recovery

With the inclusion of support for watchdog hardware, systemd can now perform the function of a watchdog daemon Linux. On Dell PowerEdge systems, this hardware could either be the chipset watchdog timer built into the platform’s chipset (like Intel ICH9) or Dell iDRAC’s IPMI-compliant BMC watchdog timer.

Dell iDRAC provides Automated System Recovery which, in addition to recovering from operating system lock-ups, can capture a screenshot for analysis later. It was necessary to additional software on the operating system to enable this. With newer distributions supporting systemd, this feature works with software available natively in a distribution, eliminating the need for add-on software.

It was however possible to use the watchdogd daemon on Linux, but there was a probability where the daemon itself could lock-up while the rest of the system was operational. systemd acts as the software watchdog for all system services and the BMC watchdog timer acts as the hardware watchdog for systemd itself. So if systemd is nonoperational, there is a good chance that the system is unusable in general. So we now have a more reliable method for all system services, the manager of the services (systemd) to be 'watched' by the BMC’s watchdog timer.

The glue between systemd and Dell iDRAC’s BMC watchdog is the ipmi_watchdog kernel module, which provides Linux watchdog API access to the BMC watchdog using /dev/watchdog. Systemd uses this interface to kick the watchdog periodically.

Setting up systemd with ipmi_watchdog

Systemd can be configured to use iDRAC BMC watchdog with these steps (on Fedora 19):

1. Since the system has two watchdog timers (chipset and BMC), we can use either of them. For this example, we disable the chipset watchdog. The chipset watchdog can be disabled by setting the "operating system Watchdog Timer" option in the System BIOS to "Disabled" (default).
2. Arrive at a timeout value for the watchdog, say 180 seconds.
3. Enable the ipmi_watchdog kernel module to load at system startup with the timeout from above:

• Method 1: Create /etc/modules-load.d/ipmi_watchdog with the following content
  • Options ipmi_watchdog timeout=180
  • Deny list iTCO_wdt  # Optional. If the chipset watchdog is not disabled in BIOS setup.
• Method 2:
  • Install OpenIPMI rpm
    • $ sudo yum install OpenIPMI
  • Set IPMI_WATCHDOG=yes and IPMI_WATCHDOG_OPTIONS with the timeout in /etc/sysconfig/ipmi.
  • Enable the ipmi service to startup automatically
    • $ sudo systemctl enable ipmi
• Enable systemd’s watchdog:
  • Uncomment and set RuntimeWatchdogSec=180 in /etc/systemd/system.conf
• Restart systemd
  • # systemctl daemon-reexec

Test if this works:

1. Check if the watchdog is active
   • $ sudo journalctl |grep -i 'hardware watchdog'   # should show that systemd is set up to use IPMI watchdog.
   • $ sudo ipmitool mc watchdog get        # check if the "Watchdog Timer Is: Started/Running."
2. Test by simulating a Kernel Panic (do not do this on a production system). Ensure that kdump is disabled.
   • $ sudo echo c > /proc/sysrq-trigger
3. After system reset, verify that an image of the failure screen is available in the iDRAC
   • Log in to iDRAC web UI
   • Overview -> Server -> Troubleshooting -> Last Crash Screen.