Configuring NVDIMM-N on PowerEdge Servers with Windows Server 2019

This article describes how persistent memory (NVDIMM) works and how to configure NVDIMM on a PowerEdge Server with Windows Server 2019.Persistent Memory (also known as Non Volatile Memory (NVM), or Storage Class Memory (SCM)) is a random access memory type whose contents remain even when system power goes down in the event of an unexpected power loss, user initiated shutdown, system crash etc. Dell EMC introduced support for NVDIMM-N from their 14th generation of PowerEdge servers. Microsoft provides NVDIMM-N support From Windows Server 2016 and extends further in WS 2019 with new features. The NVDIMM-N resides in a standard DIMM (memory) slot, placing data closer to the processor thus reducing the latency and fetch maximum performance.

Before using NVDIMM-N on 14th generation of PowerEdge servers, we need to make sure that in SystemSetup/System BIOS Settings/Memory Settings/Persistent Memory, Non-Volatile DIMM is enabled and Battery Status is "Present-Ready"


Figure 1 - BIOS Memory / Persistent Memory Configuration
   

 
NVDIMM Driver Architecture Overview:

There are two types of device objects related to persistent memory on Windows – physical NVDIMMs and logical persistent memory disks. In Device Manager, physical NVDIMMs appear under "Memory devices", while logical disks are under "Persistent memory disks". The screenshot below shows a system with 2 NVDIMMs:


Figure 2 - Device Manager

All NVDIMM-N devices are controlled by the nvdimm.sys driver, while the logical disks are controlled by the pmem.sys driver. Both drivers are the same for all types of persistent memory, such as NVDIMM-N and AEPs. 

Both types of device objects are created by scmbus.sys, the bus driver for persistent memory. You can find the bus driver object in Device Manager under "System Devices".


NVDIMM-N behavior when battery is not proper or it is configured as "Read Only" at BIOS:
When a battery is not functional, or NVDIMM-N is configured "Read Only" in BIOS, by design the OS will not make PMEM disk as read only and it will not stop ongoing write operations. The default behavior is that Windows will allow read/write to the disk, but all the written data will go off after reboot.

If user need to make NVDIMM as read only when battery is not proper or it has been configured as "Read Only" at BIOS then below setting has to done at registry of OS :
"HKLM\System\CurrentControlSet\Services\pmem\ReadOnlyOnPersistenceLoss (REG_DWORD): non-zero to make it read only on persistence loss"


New features with WS 2019

1. Label support and Namespace management

With WS2019, OS provides support for Label and Namespace management. When NVDIMM will be exposed to OS first time or after factory-reset/secure erase, Namespace will not be created by default. Users need to create the namespace before using it as storage device. User can use the below commands to manage namespace of NVDIMM.


Figure 3 - Available PMEM PowerShell Cmdlets

Examples:


Figure 4 - List PMEM Unused Regions, PMEM Physical Devices, and PMEM Disks


Figure 5 - List/Remove PMEM Disk

---------------------------------
PowerShell Cmdlets
---------------------------------
#Get-PmemDisk
- Returns one or more logical persistent memory disks.
- The returned object has information about size, atomicity type, health status, and underlying physical devices.

#Get-PmemPhysicalDevice
- Returns one or more physical persistent memory devices (NVDIMMs).
- The returned object has information about size(s), RFIC, device location, and health/operational status.

#New-PmemDisk
- Creates a new disk out of a given unused region.
- Writes out the labels to create the namespace then rebuilds the SCM stacks to expose the new logical device.
- Optional parameters:
   - FriendlyName gives the persistent memory disk a friendly name.  Default is "PmemDisk<N>"
   - AtomicityType lets you set BTT.  Default is "none".

#Remove-PmemDisk
- Removes the given persistent memory disk. It accepts the output of Get-PmemDisk.
- Deletes the namespace’s labels and then rebuilds the SCM stacks to remove the logical device.
- Requires user confirmation, which can be overridden with -Force.

#Get-PmemUnusedRegion
- Returns aggregate pmem regions available for provisioning a logical device.
- Returned object has a unique region ID, total size, and list of physical devices that contribute to the unused region.

#Initialize-PmemPhysicalDevice
- Writes zeroes to the label storage area, writes new label index blocks, and then rebuilds the SCM [Storage class memory] stacks to reflect the changes.
- Requires user confirmation, which can be overridden with -Force.
- This cmdlet is intended as a "big hammer" recovery mechanism.  It is not recommended for normal use.

2. NVDIMM-N Node Interleaving

WS 2019 provides NVDIMM-N node interleaving feature which further enhances performance of NVDIMM-N with various application. To enable it, @ SystemSetup/System BIOS Settings/Memory Settings/Persistent Memory "NVDIMM Interleave" should be enabled and per CPU even number of NVDIMM-N (>= 2) should be available.
After enabling this OS sees it different physical device but it exposes single logical disk under "Persistent memory disks" 


Figure 6 - NVDIMM/PMEM Interleaving in Device Manager


Figure 7

3. NVDIMM-N can now be exposed to Hyper-V VMs
With WS2019, NVDIMM can be exposed to Hyper-V VMs directly. Get more detail here.


This article was writen by Deepak Kumar.
Have any comments, questions or suggestions? Please contact us on WinServerBlogs@dell.com