Faster Block Device Performance with NVMe PCIe SSD based dm-cache on RHEL 7

dm-cache (Device Mapper cache) is a device mapper target to improve performance of a block device (e.g., a spindle device ) HDD by dynamically migrating some of its data to a faster, smaller device (SSD). NVMe based PCIe SSD will act as the cache device. 
For more details about the NVMe based PCIe SSD refer the blog in the link https://www.dell.com/support/article/SLN312382/ dm-cache can be configured using the user space tool set LVM2( logical volume manager tools).

Virtual cache created by dm-cache is made of three physical devices. (Refer Fig1).The Origin device is the slower destination storage device  (HDD).The Cache device is used to store the user data block and the meta device stores the meta data like block placement, dirty flags and other internal data. In this case we use same device [Cache device] for both data blocks and metadata


Figure 1: Steps to setup and configure dm-cache using LVM tools

Hardware Setup:

Origin Device [HDD] : /dev/sda1
Cache Device [NVMe based PCIe SSD] (Block/Meta): /dev/nvme0n1p1 

Software Setup:

Operating system : Rhel 7
Tools : LVM2 user space tools for dm-cache configuration

Configuration Steps:

Following are the steps to configure the Virtual device dm-cache using LVM .

Each step output maps to block with numbering mentioned in Fig.1

1)    Create 100 GB partition on the hard disk [Origin Device]

parted -a optimal /dev/sda mkpart primary 1 100G

This creates /dev/sda1

2)    Create 10 GB partition on the NVMe PCIe SSD disk [Cache Device]

parted -a optimal /dev/nvme0n1 mkpart primary 1 10G

3)    Create a Volume group (cache) with the Origin and Cache device

vgcreate cache /dev/sda1  /dev/nvme0n1p1

Check for the "cache" volume group by vgdisplay


[root@localhost ~]# vgdisplay

--- Volume group ---

VG Name               cache

System ID

Format                lvm2

Metadata Areas        2

Metadata Sequence No  7

VG Access             read/write

VG Status             resizable

MAX LV                0

Cur LV                2

Open LV               0

Max PV                0

Cur PV                2

Act PV                2

VG Size               103.36 GiB

PE Size               4.00 MiB

Total PE              26461

Alloc PE / Size       25399 / 99.21 GiB

Free  PE / Size       1062 / 4.15 GiB

VG UUID               Zd8dNe-6Kdt-7qgY-dmSN-8WHe-4wqQ-euM3Ql


4)    Create  Origin Device logical Volume (origin_device)

lvcreate -l 90%FREE -n origin_device cache /dev/sda1

5)    Create a cache meta data logical Volume  (cache_meta)

lvcreate -l 5%FREE -n cache_meta cache /dev/nvme0n1p1

6)    Create a cache  block data logical Volume  (cache_block)

lvcreate -l 80%FREE -n cache_block cache /dev/nvme0n1p1

7)    Create a cache pool logical volume by combining cache block and cache meta data logical volumes (combining step 5 and 6)

lvconvert --type cache-pool --poolmetadata cache/cache_meta cache/cache_block
 

 
This is the behavior of the tool

Check for the "cache" volume group by lvdisplay
 

root@localhost ~]# lvdisplay

--- Logical volume ---

LV Path                /dev/cache/cache_block

LV Name                cache_block

VG Name                cache

LV UUID                kWYQxP-Jdlr-JdxE-aleB-JJpj-3rmw-Q0cojx

LV Write Access        read/write

LV Creation host, time localhost.localdomain, 2014-06-28 09:05:32 -0400

LV Status              available

# open                 0

LV Size                5.07 GiB

Current LE             1297

Segments               1

Allocation             inherit

Read ahead sectors     auto

- currently set to     256

Block device           253:2

8)    Create a cache logical volume by linking the cache pool (step 7) logical volume and origin_device logical volume (step 4)

lvconvert --type cache --cachepool cache/cache_block cache/origin_device

This creates the virtual cache device (dm-cache)  orign_device as shown below as dm4

[root@localhost ~]# ls -l /dev/cache/origin_device

lrwxrwxrwx. 1 root root 7 Jun 28 09:13 /dev/cache/origin_device -> ../dm-4

[root@localhost ~]#
 

9)    Format the Virtual cache device with a file system and use it

mkfs.ext4 /dev/cache/origin_device

Performance data  

Performance run done using fio tool shows a great improvement of the read,write and read-write mix operations.

Table-1 shows the FIO tool output of performance difference and the improvement in percentage with the dmcache in place  compared to the normal drive (rotational drive) and Fig 2.shows the corresponding graph plot.

The test is done with 1M block size for Random Read,Random Write and Random Read-Write mix (50%) on the Original hard drive(sda) and the drive with Dmcache configured
 

Table 1: Performance Improvement with Dmcache


Figure 2: Performance Improvement with Dmcache

Conclusion : Thus  NVMe PCIe SSD based dm-cache improves the  performance of legacy spindle based hard drives to a greater extent