PowerVault ME5024 iSCSI 25gbe interface speed?

Tobiwan,

While we don't normally address performance issues, what I would first want to check is if you followed the deployment guide and the best pratices section in the admin guide.

If not then I would start there as that will be the best way to maximize the performance, or determine if something isn't configured correctly. 

Let me know what you see.

Hi Chris

thanks for taking your time with this issue. really appreciated.

I have studied all the ME5 Manuals, the best practices guide, the ME4 guides (as ME4 seems very similar) and also the iscsi best practices from VMWare. I'm confident I've read every publicly available piece of information about iscsi, ME5 / ME4 and XXV710 coming from Dell, VMWare and Intel.

I have also a case open with tech support, and E-Mail communication with the account manager.

In all Dell Documents, ME5024 performance is only mentioned once: In the specification sheet on page 1: https://www.delltechnologies.com/asset/en-us/products/storage/technical-support/dell-powervault-me5-ss.pdf

"Using fast Intel Xeon processors, Dell PowerVault ME5 storage implements a dual-active controller
architecture, 12GB/sec read and 10GB/sec write throughput and uses a 12Gb SAS backend protocol for
rapid capacity expansion."

The question how much throughput can be achieved on a single interface remains unanswered here.

Now I'm aware setup for iSCSI can be complicated, and there is ways to misconfigure it. That's why I asked the specific question above in bold.
(Not even in my specific case, lets just assume the disk array is powerful enough, the network card is optimal the configuration is perfect)

Tobiwan,

Let me do some testing to confirm what you should be seeing, I will need a little more information to do so though. Would you confirm the following for me?

What is the Read/Write percentage set to?

What is the block size set for the software your using?

Is it configured as Random or Sequential?

Which tools are you using?

Lastly, would you confirm if the best practices guide you utilized is the one found here?

Let me know.

Hi Chris

thanks again.

I started this by just a quick benchmark using CrystalDiskMark, to verify performance. I then switched to iometer to have better control over workers / queue / scenario.

I used the default profile for CrystalDiskMark:

Read then write:
-Sequential with 1MB block size, 8 Queues, 1 Thread -> ~1200MB/s Read, 1000MB/s Write
-Sequential with 1MB block size, 1 Queues, 1 Thread -> ~800MB/s Read, 600MB/s Write
-Random with 4K block size, 3 Queues, 1 Thread -> ~300MB/s Read, 170MB/s Write
-Random with 4K block size, 1 Queue, 1 Thread -> 3 ~13MB/s Read, 8.5MB/s Write

Note: I didn't try to fit this to a real world scenario, nor is that my goal. I just wanted to know maximum throughput to know my iSCSI setup is right. I also tried 512K block size (to align with ME5024 chunk size), more Queues, more Threads. Same maximum. The 4K random reads are excellent. But they wont bottleneck the 25gbe interface obviously.

With iometer I benched with 100% Sequential Read, 512K block size, 2 workers. I get the same numbers (as both benchmarks with these settings seem to reach the maximum I measure)

I like to clarify it doesn't matter how I bench this. If I spin up a 2nd VM and let both VMs bench 100% read, Sequential, 512K blocks with iometer I get the same total throughput (measured on the storage) as if I do the same benchmark on 1 VM. Same happens with 3 VMs.

However if I bench with 2VMs on separate ESXI Servers (therefor using two different iSCSI paths / interfaces) the throughput doubles. If I do that from 3 ESXI Servers, it triples.

This tells me the bottleneck must be one of these:
-the network interface card xxv710 (which I updated firmware and drivers for already, newest from DELL)
-the network interface on the ME5024 (maybe internally connected to 12gb SAS? -> This is what I want to know)
-the VMWare iSCSI software adapter somehow limiting everything.
-Some other VMWare limitation

Regarding the best practices guide, yes, I've worked through that one, too. I have MPIO and delayed ACK set like recommended (however I do not believe that they make any difference in this scenario)

Hi Joey

thanks for chiming in.

I had Jumbo Frames enabled from start (On the ME5024, the vSwitch and the VMKernel Interface as per best practice)

I tried what happens if I disable it and set MTU to 1500, and it hurt the speed by a little, so I reenabled it. As it is direct attached (no switch in between) I think it should work as expected.

I'm not seeing anything in VMWare logs that would indicate retransmits or other MTU related issues. (Not a single network related error)

I'm beginning to think it works like this: Dell specifies 12GB/s read and 10GB/s write for the whole system. We got 8 iSCSI Ports. But also 2 12Gb/s SAS ports for expansion. 10 ports total. 12GB/s divided by 10 = 1.2GB/s = My maximum read speed. If that makes sense.

Hello @tobiwan,

Meanwhile, waiting for Chris to be back and run some test, I'm unsure if jumbo frame configuration may help. https://dell.to/3xmgBVm

Try it. Page 72 https://dell.to/3BOiVqV OS vswitch set MTU to 9000. And on ME5024, in iSCSI port settings, enable Jumbo frames, page 48: https://dell.to/3BOiVqV

Let us know if it helps.

One short note about the Me4/5.  If you only create one pool... this pool and all the volume from that pool are assigned to one controller. The other controller just sits there and do.... nothing waiting.

I dont want to complain about the transfer rate... for us is more important IOPs and Latency.

About throughput use IOmeter and increase the blocksize to 1 or even 16MB.

Regards,
Joerg

Hi Joerg

Yeah I know the pool is connected to one controller and the other controller will usually not handle any I/O for that volume.

But I have a 25gb/s interface that should theoretically handle ~2.5GB/s of throughput in each direction. And I'm only getting 12gb/s total throughput. So I'm questioning things. As my servers are directly attached, and there is 3 of them, I can't pool the throughput to be available on all the servers.

BTW I agree that IOP and Latency is more important. In general I just want to make sure it's working at optimal performance.

I just configured and ran a 16MB block size 100% sequential read benchmark in iometer:

Increasing the block size over 1MB doesn't seem to improve performance at all.

This is how it looks on the storage:

Tobiwan,

Sorry for the delay. From what I see from testing is that the 4k examples you provided are indeed the expected performance, but unfortunately the 1MB block size examples are beyond my testing ability. The max I can test at, with regard to the block size, is 256KiB. 

Now if I test with your current configuration, with the following parameters;

Virtual Mode - 8x25Gb iSCSI SFP28 - with 80% read / 20% write - at 256KiB - Random (12 3.84TB SSD Adapt 8+2) then the expected performance shows at 9.6K IOPS / 2400 MiB/s.

Now if you were to change that same config from ADAPT to a Raid 5 (11+1) configuration you would see an increase to 12K IOPS / 3000 MiB/s

Unfortunately any testing above those then I would suggest you to work with the support team, as they will have additional abilities that we do not.

Hope this helps.

Hi Chris

is this with one active path / interface? Or all 8 links?

Thanks

"...-Using 1 VM on ESXI_A and 1 VM on ESXI_B: I get 2400MB/s of (total) read throughput (and 2000MB/s write) -> Array performance is not the cause for my issue..."

When this is the true than the problem looks like that the swISCSI is the bottleneck on ESXi side or its the single 25Gport on ME5 side.  To find out which side is the problem you need a pSwitch between and than connect only a single port from the ME5 to the pSwitch and work with 2 Hosts.
Right now you use a direct Connect iSCSI setup right?

When 2 Server can read with 2400MB in total the problem is not the controller and your 12 SSDs.

Regards,
Joerg

If you only have 1 port cabled, all IO would go through that port. Also note that each pool has an owning controller, so MOST IO will go through connections to that controller only. Now ALUA can sometimes use connections to the other controller, but they will be minimal and slower that the primary path.

@Origin3k
exactly my thinking.

Unfortunately, I don't have a 25gbe physical switch available to see if it's the ME5024 interface or the vswitch / software iscsi on the vmware side of things.

@DELL-Chris H 
you say all IO would go through that port. Which is surely correct, because it's the only path. But how much throughput will it allow?

Surely there must be a Dell engineer somewhere who knows how the 25gbe interface is connected to the SAS backplane internally.

Today its all swISCSI.