IO Meter Performance Stats?

Hi all,

I've done the same tests on my vnx3100 with iscsi and Nfs. The results are equal.

Matteo

Talarion,

Thanks for the information.

I have a question about your setup.  I have also set up two iSCSI servers, one on each SP.  What I am not getting my head around is how the SP's Ethernet ports are addressed.

Each SP comes with two Ethernet ports, eth2 and eth3.  Did you set up SP1 to use eth2 and SP2 to use eth3 or did you use both Ethernet ports on both SP's?  On your ESXi hosts, do you have two vmkernel ports with one vmnic assigned to each?  If you could give me some more details on your setup, that would be greatly appreciated.

Best regards.

Well i’ve setup SPA’s iSCSI server to use both eth2 and eth3 with different ip-adresses so for example.

eth2 = 192.168.100.1 , eth3  = 192.168.200.1

and on SPB’s iSCSI server

eth2 = 192.168.100.2, eth3 = 192.168.200.2

then I group the eth2 interfaces on one switch and the eth3 on a different switch.

Then I setup two different vswitches with 1 vmkernel on both lets say

vmk1 = 192.168.100.10

vmk2 = 192.168.200.10

Then I add the vmk1 and 2 to the iscsi software hba if you are using that.

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --sessionfile=xxxx swiscsi nic add -n vmk1 -d vmhba39

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --sessionfile=xxxx swiscsi nic add -n vmk2 -d vmhba39

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --sessionfile=xxxx swiscsi nic list -d vmhba39

When you execute the list command you should see both adapters.

Then you add the iSCSI servers to the software dynamic discovery, add the lun to the esx,  rescan storage, change the path to round robin on the lun.

I’ve created two luns to be able to make use of both iSCSI servers. 

Regards

Tomas

Talarion,

Very helpful.  I'll give it a shot.  Thanks for your help on this.

Did you leave the round robin at the default 1000 IOPs or did you change it?

By the way, did you ever try NFS on the VNXe?  I had to give it up because the performance was awful.  I worked with EMC Tech Support for over a month and was not getting anywhere.  I switched to iSCSI and I am getting 4 times the performance versus NFS on my IOMeter performance tests in the real world tests.

Best Regards.

There are some good threads in the VMWare community, which have been using the templates provided here

http://vmktree.org/iometer/

Its a good standard templete and many people have posted their results.  So in that sense its  a nice report from the same benchmark tests.

For those of you that have the additional IO-Modules installed, there is a known performance issue when you are using those modules.  If you stick with the onboard NICs, you get pretty decent numbers, but when you switch to the IO-Modules or combine both, performance takes a dive.  There is an upcoming release that is supposed to address the issue.

Hi Scorch,

do you have any reference about that performance problem with additional io modules ? 

We are having same poor performance problem as others.

Any info about what's the time frame for the new release  ?

Thank you,

After working with EMC Tech Support for over two months, I just got an email from them that basically confirms that there is a problem with VMware running on a NFS datastore.  The Tech support guy sent me this:

"This is a known issue with degraded NFS write performance when using VMware ESX with NFS datastores. The issue is tracked under AR 404209 and the fix will be included in MR1. With the fix the NFS performance with ESX will be greatly improved.

The issue is that Dart serializes the NFS writes and sends the replies after the last write to storage. With the fix, the code was changed to no longer serialize incoming NFS write requests, which greatly improve write performance."

He goes on to further state that the next version will be available on August 7th.

I must admit, I am more that a little concerned that it took so long for them to find the problem and so long for them to get a fix out to us.

I'm being told MR1 will be released in August as well.  So happy to have this VNXe sitting in my rack waiting for the vendor to fix it!

hey there.

what do you mean by MR1?

is this a software update for ESX or for the VNXe series? cannot find anything about it on the internet.

It is the next release of VNXe software.

MR1 is the first Maintenance Release for the VNXe Operating Environment. It was actually released yesterday for upgrades to existing systems and is now available on the Downloads page. It is listed as "VNXe Operating Environment (v2.1.0.14097)".

After you upgrade, any newly created VMWare NFS Datastores will have this performance issue addressed (by changing the default for the caching property). For existing NFS Datastores, you will need to change a setting for that datastore. To do this, perform the following operation AFTER you upgrade to V2.1.0.14097:

In the GUI:

Navigate to the storage details page for that datastore (Storage->VMWare then details for the datastore). You should end up on the "General" tab. On the right hand side you will see a new box titled "Advanced Attributes" with the VMWare logo. Click on the "Show Advanced" link. Ensure that the "Caching Enabled" checkbox is NOT selected - i.e. disabled (it will be selected/enabled for existing NFS datastores). Click on the "Apply Changes" button on the lower left of the screen and you should see the performance issues addressed for that datastore.

Via the CLI:

Something along the lines of the following:

I applied the MR1 update that was recently released and so far it looks like the IO-module NICs now perform on-par with the onboard NICs.  This is good news.

Unfortunately, I'm seeing that the IO-module LAG is not being load-balanced.  Traffic flows to/from a single NIC and the 2nd NIC is barely used, if it's even used at all.  The onboard LAG seems to balanced the load evenly between the 2 NICs.  Is anyone else seeing this same behavior?

I'm not sure what I could have done that would have caused this to happen, but I'm sure glad I don't have this thing in production yet.

New Numbers

I have upgraded my software to the new version and I am please to report that the NFS performance problem that I first brought up a couple of months ago has been resolved, at least in my case.  Here are the results from two identical VMs running on the same host where one is hosted on iSCSI and the other is on NFS.  Again, this is without any link aggregation or changes to the MTU.  In my case, NFS seems to actually perform a little better that iSCSI which, from what I have read on the VMware best practices pages, should be expected.

Best Regards.

iSCSI

NFS