Unsolved
This post is more than 5 years old
5 Practitioner
•
274.2K Posts
0
9756
Path Selection Policy "rr" and "iops"
Hello,
Where did the difference between "rr" and "iops" as the policy come from?
Our customer got two type of policy as a command result of "esxcli storage nmp device list" after he set his IO Operation limits by the following two ways following "Using EMC VMAX Storage in VMware vSphere Environments" page 84.
It turn out the output both way show different result (highlighted in red).
1. Set by
esxcli storage nmp satp rule add -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1"
And got from "esxcli storage nmp device list"
Path Selection Policy Device Config: { policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
2. Set by
esxcli storage nmp psp roundrobin deviceconfig set --device= --iops=1 --type iops
And got from "esxcli storage nmp device list"
Path Selection Policy Device Config: { policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
I found the similor question "XIO ESX 5.5 setting" in a private space so that it hasn't been answered yet.
Best regards,
Hiroki
WorldPolice
123 Posts
0
October 5th, 2015 06:00
So this is simply because the first statement is changing the default rule (for type rr) for all new devices to use 1 iops instead of 1000 for the SATP. The second statement is changing the behavior of an individual device, and that is done by iops or size (bytes). There is no 'rr' type. You'll notice that the columns you are not showing, stay the same on both statements for the device:
Storage Array Type: VMW_SATP_SYMM
Storage Array Type Device Config: {action_OnRetryErrors=off}
Path Selection Policy: VMW_PSP_RR
So, not speaking for VMware of course, my assumption is the policy name differs so that the consumer knows when it is a rule at the system level, or was changed at the device level. The behavior is the same, however.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
October 5th, 2015 18:00
Hello Drew,
So array level settings are displayed by "esxcli storage nmp device list" if only SATP setting has been done and no PSP setting hasn't been done yet? And the device level settings are displayed by the same list command "esxcli storage nmp device list" if the device level settings (PSP) for the devices ("IO Operation limit") has been changed for the specific devices once?
I'm sorry but I can't find the document for the detail information for the command "esxcli storage nmp device list".
Thank you for answering.
Best regards,
Hiroki
DCsafe
8 Posts
0
October 6th, 2015 03:00
Hi Hiroki,
Found some information on things you are looking for. See if that helps
* Storage Array Type Plug-Ins (SATPs) run in conjunction with the VMware NMP and are responsible for array-specific operations.
https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-DB5BC795-E4D9-4350-9C5D-12BB3C0E6D99.html
* To check if SATP for particular storage array is loaded or not
~ # esxcli storage nmp satp list
eg output
Name Default PSP Description
------------------- ------------- ------------------------------------------
VMW_SATP_ALUA VMW_PSP_MRU Placeholder (plugin not loaded)
MW_SATP_ALUA_CX VMW_PSP_FIXED Placeholder (plugin not loaded)
VMW_SATP_SYMM VMW_PSP_FIXED Placeholder (plugin not loaded)
VMW_SATP_CX VMW_PSP_MRU Placeholder (plugin not loaded)
* Please go through
http://buildvirtual.net/configuring-multipathing-policies-on-esxi-5/ for more information on PSP, SATP, IOPs
WorldPolice
123 Posts
2
October 6th, 2015 07:00
I don't want to confuse things more so let me see if I can expand in a simpler way. VMAX has an SATP plug-in which is part of the ESXi code. This plug-in ensures that all VMAX devices presented to an ESXi host will be assigned a path selection policy of RoundRobin when using NMP and a default IOPS=1000. When you run this command "esxcli storage nmp device list" it is going to tell you about all the devices on the box. Here is one of the devices on my VMAX/ESXi 6 environment:
naa.60000970000197200056533030303036
Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)
Storage Array Type: VMW_SATP_SYMM
Storage Array Type Device Config: {action_OnRetryErrors=off}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5
Is USB: false
That device was recognized as a VMAX device and was assigned the default rule associated with a VMAX device which says use PSP RoundRobin and IOPS=1000. Now if I execute #1 in your question to change the rule and then reboot my box my device will look like this:
naa.60000970000197200056533030303036
Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)
Storage Array Type: VMW_SATP_SYMM
Storage Array Type Device Config: {action_OnRetryErrors=off}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5
Is USB: false
Now if I don't want to reboot my box, I can execute #2 from your question "esxcli storage nmp psp roundrobin deviceconfig set --device= --iops=1 --type iops". After I run this command I now issue that same command which lists all devices "esxcli storage nmp device list". Here is what my device looks like now:
naa.60000970000197200056533030303036
Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)
Storage Array Type: VMW_SATP_SYMM
Storage Array Type Device Config: {action_OnRetryErrors=off}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5
Is USB: false
You can see my iops are now "1" and VMware has changed the policy to "iops", letting me know that this device has been altered from the rule for VMAX devices. By default, VMware uses iops to decide when to change paths. Note that you can also use 'bytes' which is why that is also shown above. Here is what my device looks like if I alter the policy to change paths every 5000000 bytes:
naa.60000970000197200056533030303036
Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)
Storage Array Type: VMW_SATP_SYMM
Storage Array Type Device Config: {action_OnRetryErrors=off}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=bytes,iops=1,bytes=5000000,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5
Is USB: false
So what all this means is that the command "esxcli storage nmp device list" will simply tell you about the current policies in place for NMP for a particular device. If you see policy=iops or policy=bytes you know that individual device's configuration was changed. If it says policy=rr for the VMAX device it means it was assigned by rule. As my TechBook notes on page 84, however, the VMware default for iops is 1000 so you can either change the rule (#1 in your question) and reboot, or individually change each device (#2 in your question).
I hope this explanation was helpful.
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
October 7th, 2015 02:00
Hello DCsafe,
I'll look at them.
Thank you for the information.
Best Regards,
Hiroki
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
October 7th, 2015 02:00
Hello Drew,
The actual path policy, which is applied to the device EMC Fibre Channel Disk (naa.60000970000197200056533030303036), hasn't changed after you issued "esxcli storage nmp psp roundrobin deviceconfig set --device= --iops=1 --type iops"? And just the result of the command has chnaged? Otherwise the actual path policy has also changed?
So the difference of the result of the list command just comes from if the command #2 (PSP) has been issued to specific devices?
I konw the command #1 (SATP) requires rebooting where the command #2 (PSP) doesn't reqire it.
Thank you for replying.
Best regards,
Hiroki
WorldPolice
123 Posts
0
October 7th, 2015 05:00
Yes if I understand your interpretation that is correct. The only difference in the list command is that the first device list was set by the rule, the second by me issuing the iops type. In the end, though, the path handling is the same for:
{policy=rr,iops=1
{policy=iops,iops=1,
Anonymous
5 Practitioner
5 Practitioner
•
274.2K Posts
0
October 8th, 2015 02:00
Hello Drew,
I got that the first list shows it was set by the policy rule by SATP, and the second list shows it was set by device by iops. And the both have been set to policy=Round Robin and iops=1.
I really appreciate.
Best regards,
Hiroki