Unsolved
This post is more than 5 years old
1 Message
0
2697
Best Practice SAN connectivity
Hi, I have been in a debait with another engineer on how we should connect our SAN fabric to our servers and CX4 series SAN array, via dual Cisco Fabric switches. i am looking for some input , to ether validate my argument or see if he is correct :-
we have currently :-
- All SP-A connections going to First switch which is Fabric A
- All SP-B connections going to second switch which is Fabric B
- Servers are Dual HBA cards. one connection from Fabric A and one from Fabric B
So if one switch Fails, the servers loose connectivity to the SP-B controller, and all the disks trespass.
he keeps mentioning that EMC says your not supposed to mix SPA and SPB connections in one fabric.
and that when disks trepass thats a normall function and quiet exceptable for reduncancy.
Ok yes this does work.
My understanding is that you can mix Both SP-A and SP-B connection on one Fabric.
So each HBA connection on the server can see paths to both SP-A and SP-B
- So SP-A1 connects to First Switch Fabric A
- SP-A2 connects to Second Switch Fabric B
- SP-B1 connects to First Swithc Fabric A
- SP-B2 connects to Second Switch Fabric B
and so on.
Hence if a switch fails, the server can still see connections to both A and B controllers.
Plus, if some servers only have a single HBA they can still see both controllers.
Thoughts please !!
dynamox
2 Intern
2 Intern
•
20.4K Posts
1
November 21st, 2012 05:00
your thought process is absolutly right, you want to have SPA and SPB ports on each fabric so that each HBA is zoned to both, IE:
Fabric A Zones:
HBA1 - SPA0
HBA1 - SPB3
Fabric B Zones:
HBA2 - SPB0
HBA2 - SPA3
not only it does it not need to trespass LUNs in case of fabric failure, you get better load-balancing as each HBA will be used for traffic to both fabrics.
Sridhar246
115 Posts
0
November 21st, 2012 07:00
You are correct sir, zone ports on SPA and SPB to both the switches. Tresspass is acceptable (even though I don't like tresspassing luns during production hours) but lets say for some reason (with the config you have) one of the switches and all your luns present on SPA tresspass to SPB. Now lets say before SPA and SPB were to be running at close to 50% utilization after tresspass everything is on SPB with close to 100% SP utilization adversly affecting performance.
RRR
1 Rookie
1 Rookie
•
5.7K Posts
0
November 21st, 2012 07:00
Listen to Dynamox, he is right! Connect SPA and SPB to both fabrics. Also consider that when you have mirrorview using SPA0 and SPB0 for replication that you don’t want these 2 ports on the same fabric, so when somehow 1 fabrics fails the other one still offers mirrorview connectivity.
So SPA0 / SPB1 on one fabric and SPA1 / SPB0 on the other is good.
Sridhar246
115 Posts
0
November 21st, 2012 08:00
Thing I wanted to say is with the config he has in case of one switch failure, all the luns owned by SPA (for example) will be tresspassed to SPB which will shoot up the SP utilization of SPB which might affect performance. Please correct me if I am wrong.
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
November 21st, 2012 08:00
CPU utilization has no relevance on how port connectivity is established.
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
November 21st, 2012 10:00
looking at the current config i agree, now with ALUA that could be masked because you could still talk to SPA through the cmi bus.
AndyHengst
3 Posts
0
December 6th, 2012 14:00
An obvious question would be "Where does it say that?" (keeping SPA on fabric A etc) If that's documented I'd like to know!
We also zone each SP to both fabrics. To put it another way, I'd rather have powerpath look after the LUN visibility for me than have a trespass do that. There might be another reason: if a switch on fabric A fails, a LUN on SPA *may not* trespass. There is no problem with SPA and I"m not trespassing the LUN, so why would it trespass to SPB? This is something you should test to be sure. I believe you'd simply lose connectivity.
A bit off topic, but related to zoning two fabrics: we made our life a bit easier (and less error prone) by using the same zone set for both fabrics. Even though half the zones are not needed in one fabric, and the other half are not needed in the other fabric, you maintain only one zone set and you CANNOT accidently activate the wrong zone set to a fabric.
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
December 6th, 2012 18:00
what do you mean you have one zoneset for both fabrics, can you elaborate ?
Thanks
AndyHengst
3 Posts
0
December 7th, 2012 08:00
Hi dynamox,
we use SMC with one Zone Library and one Zone Set with sufficient zones to allow all required connections to work. We activate this zone set to Fabric A, and a few minutes later activate the same set to Fabric B. Each fabric will have exactly half unknown/unused WWN zones, but that's OK.
Here are some typical zone names:
blackbox_A_CX1234_SPA0
blackbox_A_CX1234_SPB1
blackbox_B_CX1234_SPA1
blackbox_B_CX1234_SPB0
... all of them go to all switches.
(edit:) But if your environment is nearing the maximum capacity for zones then it might be time to separate these out.
dynamox
2 Intern
2 Intern
•
20.4K Posts
0
December 7th, 2012 09:00
how big is your environment, i know this may work for you but this has got to be a nightmare when you need to troubleshoot connectivity issues.
AndyHengst
3 Posts
0
December 7th, 2012 10:00
The active zone set has just over 700 zones and over 1500 members (zone set properties). This aspect of our environment has never caused problems. If anything, something that shouldn't work (HBA and CX port both plugged into wrong fabric, at the same time) will work. We also know that if all connectivity is lost on one fabric, it's not because of a mistake selecting the wrong zone set and activating that (maybe I'm using SMC-specific terms here).