Turn on disabled CX600 port

Hi Danny!

When I worked for U of M College of Engineering, I think a few of us came up and visited.  Hope all is well in Houghton!

Can you run "naviseccli -h 172.20.189.200 port -list -sfpstate" and "naviseccli -h 172.20.189.200 port -list -sp -all"?  This adds a bit of extra output, which might help determine if you have a bad SFP.

Let us know if that helps!

Karl

Yes, all is mostly well in Houghton!

Here is the output of both of the commands that you suggested. My system does not seem to support the "-sfpstate" parameter:

[sat3200student@cnsa-stor1 ~]$ naviseccli -h 172.20.189.200 port -list -sp -all

Total number of initiators: 11

SP Name:             SP A

SP Port ID:          1

Registered Initiators:        3

Logged-In Initiators:         1

Not Logged-In Initiators:     2

SP Name:             SP A

SP Port ID:          0

Registered Initiators:        2

Logged-In Initiators:         1

Not Logged-In Initiators:     1

SP Name:             SP A

SP Port ID:          3

Registered Initiators:        0

Logged-In Initiators:         0

Not Logged-In Initiators:     0

SP Name:             SP A

SP Port ID:          2

Registered Initiators:        0

Logged-In Initiators:         0

Not Logged-In Initiators:     0

SP Name:             SP B

SP Port ID:          1

Registered Initiators:        3

Logged-In Initiators:         1

Not Logged-In Initiators:     2

SP Name:             SP B

SP Port ID:          0

Registered Initiators:        1

Logged-In Initiators:         0

Not Logged-In Initiators:     1

SP Name:             SP B

SP Port ID:          3

Registered Initiators:        2

Logged-In Initiators:         0

Not Logged-In Initiators:     2

SP Name:             SP B

SP Port ID:          2

Registered Initiators:        0

Logged-In Initiators:         0

Not Logged-In Initiators:     0

Information about each SPPORT:

SP Name:             SP A

SP Port ID:          1

SP UID:              50:06:01:60:80:60:03:68:50:06:01:61:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:60:EE:F0:20:19:08:00:88:60:EE:F0

SP Source ID:        6363411

MAC Address:         Not Applicable

SP Name:             SP A

SP Port ID:          0

SP UID:              50:06:01:60:80:60:03:68:50:06:01:60:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:02:09:7A:20:0A:08:00:88:02:09:7A

SP Source ID:        6359571

MAC Address:         Not Applicable

SP Name:             SP A

SP Port ID:          3

SP UID:              50:06:01:60:80:60:03:68:50:06:01:63:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP A

SP Port ID:          2

SP UID:              50:06:01:60:80:60:03:68:50:06:01:62:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          1

SP UID:              50:06:01:60:80:60:03:68:50:06:01:69:00:60:03:68

Link Status:         Up

Port Status:         Online

Switch Present:      YES

Switch UID:          10:00:08:00:88:02:09:7A:20:10:08:00:88:02:09:7A

SP Source ID:        6361107

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          0

SP UID:              50:06:01:60:80:60:03:68:50:06:01:68:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          3

SP UID:              50:06:01:60:80:60:03:68:50:06:01:6B:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

SP Name:             SP B

SP Port ID:          2

SP UID:              50:06:01:60:80:60:03:68:50:06:01:6A:00:60:03:68

Link Status:         Down

Port Status:         DISABLED

Switch Present:      NO

MAC Address:         Not Applicable

[sat3200student@cnsa-stor1 ~]$ naviseccli -h 172.20.189.200 port -list -sfpstate

usage: port -list <-hba|-arrayhba|-sp|-gname groupname|-uid UID> <-all>

                  <-arraycommpath> <-failovermode> <-unitserialnumber>

                  <-mac> <-initiatorcount>

            -removeHBA <-o> -hbauid HBAUID|-host host|-hostip ip|-all

            -diagnose <-o> -sancopy <-clean>|-host

Invalid command line parameters: -sfpstate

[sat3200student@cnsa-stor1 ~]$

I did think that perhaps an sfp has failed, but if that is the case then I have 3 of them failed on the b channel because I have all four of those ports plugged into a switch and they all show disabled states even though all of them are lighting the green connection light on the CX-600 rear panel. I grabbed a pair of long nosed pliers and tried to pull(gently!) one of them out but no luck. Are those field replaceable? I am only using two of them on the A channel so I could swap one of them to B3 if that is possible.

Danny

On the CX600/700 the SFP on the port can not be changed, requires a SP replacement.

You know that SPB port 1 is good, so try that cable in port 3 (the cable from switch port for SPB1 to SPB3). If it still doesn't come up, try rebooting SPB. I think I remember something about the ports locking on the CX600 and needing to reboot SP.

glen

I switched the cables that were connected to ports B1 and B3. When I did that, the switch port connected to B1 came up but the one connected to B3 did not. Port B3 was still disabled. I rebooted channel b, port B3 still disabled. I powered down the entire array(this is an educational machine and we are not yet using it for lab experiments) and brought it back up, no change, B3 still disabled, B1 enabled.

So I guess that the port B3 sfp is shot. Ports B0 and B2 are also disabled, and won't light a switch port connected to them. So I guess that B0, B2 and B3 must all have defective sfp's. B0, B2 and B3 all show the green link light on the CX600 when I plug in a cable, but none of them will come out of the disabled state. It seems strange that all three of them act the same, but this is an older unit so perhaps all three have failed transmitter LED's in the SFP.

I do have a "spare" CX600 so the next step is to do some testing on my "spare" channel b unit to see if it has usable ports.

Thanks for all of the information, very useful.

Danny Miller

Did port B3 work in the past? Is there a zone on the switch for B3? I sort of remember that for the port to come on-line, you need both a cable plugged in the SP port that is connected to a switch port and a zone for that port on the switch.

glen

This system worked for over 4 years in its present configuration. I have dual fabrics(each fabric is one switch) for redundancy, the fabrics are not connected together. The port B3 went to the disabled state sometime in November 2014. Since we have dual fabrics, all of the storage servers still have connectivity via the other fabric so I finished out the fall semester using that.

The switches are zoned using the "single initiator zoning" plan. Each storage server HBA is zoned to one port on the CX600 and one port on the CX700(we have two storage processors in our SAN). I checked all of the zoning, and especially the zone that contains B3. It looks fine to me. I didn't do anything with the zoning last fall around the time that port B3 went disabled.

Their are four zones in our zone set for this fabric, one for each storage server. Each zone contains the WWN of that storage server HBA, one CX600 port, and one CX700 port. For the zone that contains B3, I just reactivated the zone, in case something had somehow changed. It didn't help.

As a test, I just connected port A2 to a switch port using a new cable. I had not been using A2, but when I connected it to the same switch the switch port immediately came up, and when i checked the port status for A2 is is now "up" and enabled. It was 'down" and "disabled" before the connection. It did that with no rebooting of the SP and no zoning of the switch, so I don't think that the zoning is needed to enable a port.

Hmm, it still seems strange that I have ports B0, B2 and B3 all connected to switch ports but none of them will bring up the switch ports and all sit there "disabled". The link lights on the CX600 for B0, B2 and B3 are all green but the switch lights on the other ends of those cables are all dark. How can all three ports have dead hba's?

It's odd that three of the four ports failed, but that's what it sounds like. You mentioned that you had a spare CX600 - the SP's are identical for SPA and SPB, so you could use either one to replace the SPB SP. Replacing the SP on the CX600 is pretty easy. First, disable System Write cache before you remove the SP - this will dump all the Write cache to the disks. Then you can pull SPB out, it's hot swapable. Disconnect the cables and reconnect to the new SP and then re-insert, it should start booting at the point.

If you want to feel safer, then power down the array first - use the power switch on the SPS backup power supplies, then should take about 2-3 minutes to power down - disable Write cache first just to be safe.

glen

I did some testing on my "spare" cx-600 yesterday. The A channel unit is not working, but the B channel unit works with all four front end ports usable(they all light up ports when connected to a switch).

So this morning I powered down my CX-600 and replaced the B channel SP with my B channel spare unit. It works, all four front end ports(B0 through B3) are now functional. The old unit was very dusty, dust was heavy on the heat sinks of the processors and on the heat sink of the large IC's mounted on the motherboard. I remember removing a lot of dust from the back and front of the SP enclosure when I received this unit 6 or 7 years ago, but I didn't think to open up the SP and remove the dust that was certainly limiting airflow to the internal heat sinks. I intend to power this unit back down, remove the A SP and clean out the dust.

Thanks much to all of you for your help. Glen, your detailed instructions were very useful.

Danny Miller