Large amount of switch port RX errors

Question

Hi,

We have Dell N1524P switches for our workstations. upstream from the N1524P switch is a Dell X4012 switch. We have SFP+ 10Gbit DAC cables between them. The N1524P switches are reporting large amounts of interface errors on the uplink port. I tried setting up wireshark and a port mirror, but I don't think the switches mirror over the bad frames. I see TCP retransmissions in the capture, but I don't think that is the same as link level errors. We have multiple workstation switches, talking to different upstream switches on different DAC cables and they all seem to have the same issue so I don't think it's a port or cable issue.

Anyone have suggestions on how to can troubleshoot this or capture the bad frames for inspection? It would be nice to see what the frame is and "why" the switch marked the frame in error so I can find the root of the problem. Ideas?

Port: Te1/0/2
Dropped: 0
Octets: 1472573779 Packets: 628363355
Broadcast: 41020689 Multicast: 30777826
CRC Align Errors: 14100319 Collisions: 0
Undersize Pkts: 0 Oversize Pkts: 3523274
Fragments: 127130 Jabbers: 2847846
64 Octets: 74544748 65 - 127 Octets: 75970468
128 - 255 Octets: 226944349 256 - 511 Octets: 111044942
512 - 1023 Octets: 6620135 1024 - 1518 Octets: 129606959
HC Overflow Pkts: 0 HC Pkts: 628363355
HC Overflow Octets: 74 HC Octets: 319300153683
HC Overflow Pkts 64 Octets: 0 HC Pkts 64 Octets: 74544748
HC Overflow Pkts 65 - 127 Octets: 0 HC Pkts 65 - 127 Octets: 75970468
HC Overflow Pkts 128 - 255 Octets: 0 HC Pkts 128 - 255 Octets: 226944349
HC Overflow Pkts 256 - 511 Octets: 0 HC Pkts 256 - 511 Octets: 111044942
HC Overflow Pkts 512 - 1023 Octets: 0 HC Pkts 512 - 1023 Octets: 6620135
HC Overflow Pkts 1024 - 1518 Octets: 0 HC Pkts 1024 - 1518 Octets: 129606959

dfg-2014 · Accepted Answer

Ok. I swapped the 5meter DAC cables to 10Gbit fiber cables instead. Now there are no issues. No CRC alignment errors. Everything is so much faster. From what I gather then 5meter length DAC does not work or at least it doesn't work with Dell equipment. Hope this helps someone else having similar issues.

DELL-Josh Cr · Answer

Hi, Is the firmware up to date on all of the switches? Is jumbo frames enabled? Are the ports set to auto for the speed or is it hard set? Is everything set at full duplex?

dfg-2014 · Answer

Thanks for the reply!

The firmware isn't too old, but I will be updating the firmware tonight for all switches to the latest. Jumbo frames are enabled and operational on the upstream switch. This switch doesn't have an option for Jumbo frames, but I did change the MTU to the max (9216). I haven't rebooted since the MTU config update, but it will get rebooted tonight with the firmware update.

On both switches (both sides of the DAC cable), port is set to 10Gbit, no auto-negotiate, full duplex, no flow control.

Any other suggestions are welcome. I'll update this thread after the firmware updates. Is there any way to mirror bad frames so I can see what the switch is having issues with? When I did a port mirror before I didn't see those frames forwarded.

DELL-Josh Cr · Answer

I am not aware of a way to mirror bad packets unless you could have them on the port that was directly connected to a client running wireshark.

dfg-2014 · Answer

Ok. All of the switches have their firmware updated. Still seeing the link errors. I did a little more investigating and it looks like the workstation switch I'm seeing most of the issues on is getting a STP topology update every 2 seconds. I confirmed this with wireshark. I'm seeing a topology change stp packet coming from the upstream connection every 2 seconds. I'm guessing that could be causing some issues if the switch thinks it has to re-adjust for a topology change that often.

I port mirrored the upstream switches and couldn't find where it was coming from. The wireshark stp packet has a source mac on it. The source mac doesn't match any of the switches, but it looks very close to the mac format of the x4000 series switches we have which are our core switches. I checked all the ports on our core switches and wireshark wasn't seeing the stp packet being received from any ports that were non-switch devices. It seems like the stp message has to be coming from a core switch, but I don't see why it is doing that.

Is there any way on the x4000 series switches to inspect verbose logs to see if it is generating the topology change messages?

dfg-2014 · Answer

I'm starting to think all the issues stem from the physical DAC cables. They are dell compatible, but I think there are issues over certain lengths. We had 7m DAC cables that would not even get a link on the ports that we exchanged for fiber. These problematic links are 5m DAC cables. I installed the exchanged 7m uplink fiber yesterday as well. I'm not seeing the same CRC / Align Errors in the stats for those uplinks. We are going to exchange the 5m DAC for 5m fiber to see if that is the fix. FYI, for those troubleshooting this stuff in the future. From what I've experienced with DAC it seems to work fine at 3m lengths and lower.

DELL-Josh Cr · Answer

That sounds reasonable, let me know if you need anything else after swapping the cables.

Networking General

Large amount of switch port RX errors

Was this post helpful?