Contaminants such as dust on fiber optic connector end face causes poor IO performance

• Why cleaning fiber optic cables should not be overlooked
• Considerations for on-site fiber optic cable inspection
• Diagnostic information sources
• How to clean the fiber optic cable

 Why is cleaning cables important for system IO performance?

• Provides customer choice that:
  • Saves time, through multi-paths, by facilitating cable cleaning by the customer at any time
  • Ensures redundancy by recovering a "down path" through the cleaning process
  • Accelerates recovery by reducing resolution time greatly through the cleaning process
  • Simplifies problem resolution by eliminating one of the probable causes of the failure if cleaning fails
• Identifies probable cause
  • In a previous and extensive test study by Dell, 83% of optics (over 500 examined) returned to Dell after replacement were determined to not have any fault.  Also, in a previous study by NTT Advanced Technology, 98% of installers (blue) and 80% of network owners (red) reported that issues with connector contamination was the greatest cause of network failure.
    
     

On-site fiber optic cable troubleshooting

• External damage - The symptoms of external damage are more often than not obvious as there is complete loss of transmission. As fiber cable is often routed underfloor it can be subject to vermin chewing cables or when linking buildings or different sites external works may dig up and break the cable. Less obvious causes of damage may be caused by poor organisation of cables in the rack causing the cable to become crimped between rack doors or other equipment. The cable may also be stretched while pulling it through ducts or if put under tension when negotiating tight spaces. In order to detect damage over long cable runs the administrator or cable maintainer would need to use an optical time-domain reflectometer (OTDR) to help find the location of the damage.

• No visible damage to cables or connectors - When troubleshooting in the datacenter we are more often handling fiber optic patch cables and can use visual cues generated by the LED status lights on the Host Bus Adapter (HBA) or fabric switches to observe if the link is being negotiated and what link speed has been negotiated. The switch operating system or HBA management utility will often report port statistics for connections like cyclic redundancy checksum (CRC) errors, discards or drops. If these values are high then it is a strong indicator of one of the following conditions:

1.	Poor mating contact between the transceiver module and fiber optic switch or fiber optic cable connector and transceiver module. Performance problems can occur in the event that the connectors of the fiber optic cable are not fitted properly. Check each of the connectors to determine whether this is the case and rectify it by inserting them properly. If the problem continues to occur, replace the cable with a version that makes use of spring loaded connectors that fit securely without slipping out of place.
2.	Damaged fiber optic cable so the glass is broken internally or external force being applied to cause microbending. A thorough visual inspection of the fiber optic cable will be sufficient to highlight any prominent defects that have affected the casing or the conductors. It doesn’t take a significant amount of damage to affect the ability of the cable to work properly. It is prudent to replace the cable if any issues become apparent from the inspection, such as splits or scratches.
3.	Electrical fault with the transceiver module at one or both ends. This could also be caused by incorrect specification components (e.g. plugging a 8Gb module in a 32Gb switch). Some switch manufacturers are very specific about which transceiver models they will officially support.
4	Dust or chemical contamination on the ends of the fiber optic cable connectors or transceivers module that impedes the passing of light. Cable connectors should be cleaned and when stored must be protected from dust particle or chemical contamination.
5	Stretching. The components within a fiber optic cable are delicate and can be damaged if stretched across a distance that it cannot comfortably accommodate. Avoid the risk of the fibers within the cable becoming damaged by moving the items closer together or obtaining a cable of a more appropriate length. Stretching can also occur from pulling at the jacket of the cable rather than using the grips at the connectors designed to be used to fit and remove them. Always use the grips to remove the connector from the socket that holds it.

Diagnostic information sources

Fiber optic end face as seen through fiberscope showing the difference between a clean end face and end face with dust.
Dust, even oil contamination originating from human fingers on fiber optic LC connectors impedes signaling leading to any or combination of the following symptoms:   Lower than expected link speed negotiation Frequent host IO aborts and/or connection retries Higher optical loss (dB). Degraded and poor IO performance sometimes with high latency across affected network paths. Frequent path changes on host multipath connections. Port metrics such as Link failures, Invalid CRC Count and Invalid Transmission Word Count
	SFP transceiver with dust cap protector.

If your environment uses fiber channel switches,  switch port metrics can be used to identify suspect transceivers or optic cables. 

• Connectrix: How to troubleshoot Fibre Channel node to switch port or SFP communication problems by elimination, Self-Help.
  
  
• Connectrix: How to Interpret SFP Transceiver TX and RX Power Levels as Part of Troubleshooting the SFP Transceiver

How to clean the fiber optic cable

Dell recommends a wet dry cleaning process to clean the connectors. You will need 3 items:

1.	A clean lint free wipe
2.	Fiber cable cleaner fluid.
3.	A dry fiber cable cleaner.

Before cleaning fiber optic connectors, make sure the cable is disconnected from both ends and turn off any laser sources. Always wear safety glasses and inspect the connectors before you clean them. Before a connection is made, the connectors have to be inspected and cleaned. A connector housing should be used for plugging or unplugging a fiber. A protective cap should be placed on unplugged fiber connectors. The unused caps can be kept in a sealed container.

When cleaning fiber optics, the end face of the connector should never be touched and also the clean area of a tissue or swab should not be touched or reused.

When using alcohol, the portion of a tissue or swab where it has been applied or the dispensing tip of an alcohol bottle should never be touched. Alcohol is flammable, so it should not be used around an open spark. Also, never leave alcohol residue on the end face because this could damage the equipment.

If you have access to specialized equipment, the fiber end should be inspected with a fiberscope of at least 200x magnification, and if it is contaminated, it should be cleaned with the dry cleaning method.

If you have never done this before watch the video for instructions.

The fiber should not be used if the cleaning is unsuccessful because the contamination could be due to scratching, improper polishing, or some other damage.

NTT-ATʼs CLETOP fiber optic reel-type cleaner provides thorough cleaning of the ferrule end face by using a dry alcohol-free cloth.  Operators can consistently achieve high quality cleaning without alcohol or other solvents. The CLETOP has anti-static properties to avoid reattachment of dust to the end face after cleaning.   Fiber optic cleaning kits are available from many online retailers.   

For Storage Arrays

• Storage Center Alert: How to Troubleshoot a Storage Center's Fibre Channel Path Down to a Server Error
  
  
• Deployment KB: Dell EMC Unity: How to Troubleshoot Fiber Channel Port connectivity issue in Unity.
  
  
• Deployment KB: Dell EMC Unity: How to identify the supported Fiber Cables
  
  
• Solutions Enabler: How to check light levels of SFPs associated to the FA ports in a PowerMax array