NetWorker: Troubleshooting SCSI Resets and Overwritten Labels on Tape Devices

Some events which may lead to SCSI resets being generated against tape devices:

• Zoning changes which allow unexpected hosts access to tape devices.
• Zoning changes which accidentally place multiple initiators in a single zone.
• Zoning changes which place disk and tape targets in the same zones.
• Zoning changes which have initiators zoned, even separately, to tape and disk targets.
• Zoning changes which have even separate initiators, if on the same HBA, zoned to tape and disk targets.
• Power events or malfunctioning SAN hardware.
• Installation or changes to any software on any zoned host which may perform any type of inquiry or tape access.
• Operating System tunable, driver or firmware changes for any zoned host.
• Data loss for virtual and physical tape devices

• Unmountable volumes
• Potential robotics issues
• Renamed devices for Plug-n-Play Operating Systems

SCSI protocol provides for an initiator to be able to issue a SCSI_RESET command, which may have any of various effects based on the class of device to which the reset is issued, and the state that device is in. For the purposes of this article, we are discussing resets issued to tape-class devices. For a tape-class device, a SCSI_RESET not only breaks reservations, it also causes the tape device to rewind.

Resets are usually the result of one of two conditions: malfunctioning hardware in the transport environment (rare), or a process attempting to communicate with a busy device. For the latter condition, the SCSI protocol demands a reset when the device does not reply to the process' request.

SCSI’s design assumes a simple, single-accessor environments where one host and one process handle all communications with a device. Resets can occur when multiple hosts or processes access SCSI devices because unrelated processes lack coordination outside multihost suites like NetWorker.

Resets are damaging in tape environments, since a reset causes a tape to rewind. Most software never expects a tape to rewind during use; drivers finish writing and leave the tape at End of Data (EOD) for the next session. Unexpected rewinds cause damage by forcing an intended EOD append to start writing instead from the tape’s physical beginning.

Resets are often difficult to detect. UNIX operating systems may contain some indication of a system reset and device logs; Windows generally does not. Similarly, NetWorker does not detect when a reset has occurred, and because a reset may rewind a tape, labels may be inadvertently, and silently, overwritten.

The following sequence of events shows the structure of a tape after NetWorker labels, and begins writing to, a virtual or physical tape. This shows two 32 KB label blocks, 256 KB data blocks, filemarks between sessions, and a double filemark marking the logical End of Data:

This representation shows the physical beginning of the media, with no representation of the physical end.

At this stage, NetWorker's nsrmmd process is waiting for more data, is actively connected to the driver, and in the UI is reflected as 'Writing, Idle'. If a SCSI reset is issued by some other process, the device silently rewinds. More data sessions begin with this same nsrmmd, who continues to write, but from Beginning Of Tape:

Now, the label is overwritten, and the data originally occupying the span of tape at the beginning. Even data not overwritten becomes inaccessible because the new double filemark/logical EOD blocks the drive from reading further.

There are a relatively large number of known causes for SCSI resets:

• Frequent Test Unit Ready (TUR) signals on Microsoft Operating Systems (Auto-Run)
• Other backup software configured to use the same tape driver.
• Monitoring software which may interrogate tape devices or perform full HBA device scans.
• Individual processes or scripts accessing tape drivers (mt, tar, some programs used by udev so forth).

SAN Zoning

• As noted previously, follow these zoning best practices for easier troubleshooting and prevention of resets:
• Only NetWorker hosts from the same Datazone configured to use the devices should be zoned at all (do not include other Datazones).
• Single-initiator zoning is required; 1:1 zoning is preferred; disk and tape may not share the same zone, and ideally, not the same initiator or even HBA.
• Ideally, tape and disk traffic should be separated at the switch level for the best performance and reliability.

Zoned Host Processes

• For each host, check for installed software, services, or scripts that may access tape drives in any way.
• No other backup software of any kind should co-habit with any NetWorker Storage Node or Server.
• No security or monitoring software should be running which attempts to communicate with tape devices.

Zoned Host Operating System Configuration

Windows

• Ensure StorPort driver is current.
• Disable Test Unit Ready for the tape driver: Microsoft Knowledge Base Archive 
• Reboots of a Windows host can issue resets to all zoned devices

Linux

• Do not use udev rules which use any utilities which interrogate the drives themselves (out-of-box udev rules are recommended).
• Enable CDI and the Simple Reservations feature of the device configurations.

Solaris

• Do not use CDI with ATape drivers; as CDI is expected and highly recommended, native drivers are recommended instead.

HP-UX

• Disable dm_stape module: Edit /var/stm/configuration/tools/monitor/dm_stape.cfg to use the value POLL_INTERVAL=0 and restart EMS.
• Ensure PHKL_40389 tape hotfix is installed.
• Ensure that OS-level reservations are reset by setting the kernel tunable st_san_safe=1.
• Run scsimgr set_attr -d estape -a norewind_close_disabled=1.
• Run scsimgr save_attr -d estape -a norewind_close_disabled=1.

AIX

• Ensure that reservations are turned off at OS level: /usr/sbin/chdev -l rmt<#> -a res_support=no.
• Disable ATape dynamic tracking feature.

NetWorker Configuration Work around If, after the above points have all been investigated and rectified, the cause cannot be identified and the issue persists:

1. Enable CDI on your tape devices (SCSI Commands) and Persistent Reservations on all tape devices in your Datazone.
2. Enable SCSI reservations (ideally Persistent/SCSI-3) on your filer Operating Systems as well (contact the vendor for details).

This workaround is not ideal and normally should not be needed, but it may offer partial relief when the root cause cannot be identified.

NetWorker Forensics

• Affected tapes fail to mount, and be marked as 'unlabeled' by NetWorker (though the media database entries remain intact, but incorrect).
• Volumes may appear to hold an inordinately large amount of data (due to multiple undetected rewinds).
• Backups failing with the error that tapes are being 'marked full prematurely' may indicate midwrite SCSI resets.
• The first 20 lines of NetWorker are the clearest proof that SCSI resets are occurring:
  • Unexpected file number, wanted 2, got34 (actual number may vary).
• You may be able to determine when the reset occurred by comparing the got number in the error:
  • Save sets shown by mminfo with a lowest mediafile value of that number, minus 3, are those which finished writing before the reset occurred, and whose completion led to the Writing, Idle period when the reset took place -  the sscomp(22) mminfo value is before the reset.
  • Save sets shown by mminfo with a lowest mediafile value of that number, minus 2, are those which began writing after the reset occurred, and who started during the Writing, Idle period (after the reset took place) -  the ssccreate(22) mminfo value is after the reset.
• The host performing the write at the time is not necessarily the Reset initiator; it may be a separate host with different software which is zoned to the device, local software interfering, or other driver- or SAN-level issues.

This article is one of a series in Troubleshooting Media Problems with NetWorker. The list is here:
NetWorker: Troubleshooting Tape Libraries home page