PowerScale: Isilon: OneFS - How to Collect PCAPs for NDMP Traffic

Part I - Gather NDMP and Network Information
1. Record the node the NDMP session is running on and the DMA's IP address. In this example, the NDMP job is running on node 10 with session ID 68221. The DMA's IP is x.x.188.14

cluster-1# isi ndmp sessions list
Session Data Mover OP Elapsed Time Bytes Moved Throughput
----------------------------------------------------------------------
10.68221 ACTIVE IDLE B(tS10+) 3h 32m 1.0775TB 162.09MB/s
----------------------------------------------------------------------
Total: 1
cluster-1# isi ndmp sessions list --verbose
Session: 10.68221 <--- Node 10, session ID 68221
Client: x.x.188.14 <--- DMA IP address 
Remote DS: Local
Start Time: Tue Jun 30 06:46:10 2020
Elapsed Time: 3h 32m
Operation: B(S10+)
Source Path: /ifs/data
Destination Path:
Data State: ACTIVE
Mover State: IDLE
Bytes Moved: 1.0792TB
Throughput: 161.50MB/s
Tape:
Media Changer:
cluster-1#

Part II - Capture Packets and Upload Files to Isilon Support 
1. Create a directory to store the packet capture in.  

# mkdir -p /ifs/data/Isilon_Support/$(date +%F)

2. Start the packet capture. The values are defined as follows:

- <DMA_IP>. The IP address of the DMA (Part I step 1).


> All interfaces on a node (node number from Part 1 Step 1).

# for iface in `isi_for_array -n <node_number> ifconfig | grep -B2 ether | grep flags | egrep -v "laggport|lo0|ISIINTERNAL" | cut -d: -f1` ; do isi_for_array -n <node_number> tcpdump -i $iface -s 600 -C 200 -W 4 -w /ifs/data/Isilon_Support/$(date +%F)/`hostname`.${iface}_$(date +%F_%H%M%S).pcap host <DMA_IP> &>> /ifs/data/Isilon_Support/$(date +%F)/$(uname -n).$(date +%F_%H%M%S).$iface.log & ; done

3. Once the event has been re-produced, kill the tcpdump process cluster-wide, and validate there aren't any stale PIDs: 

# isi_for_array -X 'killall -SIGINT tcpdump'
# isi_for_array ps -auxx | grep tcpdump 

4. Upload full logs and PCAPs to Isilon Support. 

# isi_gather_info -f /ifs/data/Isilon_Support/$(date +%F)