isi statistics command document

Hi k-ryu,

There is not much in the way of complete documentation. There are a couple of articles out there to assist but none that speak specifically to the command that you are using:

382740 : OneFS:Using the isi statistics command https://support.emc.com/kb/382740

459545 : How to Understand Isi Statistics Drive Output https://support.emc.com/kb/459545

459506 : How to Read and Understand the Output of Isi Statistics Protocol Command https://support.emc.com/kb/459506

The throughputs are gathered from three distinct areas of the system,

and the example helps illustrating how Isilon OneFS works overall:

1. Protocols:

SMB,  FTP,  HTTP (also RAN=Restful Access to Namespace), ISCSI  (deprectaed), NFS,  HDFS (Hadoop)

Each colum is the combined read + write throughput.

"Total" is the computed sum of all protocols.

2. Network:

Only the external = client-facing network throughputs are shown.

NetIn = write to Isilon cluster, NetOut = read from cluster.

The sum NetIn+NetOut "could" match the protocols Total, but doesn't always (as in your example):

- network throughput is sampled differently -- things get clearer with more stable client workloads

- network throughput will also include other activities like SyncIQ or NDMP backups and other "noise"

3. Disk media:

DiskIn = write, DiskOut = read

On cluster level (the "All" row), the sum DiskIn + DiskOut "could" match protocols Total,

and in fact it roughly does  in your example.

Looking at the individual nodes in the example, one can nicely see how all nodes' disk contribute

almost evenly to the only NFS activity on single node 3. That's an essential aspect of OneFS.

In other situations, looking at the entire cluster:

- disk throughput can be HIGHER than protocols throughput: e.g. with internal restripe jobs

- disk throughput can be LOWER than protocols throughput: e.g. with good cache hit rates

Also note that node 3 has a much higher CPU load than the rest,

due to the active NFS workload.

Makes sense? Too deep already? or left out other questions?

-- Peter

Hello sjones5,

Thanks for reply.

I check the KB information you told.

Hello Peter,

Thank you for telling me deep information the man isi-statistics command didn't show.

I could deepen my understanding about the "isi statistics system" command data.

So, Could you tell me about "isi statistics client" command data ?

When I run the following command,

  # isi statistics client --long -i 5

the following output displayed as result.

In this data, is there deep information which "man isi-statistics" does not show ?

Quick note, in the second row,  all units starting from N/s should be shifted one column to the right,

NumOps has no unit, it's just the count of operations in the 5-second sampling interval,

while Ops is the per-seconds operations rate -- both columns have equivalent information.

Naming and order of the In/Out columns are confusing at best...

InMax, InMin and InAvg refer to request block sizes in Bytes,

whereas "In" shows the input data rates in Bytes/s.

Same for OutMax, OutMin, OutAvg and Out.

TimeMax, TimeMin and TimeAvg show the processing times or "latencies".

Starting from OneFS 7.2, the reported NFS protocol latencies can be impacted

by slow clients which makes performance analysis a bit more difficult than before.

The operation "Class" should be interpreted in the context of each Protocol.

Read and write classes are clear, but e.g. NFS namespace_read includes

NFS lookups as well as getattrs. The "isi statistics protocol" command

can break things further down to operations level (leaving out "Users" in exchange.)

UserName is the result of the cluster doing a user lookup for the

given numerical UserID -- in your example this was not possible for UID 2301.

LocalAdress and RemoteAddress refer to the used IP addresses on the Isilon Node

and on the client, respectively. The cluster tries to make DNS lookups for you

to produce LocalName and RemoteName.

General tips:

- select columns to show with the  --output option

- get overviews by aggregating rows with the --totalby option,

   e.g. --totalby UserID or --totalby Node

   cool tip: --totalby accepts multipe fields like: --totalby UserID,Node

Happy monitoring;  let me know if there are more questions!

-- Peter

Hello Peter,

Thank you for reply.

If I want to get NFS network data transfer performance,

should I run "isi statistics protocol --protocols nfs3 --long" command

rather than "isi statistics system/client" command ?

And, does "isi statistics protocol --protocols nfs3 --long" command output data not include

other data (e.g. SyncIQ, NDMP backup and so on) ?

The various statistics sub-commands offer different break-outs,

so you can choose any of them, according to your needs.

For just combined In+Out throughput by node, "isi statistics system" is fine,

for troughput and activity by users and/or clients, "isi statistics client ..." is the only way to go,

while insights down to protocol operations level are only provided by "isi statistics protcol ...".

In either case, reported NFS throughput never contains other traffic, such as SyncIQ or NDMP etc.

Cheers

-- Peter