Windows performance counters and Symmetrix disks

Hi

We don't have any EMC kit to mention of, but we have a customer who has a customer that has an issue.  We make a capacity management product called Athene, it captures performance data from lots of environments and for Windows we capture the standard performance counters.  Our customer's customer (who is in Japan just to make it even more interesting) are complaining that the Logical Disk and Physical Disk % Idle Time metrics we are capturing are incorrect because they are negative.

We looked at the data in our database - it was wrong, so we looked at the raw data directly from Windows, and that was wrong too, so we aren't convinced it's us doing anything badly.  Microsoft disagree (bless them) so our customer's customer is looking to us to "fix" the issue.

Essentially Windows performance counters are "gas meters" - they just go on getting bigger until the system restarts.  We figure out what happened in a given period of time by subtracting the "new" reading from the "old" one and store that difference.

In this case, however, the "new" numbers are smaller than the "old" ones - and no, the system hasn't restarted, or the counter overflowed.

This is a dump of a couple of consecutive counter values we have seen:

(1) Actual capture time: 12/03/2013 01:05:00.091

Performance counter  543081483241 Performance counter freq  2474111 Performance counter 100ns  130075239000919494

Logical Disk H:

PERF_PRECISION_100NS_TIMER 130362426000  PERF_LARGE_RAW_BASE  130075239011309494

Physical Disk 3 H:

PERF_PRECISION_100NS_TIMER 130365316000 PERF_LARGE_RAW_BASE 130075239011309494 

 

(2) Actual capture time: 12/03/2013 01:09:59.998

Performance counter  543823509171 Performance counter freq  2474111 Performance counter 100ns  130075241999989494

Logical Disk H:

PERF_PRECISION_100NS_TIMER 130352413000 PERF_LARGE_RAW_BASE 130075242000239494

Physical Disk 3 H:

PERF_PRECISION_100NS_TIMER 130355302000 PERF_LARGE_RAW_BASE 130075242000239494 

You can see that although the BASE counters in (2) have gone up from (1) but the PERF_PRECISION counters have gone down.  By all the laws of Windows counters, this should not happen.  In subsequent snaps of the data, it all goes back up again.

The customer says they are running Windows 2008 R2 and have VMAX disks (I know nothing more!)

Does anyone have any experience of this kind of thing?  I can't make the Search Support work for me, I get stuck signing on over and over, hence the question here.

Thanks

Nick V