ISM Book Chapter 2 - performance and IOPS

Hello Sebastian,

I do not want to comment on the details of any particular disk. However, here is as an explanation as to how you might approach some of these statistics (and ideally benchmark/test).

Let  S = Service Time = average time taken to complete an I/O request.

So,  S = {mechanical delay reading from platters} + {transfer time through the internal drive buffer onto the external transport}.

So,  S = M + T

Where     M = mechanical delay reading from platters
And          T = transfer time through the internal drive buffer onto the external transport

Consider M first.

M = {time to seek to correct track} + {average rotational latency}

M = {time to seek to correct track} + {time for 1/2 of a full revolution}   .............     (average of 1/2 is defined by ISM)

M = {3.6 ms} + { 1/2 x  60/15000 x 1000 ms}      ............. (I just chose 3.6 as an example)

M = {3.6 ms} + {2 ms} = 5.6ms

Now consider T. Well, we do not have enough information, but that is OK for the sake of this. Let's imagine that the drive internal transfer rate is a modest 40MB/s and that the application random I/O size is 4KB.

T = 4/(40 x 1000) = 0.1 ms     (i.e. A small percentage of S)

So, M = 5.6 + 0.1 ms/IO = 5.7 ms/IO

Therefore IO/s = IOPS = 1000/5.7 = 175.

Now, around Little's law and the disk controller, you do not want disk controller Utilization to go above 70%, otherwise the response time increases exponentially.

So,

0.7 x 175 = 122 IOPS as a working limit for the disk. If you have an application requiring 1000 IOPS, then a recommended drive count (to manage random performance) would be 1000/122 ~ 9. Always nice to be able to benchmark and test these figures, especially to test our understanding.

Of course, we then need to build redundancy using a suitable RAID level.

HTH, best regards, Richard.

Hey Richard,

first of all thanks so much for you answer!

I see now where part of my error lies. The calculation for the time needed for 1/2 of a full revolution. Somehow I got my math screwed up in that part, but regarding little's law, do we really just take the maximum amount of theoretical IOPS that a controller can handle, or do we take the maximum amount of IOPS for a single disk and just use 70% of that? Just from a theoretical point of view I would say that with one disk, the limit is not with the controller but with the physical disk itself? As soon as we start talking about arrays I would assume the limit would be located with the controller since we get a combined value for the I/O that an array of disks can handle. Or have I overlooked something (again)?

Cheers,

Bas

Hi Bas,

Thanks.

Actually in my explanation I am referring to the disk controller (not a RAID controller). The disk controller is physically part of the drive. When sizing for random I/O we should use a figure of 70% of the maximum IOPS that the disk can theoretically support (disk here being the combination of platters, mechanics and disk controller). If we go above 70% of disk controller utilization then response times can increase very dramatically. In my example calculation, I have used 70%. (The disk controller is doing a lot of good work - e.g. command re-ordering / tag queuing etc).

You bring up a good point. If the drives are behind a RAID controller (with intelligent cache algorithms, and support for read-cache hits, and write-behind cache operations), then things can only improve. In my example calculation, you might decide to use 9 disks as a basis/foundation for a benchmark. And then you can re-benchmark with a different drive count, and work to understand the results. You will never really know for sure until you have performed a benchmark in some way.

Cheers, Richard.

Hmm, this poses some interesting questions. When I take in to account things like smaller external disk arrays (for example HP MSA30 or MSA50) I don't always have the full spec on the disks used, and my performance will vary depending on the configuration of the array that I decide to implement. In such a case I need to go with the amount required for my application (or perhaps even calculate the maximum my controller or adapter can give me) and check with the vendor if they are able to match a product to my requirements.

One things that comes to mind is if the manufacturer of such solutions also calculated according to these calculations and gives out numbers with a certain percentage of saturation? As far as I can recollect I haven't seen that many manufacturers post numbers on performance in combination with assumed saturation (be it EMC or any other provider).

Could it be that it all boils down to "pinning down" providers to the SLA's or performance spec they guarantee? If so, this info is useful in a sense that it helps me understand performance wise, but not really relevant when implementing a new storage system? I calculate what performance I need and go to the manufacturer of my choice and ask if they are willing to put in writing that their solution can live up to the IOPS I require?

I know, probably the wrong place to ask, and not really something technical, but still interesting in my opinion.

Cheers,

Bas

Well... I hadn't quite finished, but I have just learnt that if you hit (as I do automatically in my preferred editor of choice), then this will post the page. Anyway, I think I had just about finished, apart from a spell-check.

Cheers, Richard.

Hello Bas,

More good points. I think the example calculation I gave is just to help you think about configuring the correct number of disks that are required in the stripe. If you cannot get any specifications on the disk capabilities, then that may be a problem. Ideally, you would understand the I/O profile of your application, and the disk capabilities as published by the disk vendor, and then size an appropriate number of disks for the stripe. Oh, and it would be very nice indeed if we always had the luxury and time to benchmark. I realise commercially, many engineers are under pressure to complete the task yesterday   I agree, I have not seen too much reporting of saturation, but the 70% guideline can be tested through the benchmarking process.

As you know, as far as performance is concerned, the performance 'stack' is huge. Consider that a software house writes a tele-sales order entry system application for a customer; and the order entry clerk always wants sub-second response times when keying-in the customer's order and moving from one screen to the next (effectlively defining an application SLA). Here are just some of the physical and logical objects in the performance 'stack' (I thought it might be a good place to reference an 'incomplete' list - I expect many people would want to add additional points).

- how well does the compiler optimize

- how well is the application coded

- has the relational database been designed according to sound relational database design principles

- how well are structured queries optimized (relating to the previous point)

- does the filesystem have a suitable block size

- tuning of kernel drivers

- server CPU utilization

- server memory utilization

- OS memory paging concerns

- qdepth on HBA ports

- HBA fcode optimization

- physical integrity of transport from host to FC switch (injected bit errors leading to kernel driver timeouts)

- ASIC layout on fabric switches

- switch <-> switch flow control

- supported queue depth on storage ports

- storage port utilization (cf. fan-out ratio)

- RAID controller cache algorithms

- how well has the RAID stripe been configured (this is partly covered in the discussion above)

- is the logical address space seen by the host application aligned on a stripe boundary

- physical placement of data on the platters

- drive mechancs and supported IOPS

In my example, if the software house is going to provide a sub-second SLA, then they would really need to work with the customer, and the customer's equipment in order to benchmark and understand the results.

++ will do the same for you in Firefox, but I know what that feels like.

Let me start by agreeing with you. Benchmarking and proof of concepts are highly valuable in my opinion and be performed if you have the chance to do so. Also, the 70% rule is a good one (I found that it's the same for CPU load on a system. As soon as you go >70% load, your I/O performance will plummet) but it would be interesting to know if the manufacturers of the various products/arrays also implemented this 70% guideline and communicate their IOPS based on the 70% guideline. Perhaps you know how EMC handles this?

I just looked over your list, and as a true techy you list some valid points there. You could also include things like IO optimization of driver and kernel source code or just plain and simple things like RAM type (clock speeds of RAM, etc) and CPU speeds. Although some of those things will obviously have a greater impact on the total amount of max. IOPS than others.

If this is actually the only way to get data that you can rely upon you really have no other choice then working with people who know this by heart and are experts in the implementation scenario, benchmark all viable options you are considering, or perhaps even (taking new tech like cloud in to account) agree upon a certain service/performance level for parts of the stack and go see a service provider for your requirements.

Perhaps we can add to the list a bit more, and people might be able to comment on items and give some comments on the item itself and how they found it relates to the I/O performance of a system as a whole?

Hi Vasily.... thanks for that (and sorry for the delay in my reply.... just looking through some old posts..... better late than never.... Cheers and best wishes).

Hello Richard,

Hello Betty,

Hello again Betty,

Am surprised how does it goes in RAID controller and SAN situations ? all above calculations ?

Hello Nee,