In the second part of this series, we are going to target 3 different types of potential bottlenecks that a system might suffer from.  Specifically these are:

  • IO Bound
  • CPU Bound
  • Memory Bound

Most performance problems can be placed into one of these categories.  By using what we learned in Part 1, we can use the information from vmstat to provide more information about what aspect of a system is limiting it from handling a greater load….and therefore what should be upgraded to provided greater capacity.

IO Bound System

An IO bound system is a machine that is usually suffering from performance limitations due to disk speed.  While there are other types of IO a server could be waiting on, usually it is disk and disk speed that are the problems.  This can be culled from reviewing vmstat by looking at both the uninterruptable sleep column (b column) and a high value in the wa column.    An example from vmstate would look as follows:

#vmstat 5

procs      ———–memory———-             —swap–       —–io—- –system–                          —–cpu——
r  b        swpd   free   buff  cache                   si   so          bi    bo   in   cs                             us sy id wa st

1  8      5888 140136 167700 1506440        0    0        6783 17333  1738  16648                   6  2 9  83  0
1  6      5888 140136 167700 1506440        0    0        6885 18353  1831  18658                   8  4 9  77  0
1  6      5888 140136 167700 1506440        0    0        6482 12323  1534  17632                   16  12 9  63  0

In the above, you can clearly see both issues at work as predicted above.  The wa is quite high 63-83 and the number of processes in uninterpretable sleep is between 6 and 8.

Memory Bound – Swapping

The second type of system we will look at is a system that is attempting to allocate more memory than physically exists in the system.  It is normal for a linux/unix system to use swap to augment its available memory.   As such, any system can show some amount of memory allocated to swap.  However, this only becomes a problem when the system starts waiting for memory to be paged in and out of swap to satisfy its processes.  From a vmstat perspective, this would appear in 2 ways.  First is the swpd column.  However, the presence of large values in the swpd column does not, in and of itself, constitute swapping.  The more important demonstration of this is high values in si and so columns.  Here is a sample vmstat that shows some swapping.

#vmstat 5

procs      ———–memory———-                   —swap–       —–io—- –system–              —–cpu——

r  b        swpd   free   buff  cache                    si           so          bi    bo   in   cs               us sy id wa st
1  18   3794291 140136 167700 1506440     19851    9780        16783 27333  1738  1648        6    2 29  63  0
1  16   3797936 140136 167700 1506440     15913  30807        38353 28331  1658  3342        8  14 19  57  0
1  10   3847364 140136 167700 1506440     18334  22355        26482 32323  1534  1732      16  12 29  33  0

CPU Bound Machine

The last type of system we will review is a CPU-Bound machine.  A server that is experiencing this problem simply does not have enough processing power to properly perform its operations.  Usually this system will show a high number in the us column.  Again, remember the us column shows us the amount of time spent processing non-kernel code.  It is also quite normal in this scenario to have a larger number in the r column as they are queued up waiting for cpu time.

procs ———–memory———-      —swap–      —–io—-      –system–      —-cpu—-
r  b   swpd   free   buff  cache   si   so         bi    bo        in    cs      us sy id wa
5  1    648 755300  31460 2095012         0    0         76  3458 2013  1369      95  5  0  0
5  0    648 741196  31568 2103336         0    0         24  1852 2063  1481      88 12  0  0
8  0    648 706828  31616 2115936         0    0         50  2590 2118  1609      90 10  0  0
14  0    648 700628  31664 2133024       0    0       117  4320 2138  1290      90 10  0  0