Archive for category Dedicated

Reading vmstat in linux – Part 2

In the second part of this series, we are going to target 3 different types of potential bottlenecks that a system might suffer from.  Specifically these are:

  • IO Bound
  • CPU Bound
  • Memory Bound

Most performance problems can be placed into one of these categories.  By using what we learned in Part 1, we can use the information from vmstat to provide more information about what aspect of a system is limiting it from handling a greater load….and therefore what should be upgraded to provided greater capacity.

IO Bound System

An IO bound system is a machine that is usually suffering from performance limitations due to disk speed.  While there are other types of IO a server could be waiting on, usually it is disk and disk speed that are the problems.  This can be culled from reviewing vmstat by looking at both the uninterruptable sleep column (b column) and a high value in the wa column.    An example from vmstate would look as follows:

#vmstat 5

procs      ———–memory———-             —swap–       —–io—- –system–                          —–cpu——
r  b        swpd   free   buff  cache                   si   so          bi    bo   in   cs                             us sy id wa st

1  8      5888 140136 167700 1506440        0    0        6783 17333  1738  16648                   6  2 9  83  0
1  6      5888 140136 167700 1506440        0    0        6885 18353  1831  18658                   8  4 9  77  0
1  6      5888 140136 167700 1506440        0    0        6482 12323  1534  17632                   16  12 9  63  0

In the above, you can clearly see both issues at work as predicted above.  The wa is quite high 63-83 and the number of processes in uninterpretable sleep is between 6 and 8.

Memory Bound – Swapping

The second type of system we will look at is a system that is attempting to allocate more memory than physically exists in the system.  It is normal for a linux/unix system to use swap to augment its available memory.   As such, any system can show some amount of memory allocated to swap.  However, this only becomes a problem when the system starts waiting for memory to be paged in and out of swap to satisfy its processes.  From a vmstat perspective, this would appear in 2 ways.  First is the swpd column.  However, the presence of large values in the swpd column does not, in and of itself, constitute swapping.  The more important demonstration of this is high values in si and so columns.  Here is a sample vmstat that shows some swapping.

#vmstat 5

procs      ———–memory———-                   —swap–       —–io—- –system–              —–cpu——

r  b        swpd   free   buff  cache                    si           so          bi    bo   in   cs               us sy id wa st
1  18   3794291 140136 167700 1506440     19851    9780        16783 27333  1738  1648        6    2 29  63  0
1  16   3797936 140136 167700 1506440     15913  30807        38353 28331  1658  3342        8  14 19  57  0
1  10   3847364 140136 167700 1506440     18334  22355        26482 32323  1534  1732      16  12 29  33  0

CPU Bound Machine

The last type of system we will review is a CPU-Bound machine.  A server that is experiencing this problem simply does not have enough processing power to properly perform its operations.  Usually this system will show a high number in the us column.  Again, remember the us column shows us the amount of time spent processing non-kernel code.  It is also quite normal in this scenario to have a larger number in the r column as they are queued up waiting for cpu time.

procs ———–memory———-      —swap–      —–io—-      –system–      —-cpu—-
r  b   swpd   free   buff  cache   si   so         bi    bo        in    cs      us sy id wa
5  1    648 755300  31460 2095012         0    0         76  3458 2013  1369      95  5  0  0
5  0    648 741196  31568 2103336         0    0         24  1852 2063  1481      88 12  0  0
8  0    648 706828  31616 2115936         0    0         50  2590 2118  1609      90 10  0  0
14  0    648 700628  31664 2133024       0    0       117  4320 2138  1290      90 10  0  0


Server Clustering Solutions

One of the mainstays of our hosting business is the service and support of high usage PHP-MySQL applications. There are many ways to support these types of environments. One of the ways that we are assisting our customers is by providing implementation of server clustering. Server clustering provides both fault tolerance and greater scalability to hosting environments.

Very few websites start out needed a fully clustered solution. Most customers start on the low end and grow their environment as needed. The route most people take is as follows:

  1. Shared Web Hosting
  2. VPS Hosting
  3. Dedicated Server
  4. 2 Server Cluster
  5. Load Balanced Server Cluster

While this is by no means an exhaustive list, it does show the progression to more complicated solutions.

As mentioned earlier, there are a couple different reasons to look to clustering.  Primarily we will focus on solving the performance problems in this issue.  Once a site has outgrown a single server solution, multiple servers must be employed to provide for further grown.  If the website is a simple html/php site with no database access, multiple web servers will suffice.  However, most modern websites are dynamic, database-driven sites.  The simplest method for clustering is to separate the database function from the web serving function.  It is also important to ensure a high speed connection between the database server and web server.

The dual server cluster is often enough for many sites, but again, there are some sites that will need more resources available to them.   At this point, it is important to have an experienced administrator audit the server to see where the bottlenecks are occurring.   Depending upon the outcome of this analysis, additional servers may be needed in either role, web or database servers.

Tags: , ,