Capacity Planning for the Masses -- Using the SE Toolkit and Orca

Justin Buhler and Adrian Cockcroft

Capacity planning is often overlooked by systems administrators until they need to justify additional hardware resources, and that's the wrong time to start. In this article, Adrian Cockcroft (author of the Sun Performance and Tuning book) and I intend to show that starting a capacity plan within your Solaris environment is not as difficult as it sounds. We will introduce some key principles for beginning your capacity planning and will show how we applied those principles and the tools presented in this article during the Winter 2002 Olympics.

Capacity Planning Principles

The fundamental principle behind any capacity plan begins with establishing an accurate baseline of your application's behavior over a period of time that can be easily interpreted. The following is a breakdown of this idea with the tools we use to achieve the fundamental principle.

1. A capacity plan starts with accuracy, so use the right tools.

A starting point is to capture accurate system behavior information by using microstate accounting tools. This means your tools can harvest the performance data from the system without becoming one of the performance problems that needs to be investigated. The SE Toolkit developed by Rich Pettit and Adrian Cockcroft is able to accomplish this; not only are the access methods very lightweight, but the interpreter itself is written such that execution speed is very fast. SE Toolkit is available free from:

http://www.setoolkit.com

By breaking down the system into workload sets, you can intuitively analyze the machine in terms of categories or "slices of a pie". This categorizing or slicing is called workload analysis. Because every system runs a mixture of workloads, you just need to pick out the relevant processes and then aggregate the results. The rest is considered overhead of unplanned activity.

The workollator.se class, which is included in the SE Toolkit 3.3, was written during the Winter 2002 Olympics, to ease the collection of accurate data about a defined workload set in a single consistent daily log. This allows you to see what each process or workload is doing during the given period of time.

3. "A picture is worth a thousand words", so make the raw data intuitive to understand.

You need to be able to make sense of the data that you are collecting about your workloads. Looking at thousands of lines of performance data is not intuitive, even for gurus. Graphing your performance data gives you the best visual representation of all of those numbers and is much easier to decipher. One tool that enables this visualization of your performance data is Orca, which is available from:

http://www.orcaware.com

http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

and allows long-term data views ranging from hours to years. Orca is also extended to use the SE Toolkit, and a new configuration file described below adds workollator.se workload data parsing and a set of standard plots.

Now, let's apply these principles and tools to a real-world example. We used these tools to determine the capacity for the Winter 2002 Olympics. To begin, we must clarify the location and the purpose of each of the tools. They are deployed in the following manner:

• SE Toolkit and workollator.se are installed and configured to run on the Solaris systems to be monitored.
• Orca, orca_gather.pl, and workload_extract.sh are installed on the Solaris server, where information is to be centrally collected and processed.
• The Web server is to be installed on the same server where the Orca html output is stored. It is only used to serve html, and no CGI configuration is needed.

Next, we can install the tools on the server to be monitored and start harvesting our application's stats.

Download and install the SE Toolkit 3.3 from http://www.setoolkit.com. We will next configure a shell script to capture your system's workloads using workorallator.se on the host to be monitored. This script uses environment variables that identify the processes, which make the different workloads on the system. The environment variables are:

PW_CMD_WK# -- Filter based on the command name.

PW_ARGS_WK# -- Filter based on the argument strings.

PW_USER_WK# -- Filter based on the user name.

Listing 1 shows an actual example from SLC Sun servers to capture the workload information from the Winter 2002 Olympic Games. Listings for this article are available from the Sys Admin Web site at: http://www.sysadminmag.com.

Note that the PW_CMD_3 and PW_USER_3 on lines 5 and 6 have the same workload number of _3. This is because we can stack variables and then filter. We are filtering all the Java processes running as the cis user. Also, the 300 after the workollator.se on line 14 represents the number of seconds after which the process will wake up and capture the workload data in a daily log file, located in the directory as defined by the OUTDIR variable.

An example of the log file created on the system by the workollator.se configuration is shown in Figure 1. The file is created daily in the directory /opt/orca/workload/ with the name of iso-cissvr-001-2002-02-19.

Collecting Data Centrally Using rsync and SSH

The workload data from the servers must be collected centrally on to the Orca server for processing. In Salt Lake, we used a simple Perl script using rsync (http://www.samba.org/rsync) and SSH to read from a file the IP addresses and hostnames of all the servers from which data needed to be collected. The SSH account that was used to authenticate needed only read access to the workollator logs in the directory as defined in the OUTDIR variable in our cis_workollator.sh script (see Listing 2).

Installing Orca

The next step is to install Orca on a Web server machine. Orca provides an excellent "Orca installation and configuration guide", located in the contents of the package, which can be downloaded from:

http://www.orcaware.com/orca

Introduction to Orca and RRD

Orca is a toolkit that will graph arbitrary data into HTML documents using PNG or GIF graphics generated from an RRD (Round Robin Database). RRD is the datastore, or database, that is used to store data in units. RRD will generate time-defined graphs of the data stored within the given RRD.

Orca is managed and configured from a single configuration file, called orcallator.cfg (see Figure 2). The configuration file is passed to the Orca application, which provides site-specific information, such as where to go to find the files, the name of the files, and what type of graphs to plot. The command to run after you have configured Orca is normally:

#>./orca -v -once  ../lib/orcallator.cfg

We have provided a replacement for the orcallator.cfg, called workollator.cfg, which should be passed to the Orca binary for the configuration information about how to generate the RRDs and graphs using the workload files. You must move the workollator.cfg file to the same location as the orcallator.cfg, which is usually /usr/local/lib. Next, you must modify the workollator.cfg and give it the relevant parameters specific to your site.

The following parameters tell Orca where to find or place RRDs and HTML files, as well as Orca's installation directory. Set these to the proper places. See Figure 4:

rrd_dir -- Location in which RRD databases should be created or already exist.

html_dir -- Location to write or update the html files.

base_dir -- Location of the Orca installation.

The regular expression shown in Figure 5 tells Orca where to find the files and the name of those files to be loaded into RRDs. Note that there is a bug in the current distribution of the workollator script, so you need to remove the following line:

<a href="http://www.sun.com">

Once all of the workload data has been transferred from the Solaris servers to the Orca server, we must parse each of the daily workload logs into individual files per workload. The workload_extrctor.sh script (Listing 3) performs this task. Run this script in the directory containing all of the workload files. The result should be that additional directories are created (one for each hosts workload) under the OUTPUTDIR directory, which each contains log files for each day.

Examples of the workload directories created on the system by the workload_extractor.sh configuration are shown below. A daily log file is created under each of the host-workload directories under /opt/orca/workload/, with the name of workol-2002-02-19.

#> ls /opt/orca/var/workload
iso-cissvr-001____
iso-cissvr-001___bb
iso-cissvr-001___cis
iso-cissvr-001___root
iso-cissvr-001__tripwire_
iso-cissvr-001_httpd___
iso-cissvr-001_java___
iso-cissvr-001_java__cis
iso-cissvr-001_se.sparc___
iso-cissvr-001_Total___

#> ls /opt/orca/var/workload/iso-cissvr-001_java__cis
workol-2002-02-17
workol-2002-02-18
workol-2002-02-19
workol-2002-02-20
workol-2002-02-21
workol-2002-02-22

You should now be able to run the Orca application and generate your data. The -v flag will enable verbose logging on all of the processing, which is useful when running Orca, especially if it is the first time to run Orca:

./ORCA -v -once ../lib/workollator.cfg

An example of a process with high chld% percentages is our system monitoring package, which executes a lot of shell scripts. See Figure 9.

Before we get into the actual planning model and examples, you'll need to be familiar with the various performance data values and how and why they are important. For a quick introduction, please see the "Performance Data Values" sidebar.

Workload Capacity Planning Model and Example

Now that we have shown how to get the workload data into a nice format, let's finish up with the formula that can be used to scale the capacity of a system.

The formula for workload-based capacity planning should be based on the workollator data, because this shows how much of the CPU is being used by the workload you are interested in rather than a system total. The system total can be considered as overhead or possible bottlenecks to the workload being scaled. Note that the workload shows what the workload uses in (%usr time) and what the system does to service the workload in (%sys time).

The formula used in the workload-based capacity planning model to scale the workload capacity is:

Total Sizing = Overhead + (Workload * Factor)

Overhead is everything else that is not included in the workload being modeled or Overhead = (Total - Workload). Factor is how much the workload has changed by or scale.

Here is an example of HTTP Server Single Workload CPU Usage:

HTTP Web Server Workload is using 15% (10 %usr + 5 %sys)
OVERHEAD is 5% total usage
Growth Factor is 5
Total CPU Usage 80% = 5% + (15% * 5)

The estimated total CPU usage if our workload grew by 5 times would be approximately 80%.

Here is an example of a Mixed Workload Oracle and HTTP Web server:

ORACLE Workload is using 45% (35 %usr + 10 %sys)
HTTP Web Server Workload is using 15% (10 %usr + 5 %sys)
OVERHEAD is 15% total usage
Total Usage = 75%
Growth Factor for BOTH is 2
Total CPU usage is 135% = 15% + (15% * 2) + (45% * 2)

Remember that if you are using a multithreaded application, you can think of 100% per CPU. For the workload above, we would be running a workload that demands 1.35 CPUs; or if this were a single CPU server, you would need to look at purchasing some additional hardware.

The more traditional method for sizing is (Total Sizing = Total * Factor). This is generally a less accurate formula because it assumes that all resources on the system scale with the same factor, which is not generally the case.

Conclusions

By using the SE Toolkit and Orca, you will quickly be able to establish a visual representation of your Sun servers' behavior and the applications that are installed. This will enable you to see and communicate a clearer and more accurate picture regarding the resource usage of all the applications. Finally, coupled with some general points, you should be able to develop a high-level understanding of the capacity of your system based on the current workloads configured on your system. With a few workload-based formulas, you can begin your capacity planning.

Acknowledgements

I extend special thanks to Adrian, who is a great guy to work with and truly an expert in the field of capacity planning and performance analysis. Thanks also to my wife and daughter who show continuous support through their patience.

Justin Buhler is currently Senior UNIX Administrator, Schlumberger, Major Event's division for the Athens 2004 Summer Olympics. His previous projects include Salt Lake 2002 Winter Olympics. He can be reached at: jpbuhler@athens2004.com.

Adrian Cockroft is Chief Architect for High Performance Technical Computing at Sun. The author of many books regarding performance and tuning and capacity planning, most notably Sun Performance and Tuning: SPARC and Solaris, published by Sun Microsystems Press/PTR Prentice Hall. Adrian is also co-author of the SE Toolkit.