 Web
Polygraph
Mihalis Tsoukalos
Web Polygraph is a freely available benchmarking tool for caching
proxies, origin server accelerators, L4/7 switches, content filters,
and other Web intermediaries. It is offered in two versions: a freeware
version and a $997/year development version. I use the development
version, which is shown in some of the examples in this article,
but the features I describe are all available in the freeware version
also. The installation and the running process are exactly the same
for the two versions.
Installing Web Polygraph
The compilation can be done following the usual procedure after
extracting the source code archive, as follows:
1. Configure:
~/code/polygraph-peak-finder-mid3$ ./configure --prefix=/usr/local/polydev
2. Compile:
~/code/polygraph-peak-finder-mid3$ make
If there are missing libraries (such as the ssl development files),
the compilation might fail but installing the missing libraries will
solve the problem.
3. Install:
~/code/polygraph-peak-finder-mid3$ su -c "make install"
The option --prefix=/usr/local/polydev tells the configure
script what the base installation directory will be. The compilation
might take a while, especially on a slow machine with little memory.
Web Polygraph in a Nutshell
For those who are not as experienced in this particular area,
Web Polygraph may appear somewhat complicated at first. However,
the online documents and the Web Polygraph mailing list are very
helpful in understanding and successfully running the software.
The first thing to understand is that for Web Polygraph to operate,
two programs must run: polysrv and polyclt. The first is the server
process, and the second is the client process.
Polyclt is the process that requests URLs from the proxy server
(the Squid proxy server will be used), and polysrv is the process
that will serve the proxy server. There is also a configuration
file that must be set up properly in order to run the test. This
file defines the workload that will be used for the test as well
as other test parameters. The client process creates robots and
the server process creates (of course) server processes for the
benchmark. Figure 1 shows this in a graphical way.
A "workload", a "test", and a "benchmark"
are used in this article to describe the same thing. A workload
is a specific way of testing a proxy server. A workload has information
about the duration or the size of the data to be requested, the
number of simultaneous connections, the type of the contents (gif,
html, cgi scripts, etc.), and so forth. Some workloads that have
been used in previous tests (industry-wide competitions for Web
caching proxies) have specific names. The Web page of the program
describes a workload called PolyMix-4:
PolyMix-4 is based on our experience with using PolyMix-3 during
the third cache-off and other tests. We have eliminated some of
the known problems of the old workloads and added new features.
The ultimate goal is, of course, getting our model closer to the
real worlds.
Another term that is being used is the "robot". A workload
consists of robots sending and receiving requests and servers serving
those requests. A robot is the main logical thread of execution
in polyclt. Robots submit requests and receive replies.
A test can have many phases, each one of them doing a different
thing. One phase can fill the proxy cache disk space before doing
the main benchmarking. Another phase can "wake up" the
proxy server by requesting a low rate of documents for 30 minutes.
And another can stress-test the server for 4 hours or check the
proxy server for a certain type of URLs, such as images, scripts,
static pages, etc.
Testing Network Speed
The network speed is very important for the results because to
achieve a certain workload, the network speed must be able to support
it. In other words, 500 requests/second with a small response time
cannot be achieved with a 10Mbit/sec network speed. If your network
can operate on full-duplex mode, that's even better.
So the network speed must be tested before running the benchmark.
This can be done by using netperf, which is a network performance
benchmark server. A simple test for the network speed is the following:
router:~# netperf -l 30 -H 10.100.1.3 -t TCP_STREAM
TCP STREAM TEST to 10.100.1.3
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 106bits/sec
87380 16384 16384 30.00 89.17
router:~#
This tells us that the network speed between our computer and the
computer with IP 10.100.1.3 is approximately 89.17 Mbits/sec. The
-l parameter allows us to define the duration of the test.
In this example, it is 30 seconds.
Proxy Servers and Web Caching
A proxy server can have multiple roles apart from letting clients
access resources on the Web. These include sharing various resources,
caching, anonymization, transformation of requests and/or responses,
and filtering requests/responses.
A proxy server acts both as a server to its clients and as a client
to other proxies or Web servers. Proxies can exist in many places
inside the path between a home computer requesting a Web page and
the actual Web server that stores the Web page. In many situations,
a proxy can filter the contents of a page and either serve the client,
or not, depending on criteria given by us. The first thing a proxy
server must do is to decide whether the page served is cacheable.
A proxy server must also remove objects from its cache once the
cache is full. Many algorithms exist for replacing the data of the
proxy cache.
The need for benchmarking comes from the need to identify whether
a proxy server can meet given requirements. For example:
- Can our company's proxy server serve 50 requests per second,
which is the load at peak hours?
- If the proxy cache size is tripled, do we get any improvements
in response time?
- Is it better to double the memory of the proxy server, or replace
the IDE hard disks with SCSI ones?
- Is our proxy server the point of delay, or is it the network
or our DNS server?
- Which replacement algorithm is better for our kind of data?
A simple test of the proxy server was run with the following two
commands:
mtsouk@racoon:/usr/local/polydev/bin$ ./polysrv --config
/usr/local/polydev/workloads/SysAdmin.pg --verb_lvl 10
--log /var/tmp/srv.log
mtsouk@racoon:/usr/local/polydev/bin$ ./polyclt --config
/usr/local/polydev/workloads/SysAdmin.pg --verb_lvl 10
--log /var/tmp/clt.log --proxy 10.100.123.123:3128
The proxy server must already be running and properly configured for
the test to proceed. For the purposes of this simple test, an existing
workload was used, which is named simple.pg. I changed the name of
the file and made some changes to it. I then renamed the file SysAdmin.pg.
The new contents of the file are shown in Listing 1.
At first, this seems a little cryptic, but it is not that difficult.
The only changes made to the original simple.pg file are the two
addresses fields in Server S and Robot S blocks. This is a never-ending
test, so it had to be terminated by pressing Ctrl-C. Note that this
simple test ran on only one machine, with the IP address of 10.100.123.123.
Polyfill-2.pg
Polyfill-2 (Listing 2) is a test that does have an end, so there's
no need to terminate it manually. Again, we must modify the polyfill-2.pg
file in order to use our network addresses. See the sidebar "Polyfill-2"
for a more complete description of this test.
Make sure the clocks between the machines are synchronized before
running the tests. Otherwise, error messages will appear on screen.
In this benchmark, the --console parameter was added to get
the output in a file instead of the terminal screen. The polysrv
process was run as follows:
mtsouk@home:/usr/local/polydev/bin$ /polysrv --config
/usr/local/polydev/workloads/POLYFILL-2.pg
--verb_lvl 10 --log /var/tmp/ POLYFILL-2.srv.log --console
/var/tmp/POLYFILL-2.srv.console
A Real Benchmark -- Polymix-2
Polymix-2 is a relatively simple but real benchmark included with
Web Polygraph. It's a good example of the kind of tests you
can perform with Web Polygraph. You may need to make some routine
modifications to Polymix-2 to run it in your environment. The entire
test takes about 14 hours and is divided into the following 7 phases:
1. Phase inc1: (00-01) The load is increased during the first
hour to reach its peak level.
2. Phase top1: (01-05) The period of peak "daily" load.
3. Phase dec1: (05-06) The load steadily goes down, reaching a
period of relatively low load.
4. Phase idle: (06-08) The "idle" period with load level
around 10% of the peak request rate.
5. Phase inc2: (08-09) The load is increased to reach its peak
level again.
6. Phase top2: (09-13) The second period of peak "daily"
load.
7. Phase dec2: (13-14) The load steadily goes down to zero.
See the sidebar "PolyMix-2" for more details on the
Polymix-2 benchmark.
Web Polygraph Output
As Web Polygraph is running, output is generated either on screen
or in a file, depending on our specifications. The output of polyclt
is more informative and richer than the output of polysrv, so polyclt's
output will be presented here.
Generally, polyclt output can be divided in three sections. The
first section contains general information, like the command that
was used to execute polyclt, configuration info, server content
distributions, the phases of the benchmark, the number of available
file descriptors, and the generation of each robot that will take
part in the benchmark.
The second section is the most interesting part of the output;
it is divided into nine columns:
(1)~~~~(2)~(3)~~(4)~~~(5)~~~~~~~(6)~~~~(7)~~(8)~~(9)
005.36| i-meas 57610 132.73 87 0.90 0 51
005.44| i-meas 58929 263.70 59 3.87 0 59
005.52| i-meas 59546 123.37 144 5.83 0 50
005.61| i-meas 60729 236.53 53 4.56 0 55
005.69| i-meas 61331 120.25 93 3.16 0 50
The columns are as follows:
Column 1: Minutes since the start of the benchmark.
Column 2: "i" denotes interval based and "p"
denotes phase-based stats. This part of the output is taken from
the Polyfill workload.
Column 3: The name of the phase as defined in the workload file.
Column 4: The number of replies up to now, across all phases.
Column 5: Replies per second.
Column 6: The mean response time in milliseconds.
Column 7: The hit ratio in percents.
Column 8: The number of the transaction errors during the current
interval or phase only.
Column 9: Number of open sockets. This number is usually very
similar to the number of pending operations.
The third section has some informational statistics about the
whole run. It also informs us about the reason for termination:
112.64| fyi: current state (2) stored
112.64| got 827131 xactions and 0 errors
112.64| shutdown reason: all 3 phases met their goals
Report Generation -- Results
After successfully running the benchmarks, another step must be
done in order to examine the results of Web Polygraph. Reports must
be generated from the log files that we kept. The new method is
with the reporter tool found in the bin directory. This is the preferred
way of generating reports. The old way involves the use of label_results
and make_report found in ReportGen directory. The new method, which
will be used here, is a one-step process, while the old method takes
two steps for the creation of a report. The new method is recommended
by the people who created Web Polygraph.
The reported tool can be executed as follows:
/usr/local/polydev/bin/reporter --report_dir /tmp/SA POLYFILL-2.srv.log
POLYFILL-2.clt.log
The --report_dir tells the reporter tool where to put the output
files of the report. If the directory does not exist, it will be created.
The output is very informative. A few images of the output will be
presented in Figure 2. For the full report, please see the Resources.
Because I am using the Squid proxy server, I find it very useful
to use the Calamaris software against the Squid log files generated
during the Web Polygraph benchmarks so that I can get more statistical
information. It is also useful to rotate the Squid log files after
each benchmark (see Figure 3).
Summary
Web Polygraph is an advanced benchmarking tool. In this article,
I have provided an introduction, but there is much more information
available from the Web site. Linux was used in this article although
the developers of the software prefer other Unix variants. The proxy
server can run on your platform of choice (even Windows). The mailing
list for Web Polygraph is a very low-volume one, so don't be
afraid to subscribe. Please read the online documentation well to
avoid posting an obvious question to the mailing list. To efficiently
run Web Polygraph requires a lot of experimentation in advance,
but it is a very useful tool that can fully be adjusted to your
needs. With a little effort, Web Polygraph can help you successfully
run simple or more complex benchmarks.
Acknowledgements
I owe many thanks to the people of the mailing list for answering
my questions. I also thank my colleagues at N.C.S.R. Demokritos
in Greece for giving me the opportunity to learn such a useful tool.
Resources
Web Polygraph -- http://www.web-polygraph.org/
Web Polygraph mailing list -- http://www.web-polygraph.org/support/list.html
Useful documents about Web Polygraph -- http://www.measurement-factory.com/support.html
PolyMix-2 information -- http://polygraph.ircache.net/Results/cacheoff-2/
Web Polygraph report (Polyfill-2) -- http://users.sch.gr/tsoukalos/sa_webpolygraph_report/
Netperf -- http://www.cup.hp.com/netperf/NetperfPage.html
Squid Web Proxy Cache -- http://www.squid-cache.org/
Calamaris -- http://cord.de/tools/squid/calamaris/Welcome.html
Mihalis Tsoukalos lives in Greece with his wife Eugenia and
works as a high school teacher. He holds a B.Sc. in Mathematics
and a M.Sc. in IT from University College London. Before teaching,
he worked as a Unix systems administrator and an Oracle DBA. By
the time you read this article, he should have started his PhD.
Mihalis can be reached at: mtsouk@iit.demokritos.gr.
| 
