|
Monitoring
LDAP Performance| Level | Description |
| -1 | Enable all debugging |
| 0 | No debugging |
| 1 | Trace function calls |
| 2 | Debug packet handling |
| 4 | Heavy trace debugging |
| 8 | Connection management |
| 16 | Print out packets sent and received |
| 32 | Search filter processing |
| 64 | Configuration file processing |
| 128 | Access control list processing |
| 256 | Stats log connections/operations/results |
| 512 | Stats log entries sent |
| 1024 | Print communication with shell back ends |
| 2048 | 2048 Print entry parsing debugging |
Each log level is additive and can be configured through the slapd.conf "loglevel" directive, or passed as an argument to slapd's "-d" option. The following example shows how to log detailed information on ACL and search filter processing:
$ slapd -4 -f /etc/slapd.conf -u openldap -g openldap \ -h "ldap://ldap.daemons.net ldaps://ldap.daemons.net" -d 160OpenLDAP will log all information by default to syslog's LOCAL4 facility. If you want to use a different facility, you can pass the facility name to slapd's "-l" option.
Monitoring Operation Times
When LDAP clients and servers are separated by routers and firewalls, sporadic network behavior (e.g., lost TCP segments or bad CRCs) can cause unexpected application behavior. To help measure the latency between an LDAP client and server, we developed ldap-ping.pl (Listing 1). Ldap-ping.pl is written in Perl and relies on the Time::HiRes, Getopt::Std, Net::LDAP, and Net::LDAPS modules.
Ldap-ping.pl works by opening a TCP connection to a directory server, issuing an anonymous bind, searching the RootDSE, and unbinding from the server. Each of these operations is measured with the Perl high-resolution timers and displayed in "ping" style format:
$ ./ldap-ping.pl -s ldap.daemons.net -p 389 -d 10 Querying LDAP server ldap.daemons.net:389 every 10 seconds (Ctrl-C to stop): Fri Nov 12 16:42:14 2004: new=0.025s, = bind=0.008s, \ search=0.067s, unbind=0.003s [local port=50377] [Normal Delay] Fri Nov 12 16:42:25 2004: new=0.011s, = bind=0.001s, \ search=0.015s, unbind=0.001s [local port=50378] [Normal Delay] Fri Nov 12 16:42:35 2004: new=0.010s, = bind=0.002s, \ search=0.015s, unbind=0.001s [local port=50379] [Normal Delay] Fri Nov 12 16:42:45 2004: new=0.009s, = bind=0.002s, \ search=0.015s, unbind=0.001s [local port=50380] [Normal Delay]The ldap-ping.pl script accepts three arguments: the "-s" option indicates the server to connect to, the "-p" option specifies which TCP port the directory server is listening on, and the "-d" option allows the admin to specify the delay between probes. If the pfiles binary is present, the script will also print the local port number.
Monitoring Performance
The OpenLDAP server can be configured to provide real-time performance statistics through the monitor branch. The currently available statistics include: bytes sent, entries returned to clients, total connections to the server, current active connections, read and write waiters, and individual operation (e.g., read, search, modify) breakdowns. The following example shows the slapd.conf directives required to set up the monitor branch and limit read access to IP address 192.168.1.8:
database monitor
access to dn.subtree="cn=Monitor"
by peername.regex="^IP=192\.168\.\1\.2\.*" read
by * none
Once the monitor branch is configured, we can view all of the available
statistics with the "ldapsearch" utility:
$ ldapsearch -x -b "cn=monitor" \ -H ldaps://ldap.daemons.net objectclass=*We can also retrieve individual statistics by adjusting the search base:
$ ldapsearch -LLL -x -b \ "cn=Current,cn=Connections,cn=Monitor" \ -H ldaps://ldap.daemons.net objectclass=* dn: cn=Current,cn=Connections,cn=Monitor objectClass: top objectClass: monitor objectClass: extensibleObject cn: Current description: 46Collecting Performance Data
The ldapsearch utility allows us to grab a point in time snapshot of the performance metrics, but how do we get historical utilization data? To address this problem, I developed ldap-gather.pl, which is included in Listing 2. The ldap-gather.pl script collects statistics from the monitor branch and writes them to the directory passed as an argument:
$ ldap-gather.pl -s ldap.daemons.net -p 389 \ -d /usr/local/orca/var/orca/ldapallator/ldap.daemons.net:389The initial invocation of ldap-gather.pl will produce a text file with a header describing the data and one line of actual data:
TIMESTAMP TOTAL_CONNECTIONS BYTES_SENT COMPLETED_OPERATIONS \ REFERRALS_SENT ENTRIES_SENT BIND_OPERATIONS UNBIND_OPERATIONS \ ADD_OPERATIONS DELETE_OPERATIONS MODIFY_OPERATIONS \ COMPARE_OPERATIONS SEARCH_OPERATIONS 1100656501 118 649271 165 0 4620 24 24 0 0 0 0 117The ldap-gather.pl script will create a new data file if one doesn't exist, and append new data if the file exists. The file name includes the word "ldapallator" and a date timestamp (e.g., filename-YYYY-MO-DD-INDEX):
$ ls -la | tail -1 -rw-r--r-- 1 orca other 6424 Nov 29 17:25 ldapallator-2004-10-29-000To automate capturing data at various intervals, we can set up a cron job to run ldap-gather.pl:
5,15,25,35,45,55 * * * * sh -c "/usr/local/etc/ldap-gather.pl -s \ ldap.daemons.net -p 389 -d \ /usr/local/orca/var/orca/ldapallator/ldap.daemons.net:389"This will cause ldap-gather.pl to grab performance data every 10 minutes. Once the data has been captured, we can use Orca to generate graphical performance reports.
Graphing LDAP Performance Data
The Orca package contains a set of Perl scripts and configuration files for graphing arbitrary data. Orca uses RRD for consolidating and storing data and is configured with a single configuration file. Orca utilizes the typical "configure," "make", and "make install" process to build the packages. The Orca Perl scripts use the Data::Dumper, Digest::MD5, Math::IntervalSearch, RRDs, and Storable modules. Orca also includes the "make modules_install" option to integrate these with an existing Perl installation. Once the installation completes, "orca" can be invoked to ensure the build process completed successfully:
$ /usr/local/orca/bin/orca /usr/local/orca/bin/orca: no configuration file specified usage: /usr/local/orca/bin/orca [options] configuration_file Options: -daemon Run Orca in daemon mode -gifs Output GIFs instead of PNGs -logfile filename Output all messages -no-html Update RRD files and images but not HTML files -no-images Update RRD files but not image and HTML files -once Run only once and do not continue to monitor input files -verbose Verbose; list multiple times for increased verbosityOrca understands the first unique command-line option, for example, -d for -daemon.
If the Perl interpreter cannot find one of the required modules, you will see a variety of errors on the console, and the process will exit. Once the Perl interpreter is happy, you can create an Orca configuration file and start putting the ldap-gather.pl data to good use.
The Orca configuration file contains the directives required to locate and graph data. The configuration file contains three main sections. The first section defines several variables (e.g., base_dir, rrd_dir, html_dir) that are used to control where the rrd and image files are stored. This section also contains several variables to define the formatting of the Web pages Orca generates.
The second section contains a series of "group" entries, which define the data to graph. A sample group entry to match the files collected with ldap-gather.pl is shown here:
group ldapallator {
find_files /usr/local/orca/var/orca/ldapallator/(.*)/ \
(?:ldapallator)-\d{4}-\d{2}-\d{2}(?:-\d{3,})? \
(?:\.(?:Z|gz|bz2))?
column_description first_line
date_source column_name TIMESTAMP
interval 600
filename_compare sub {
my ($ay, $am, $ad) = $a =~ \
/-(\d{4})-(\d\d)-(\d\d)/;
my ($by, $bm, $bd) = $b =~ \
/-(\d{4})-(\d\d)-(\d\d)/;
if (my $c = (( $ay <=> $by) ||
( $am <=> $bm) ||
(($ad >> 3) <=> ($bd >> 3)))) {
return 2*$c;
}
$ad <=> $bd;
}
}
In this example, the "find_files" keyword tells Orca which files to
use as input (the files are matched with a regular expression). The
"interval" keyword defines the number of seconds between updates for
data files in this group, and "column_description" describes where
the column descriptions are located. The "column_description" value
"first_line" indicates that the column descriptions will be located
on the first line of each file.
The third section contains a set of plots to indicate which items to graph. The following example shows which directives are required to graph the number of connections to a directory server:
plot {
title %g Total Connections
source ldapallator
data TOTAL_CONNECTIONS
data_type derive
line_type line2
legend Connections
y_legend Connections
data_min 0
}
The "plot" keyword defines a new graph, which contains the title specified
in "title." The "source" keyword indicates the "group" entry to source
for this graph. Each graph uses the "data" keyword to reference specific
columns in the data by name (e.g., TOTAL_CONNECTIONS is a column in
each datafile ldap-gather.pl collects.) The "data_type" keyword specifies
whether data will be graphed as an absolute value, a derivative of
the previous value, or a counter. The X axis values of a graph can
be controlled with the "data_min" and "data_max" values. And finally,
the legends allow you to place human-readable descriptions in the
PNG files that Orca creates.
Orca can be invoked to process data once and exit, or it can be set up to run as a daemon and continuously check for new data. The following example runs Orca once to process all of the data since the last invocation:
$ /bin/sh -c "cd /usr/local/orca && /usr/local/orca/bin/orca \ -once /usr/local/orca/lib/ldapallator.cfg"If Orca runs successfully, the HTML and PNG files should be available in "html_dir." If you choose not to run Orca as a daemon, you will need to add a cron job to process the files at various intervals:
0 0 * * * /bin/sh -c "cd /usr/local/orca && /usr/local/orca/bin/orca -once \
/usr/local/orca/lib/ldapallator.cfg" > /dev/null 2>&1
This will run Orca every day at midnight and process the data from
the previous day. Listing 3 contains a complete Orca configuration
file (http://www.sysadminmag.com). Figures 1, 2, and 3 contain
the Orca graphs for total connections, total operations, and the number
of entries returned.
Conclusion
This article provided an overview of several techniques to monitor an LDAP server, and provided an introduction to the ORCA data plotter. We used OpenLDAP version 2.2.X to test each example, but the techniques are also applicable to the commercial directory servers. For additional information on OpenLDAP, and the numerous ways ORCA can be used to graph arbitrary data, please see the references.
Acknowledgements
Ryan would like to thank Clay McClure for the original work on ldap-ping.pl. Ryan would also like to thank Quanah Gibson-Mount, and the ORCA and OpenLDAP team members for their awesome contributions!
References
Orca Web site -- http://www.orcaware.com
OpenLDAP Web site -- http://www.openldap.org
Latest Versions of ldap-gather.pl and ldap-ping.pl -- http://www.daemons.net/code
Ryan Matteson works as a systems engineer for Earthlink, and specializes in Web technologies, Storage Area Networks, high-availability systems, and the Linux and Solaris operating systems. Questions and comments about this article can be addressed to matty@daemons.net.