 Monitoring
Log Entries with Logbot
Robert Hayes
Keeping systems secure requires not only copious logging, but
also a regular review of those logs. The more frequently you review
them, the better, and the best situation is real-time alerting based
on log criteria that you define. This technology has been around
for some time through log-tailing and alerting programs (such as
Swatch) and, more recently, through syslog-ng.
Using Swatch to monitor your logs and send a page or email when
critical events occur is nothing new. But, what about monitoring
more mundane events? You may want to see when telecommuters are
using your VPN, or when your spam filter is rejecting email, or
even when someone hits your Web site using a non-standard browser.
Getting paged every time one of these events occurred would quickly
become tiresome, and receiving emails would clog your inbox. As
sys admins, we all have the ability to tail log files and grep
for certain strings, but having a simple cross-platform interface
to view log entries as they occur can help keep you and the users
you support aware of what's happening on your servers.
Logbot is a simple Perl script designed to take output from log-monitoring
software and send it to a conference room on a Jabber server. Because
myriad Jabber clients are available across most major platforms,
Logbot allows your Windows, Mac, and Unix users to view relevant
log entries as they occur. Logbot is perfect for monitoring medium-
to high-volume log entries that would otherwise incur a heavy load
if sent individually as pages or emails. If you're not logging
chat room dialogue, Logbot output is completely ephemeral: you can
run it constantly without worrying about using up valuable drive
space.
The Jabberd Server
Logbot has been written for and tested against the jabberd server
for *nix systems, although it should work with any Jabber server
that supports Multi-User Conferencing (MUC), the protocol that adds
"chat rooms" to your Jabber server. The jabberd project
lives at:
http://jabberd.jabberstudio.org
There you will find a wealth of documentation, the source code itself,
and links to jabberd mailing lists. I will not attempt to cover the
installation and configuration of a jabberd server. Doing so would
consume the entire article, and excellent documentation already exists
on the jabberd Web site. At the time of this writing, the latest stable
version of jabberd that supported multi-user conferencing (muc) was
jabberd 1.4.2 and mu-conference 0.5.2.
Multi-user conferencing does not exist natively within the jabberd
server, rather, it relies on a separate piece of software called
mu-conference, found at:
http://mu-conference.jabberstudio.org
This software is meant to supersede previous group chat protocols,
while simultaneously providing backward compatibility with those protocols.
Because Jabber is a relatively new protocol, it is under constant
development and improvement, and there have been multiple iterations
of this group chat concept.
Conference rooms can easily be created using a Jabber client that
supports the MUC protocol. But, because the name of the chat room
will be hard-coded into the Logbot script, be sure you create persistent
rooms that will not be destroyed if you need to restart the jabberd
server.
The mu-conference software comes with a set of Perl scripts that
will walk you through room creation and create a .xml configuration
file for your conference room. Because you'll be sending syslog
output to this room, you'll probably want to control who has
access to it. If you create the conference room using the mu-conference
scripts, you can make it a "members only" room and specify
which JIDs (Jabber IDs) are allowed to access the room. Adding or
removing users later is as simple as editing the .xml config file.
You'll probably also want to make the room "private".
A private room is simply a room that is not "public" and
will not be displayed in the Jabber browser unless searched for
by a user who is already logged into the room.
If you don't want your conference room service available
to other servers or the Internet at large (a good idea for a secure
server), be sure the conference host is set to conference.localhost
rather than to a fully qualified domain name. Also, because good
security involves good logging, you may want to log room activity.
However, if you're watching particularly voluminous logs, there's
no point in filling up your hard drives with redundant log entries.
For more information on the mu-conference protocol itself, see:
http://www.jabber.org/jeps/jep-0045.html
Once your jabberd server is up and running with mu-conference support,
be sure that Logbot has a JID on your server. The easiest way to do
this is by allowing registration on the server and creating the JID
with a Jabber client. Create a new user named "Logbot" (or
something more creative) with an appropriate password. Once that's
done, be sure that the Logbot user can log into the server and into
the conference room through a Jabber client, before you try it with
the script. Once you can log in and chat using a Jabber client, edit
the script and try it from the command line. There should be enough
debugging output to point you in the right direction if problems arise.
The logbot.pl Script
The Logbot script is written in Perl and makes heavy use of the
Net::Jabber and related modules, which are all available from cpan.org
(see the resources section). It also makes use of some of the example
code distributed with this module. Hooray for open source!
Net::Jabber requires XML::Stream, and if you want to use SSL connections,
be sure you have IO::Socket::SSL and Net::SSLeay. These can be painlessly
installed by running perl -MCPAN -e shell from a root prompt.
Using SSL is optional but recommended. Arguably, it makes little
difference in the overall security of your site, since system log
messages will be flying around your network in UDP plain text if
you are running a centralized syslog-ng server. But, upcoming versions
of syslog-ng will include encryption functions, and keeping plain
text off the wire is always a good idea.
Syslog-ng runs as root by default, so when it calls the logbot.pl
script, it will run as root. It will also continue running, reading
input from STDIN until syslog-ng is stopped or restarted. Syslog-ng
can be chrooted and run as a non-privileged user, but configuring
this is left as an exercise for the reader. Instead of overhauling
my syslog-ng implementation, I dropped the effective permissions
in the script as soon as possible. In this way, although the script
is called by root, it runs with the effective permissions of the
user "nobody". Check your /etc/passwd and /etc/group files
for a non-privileged account name. My "nobody" user has
a UID and GID of 99; your mileage may vary. (More information on
syslog-ng is included later.)
Overall, the script is pretty simple. It does not define any Jabber
callbacks, and as such, will not respond to messages or presence
information sent to it; all it does is take input and send it to
a conference room.
All variables are defined up front and must be edited for your
server configuration. The script is written for use with syslog-ng
or the traditional syslog facility combined with Swatch; comment
or uncomment the Swatch and syslog sections appropriately.
If you do not want to run mu-conference on your jabberd server,
but do want log alerts to be sent to a single recipient (such as
yourself), you can remove the MUCJoin section and replace:
$Connection->MessageSend(to=>"$room\@$mucserver",
with:
$Connection->MessageSend(to=>"$recipient\@$server",
As mentioned previously, the script is started by the syslog-ng user
when used with syslog-ng, and it stays open reading input from STDIN
while connected to the server. When launched by the Swatch user, the
script is called with the matching log entry passed as a command-line
argument, @ARGV. In this case, the script runs and the Logbot user
connects to the server and passes the matching log entry, then quickly
disconnects and exits. The next matching log entry will fire it up
again. If you're using this implementation, you'll notice
that it makes a bit of noise in the chat room every time the Logbot
user connects and disconnects. This is one good reason to upgrade
to syslog-ng.
Using Logbot with Syslog and Swatch
Logbot relies on a separate mechanism to monitor log files, watch
for specific content, and take appropriate action. Swatch (short
for simple watcher) does exactly this. It has been around for ten
years -- long enough to be in every admin's toolbox --
and is invaluable when it comes to keeping an eye on your systems.
Swatch is freely available and has a broad user base. The Swatch
project home page is now:
http://swatch.sourceforge.net
There you can find the most recent source code. It installs like other
Perl software with a simple:
perl Makefile.PL
make
make test
make install
make realclean
It relies on four external Perl modules, which are all available from
cpan.org.
Date::Calc
Date::Parse
File::Tail
Time::HiRes
Swatch relies on a .swatchrc configuration file and, by default, looks
for it in the home directory of the user running Swatch. Any user
can run Swatch, provided that the user has read access to the log
files being tailed. To avoid running Swatch as root, create a dedicated
"loggers" group with a low-privileged user as its member
and give read-only access to the loggers group on the log files you
want to monitor.
Swatch matches specific content using regular expressions in the
.swatchrc file; when a log entry matches your regular expression,
a specific action is taken. The action can be one of the following:
echo the matched text to a terminal, send an email, send a page,
or execute an arbitrary command. When used with the "exec"
action, logbot.pl can take the matched text as an argument array
and send it to the Jabber conference room.
A simple sample .swatchrc file that watches for the string "authentication
failure" and sends the corresponding log entry to Logbot would
look like this:
# Sample .swatchrc file
watchfor /authentication failure/
exec="/usr/bin/perl /path/to/logbot.pl $*"
The $* tells Swatch to take the entire log entry (the whole line)
and send it to the Logbot script. A $4 would send only the fourth
word in the log entry, a $1 the first word, etc.
Swatch has some other special options that may make its use preferable
over syslog-ng (discussed later). Two of the most notable are the
"throttle" and "when" options. The throttle
option takes a time unit (hours:minutes:seconds) as an argument,
and will make sure that the defined action does not execute more
than once during that time period. For example, a throttle option
of 01:00 will ensure that multiple instances of the log entry will
not be acted on if they occur within a minute of the first instance.
If you're monitoring something with a lot of activity (e.g.,
Web server of firewall logs), consider using the throttle option
to lighten the load on your syslog server, network, and Jabber server.
The "when" option will specify when a certain action
should be executed, and when it shouldn't; the argument is
passed as day_of_week:hour_of_day. The "when" option could
be helpful if, for example, you're monitoring logs in a 24x7
NOC environment, where you want to see only when your VPN is in
use outside of regular working hours, or where you want to watch
Web server logs during a specific maintenance window. See the Swatch
man page for other options and important information on the care
and feeding of Swatch.
Using Logbot with Syslog-ng
There is no doubt that Swatch is a valuable tool for admins who
want real-time alerts on log activity. But because it is separate
from the syslog facility itself, there is slight additional overhead
incurred in maintaining it. Syslog-ng is the "new generation"
of syslog and has an impressive list of powerful new features, including
built-in message filtering. Syslog-ng makes it a snap to set up
a centralized logging server with forwarded messages from other
servers easily split into different destination files, or even databases,
based on their source or content. Running logbot.pl on a centralized
log server allows you to watch for suspicious activity on all of
your servers at once, without having to configure Swatch, Logbot,
or syslog-ng on each separate machine. Plus, it will allow you to
harden your jabberd server even further by only allowing connections
from fewer IP addresses.
Syslog-ng is maintained by Bazsi Scheidler and can be obtained
at:
http://www.balabit.hu/en/downloads/syslog-ng
There is an excellent FAQ at:
http://www.campin.net/syslog-ng/faq.html
and an active mailing list. Because syslog-ng is still under active
development, subscribing to the mailing list is recommended for bug-fix
information and announcements. See the resources section for more
syslog-ng links.
The software is available for download as a source tarball, or
as a pre-compiled binary for Debian Linux users. The syslog-ng.deb
package replaces the traditional syslog package on Debian and comes
with its own syslog-ng.conf file tailored for Debian and ready for
use. If you're installing from source, don't forget to
grab libol, a required library used by syslog-ng.
The heart of syslog-ng is its configuration file, by default /etc/syslog-ng/syslog-ng.conf.
Configuring a syslog-ng.conf file from scratch can be daunting,
and the flexibility of this package makes for a complex configuration
file. But fear not -- example configuration files are distributed
with the source code, and once you get the hang of the syntax, expanding
on these files and seasoning them to suit your site requirements
is easy.
As mentioned earlier, the logbot.pl script relies on an external
mechanism to filter your logs and pass the output to Logbot's
STDIN. This is another place where the content-filtering in syslog-ng
shines. Syslog-ng allows you to define different sources for log
entries like unix socket on the host, entries from remote hosts
via UDP or TCP, a named pipe, or even a flat text file. You can
also define specific destinations for the logs entries, such as
a text file, an arbitrary UDP or TCP port on another host (e.g.,
another logging host), a named pipe, unix socket, or an external
program, which is where Logbot fits in.
A snippet of syslog-ng.conf might look like this:
source s_sys { unix-dgram ("/dev/log"); internal(); };
destination d_jabber { program( "/usr/bin/perl /path/to/logbot.pl" ); };
filter f_jabber { match( "authentication failure" ); };
log { source(s_net); filter(f_jabber); destination(d_jabber); };
Note that these lines are simply what would be required for sending
log output to the Logbot script; they do not represent a fully configured,
or even fully functional, syslog-ng.conf file.
The source line specifies that the regular /dev/log unix socket
be used as the source, as well as any messages internal to the syslog-ng
program itself. The destination line specifies the logbot.pl script
as the log entry destination. The log directive puts all of the
pieces together, telling syslog-ng that any log entry coming from
the /dev/log socket containing the words "authentication failure"
should be sent to the logbot.pl script. You can specify sources
and destinations all day. However, until they are combined with
a log statement, the log entries will not go anywhere unless they
are handled by different log and destination directives.
One of the things that makes Logbot possible is the ability to
specify multiple destinations and log directives for a single log
entry. For example, I use the following source and destination directives
on my centralized log server:
source s_net { udp(); };
destination d_net { file("/var/log/syslog-ng/$HOST/$PROGRAM"); };
log { source(s_net); destination(d_net); }; # This is the last line of the file
This serves as a catchall bucket that allows other admins in my office
to configure their hosts to log to our central server merely by pointing
the host at the log server's IP address. With some network hardware,
this is the only variable you can define. Once that host is configured,
the syslog-ng server will automatically create the directory and corresponding
log file, based on the hostname and name of the program that is creating
the logs. Having the Jabber log directive listed above this catchall
allows me to watch for important log entries by delivering them to
the Logbot script, while still delivering them to their "regular"
destination.
Conclusion
With IM servers finding a niche in corporate and enterprise environments,
chat clients are becoming a standard part of every user's workspace.
Multi-user conferencing in chat rooms allows administrators in far-flung
global enterprises to exchange information on systems and networks
quickly and easily. See Figure 1. Selectively forwarding log events
to an intuitive interface that runs across many platforms provides
the benefit of allowing multiple users -- whether they are administrators,
help-desk staff, or executive-level managers -- to see what's
happening on your servers in real-time, without giving out shell
access, file read permissions, or configuring extra ssh clients.
When monitoring non-critical, frequently occurring log events means
receiving equally frequent email or pager alerts, using Logbot allows
a chatty server to forward these log entries without causing your
pager to vibrate itself to pieces or your inbox to reach critical
mass.
Resources
Jabberd server -- http://jabberd.jabberstudio.org
MU-Conference -- http://mu-conference.jabberstudio.org
MUC protocol -- http://www.jabber.org/jeps/jep-0045.html
Net::Jabber module -- http://search.cpan.org/~reatmon/Net-Jabber-1.29
Syslog-ng home page -- http://www.balabit.hu/en/downloads/syslog-ng
Syslog-ng FAQ -- http://www.campin.net/syslog-ng/faq.html
Sample chapter from "Building Secure Servers with Linux"
-- http://www.oreilly.com/catalog/bssrvrlnx/chapter/ch10.pdf
Syslog-ng mailing list -- https://lists.balabit.hu/mailman/listinfo/syslog-ng
Robert Hayes is an RHCE and has a Bachelor's of Science
in Zoology from the University of Texas at Austin. He lives in Austin
and works as a Security Technical Analyst for the state of Texas.
He can be reached at: robert.hayes@mrhaha.net.
| 
