 Logsurfer+
Kerry Thompson
There are many useful open source tools available. So many, it
can be hard to choose one to solve a particular problem. Logsurfer
is one of the most useful tools I know for monitoring system log
files; in fact, it is quite surprising just how much can be done
with this small C program.
For example, if you had 100 systems logging to a central server
and totaling around 100 MB of messages a day, how would you configure
a log monitoring system to alert an operator if any one of those
systems stopped sending syslog messages? An elegant solution would
be difficult, and probably involve a custom-written Perl program.
But for Logsurfer, such a function is almost trivial.
Another example is port scans. Say you log firewall packet drops
into syslog, and you want to alert someone if there is a high rate
of dropped packets from a single source IP address. Again, you would
probably be looking for a specialized program, and again, Logsurfer
can do this fairly easily.
In this article, I will examine a version of Logsurfer called
Logsurfer+, which includes a few extra features, such as the ability
to alert when certain messages stop coming in, and the ability to
specify the minimum number of messages required to trigger an alert.
Logsurfer+ is an extended version of the standard Logsurfer, and
can be downloaded from:
http://www.crypt.gen.nz/logsurfer
Downloading and Installing
To begin, download the latest version of Logsurfer+ (currently version
1.6):
wget http://www.crypt.gen.nz/logsurfer/logsurfer+-1.6.tar.gz
tar xvfz logsurfer+-1.6.tar.gz
Configure Logsurfer+ with default options, compile, and install it
in /usr/local/bin:
cd logsurfer+-1.6
./configure
make
make install
That's pretty much all there is to install. Logsurfer is a single
small C program that is very easy to install and manage. You may also
want to set up the start-up and shutdown scripts depending upon your
system. For Solaris, there is one available:
wget http://www.crypt.gen.nz/logsurfer/logsurfer_init_solaris
mv logsurfer_init_solaris /etc/init.d/logsurfer
chmod 755 /etc/init.d/logsurfer
ln -s /etc/rc3.d/S98logsurfer /etc/init.d/logsurfer
Edit /etc/init.d/logsurfer and set the options for the location of
the rules file (logsurfer.conf) and the location of the log file that
you want Logsurfer to monitor.
Rules
Logsurfer's rules simply provide instructions on what to
do when Logsurfer sees a particular line in the incoming stream
of log messages. A rule line has the following fields:
match_regex not_match_regex stop_regex not_stop_regex timeout [continue] action...
where "match_regex" is a regular expression defining which
lines match the rule, and "not_match_regex" is a regular
expression defining lines that are NOT to match the rule. If a line
matches "stop_regex", then the rule will be removed from
the run-time list of active rules; "not_stop_regex" defines
lines that will not apply to "stop_regex".
The timeout entry defines how long the rule will be active, or
you can specify 0 for no timeout. The "continue" keyword,
if present, tells Logsurfer to continue processing lines that match
this rule (by default it will stop processing a line as soon as
there is a match). The "-" character can be used to mark
unused fields.
The action field specifies what to do when a successful match
is found. The action can be one of the following:
ignore -- Ignore the line
exec -- Run a progam
pipe -- Like exec, but the log line is sent to stdin
open -- Open a new context rule
delete -- Delete an open context
report -- Run program, piping in context data
rule -- Create new rule
It's easy to see what these rules can do with a few simple
examples.
'last message repeated' - - - 0
ignore
This rule simply ignores all lines containing "last message repeated".
Note that if there were a lot of log messages going into a file, you'd
want to explicitly ignore unwanted lines at the top of the Logsurfer
configuration file to avoid the CPU load of matching them against
every rule.
' ([^ ]+) unix: ' - - - 0
pipe "/usr/local/bin/start-mail operator@example.com \"Unix Messages from $2\""
This rule sends a copy of the log line in an email to the operator
(operator@example.com). Note the use of the $2 variable to match the
field before the "unix:" string. Logsurfer assigns the whole
line to $0, $1 contains the string which matched the pattern, and
the patterns in parentheses get assigned $2 up to $9. Unfortunately,
the operator will get a separate message for each log line containing
"unix:", which could be a large number if a system were
booting. A better rule for doing this is defined in the next section
on contexts.
The start-mail script is in the contrib directory of the Logsurfer
package. It is simply a sendmail wrapper script to add a subject
and place the email alert into the sendmail queue.
Contexts
Contexts are dynamically created rules that have the ability to
group log lines together according to pattern matches. Contexts
have the ability to group lines together even when there are unrelated
log entries interspersed between them. For example, you would use
a context to group together all messages relating to the ntp daemon
from a certain host. All contexts are created by rules using the
"open" rule action. The syntax for creating a context
from a rule is to use the "open" action followed by the
context definition:
match_regex not_match_regex line_limit timeout_abs timeout_rel [min_lines] action
where:
match_regex -- Lines to match
not_match_regex -- Lines to ignore
line_limit -- Maximum number of lines to match
timeout_abs -- Absolute timeout, from first matching
line
timeout_rel -- Relative timeout, from last matching
line
min_lines -- Optional minimum number of lines to match
action:
ignore -- Ignore the line
exec -- Run a progam
pipe -- Like exec, but the log line is sent to stdin
report -- Run program, piping in all log lines collected
A better "unix:" system message rule could be defined
as follows:
' ([^ ]+) unix: ' - - - 0
open " $2 unix: " - 500 1200 600
report "/usr/local/bin/start-mail operator@example.com \
\"Unix Messages from $2\"" " $2 unix: "
This rule will look for lines containing "unix:" with a
host name in the preceding field. When a line matches, it will create
a new context, which will collect further matching lines from the
same host with a maximum number of lines of 500, an absolute timeout
of 1200 seconds (20 minutes), and a relative timeout of 600 seconds
(10 minutes).
This means that it will collect all of the matching log lines
and email them to the operator if no matching line is seen within
10 minutes of the last matching line, or within 20 minutes of the
first matching line. We put a line limit of 500 lines in just to
prevent runaway messages from exhausting our memory space.
Alerting if There Are No Messages
Let's say you have a lot of systems logging into a single
file on a central log server. One useful function you may want to
set up is to configure a cron job on each remote system to send
a "ping" syslog message every 15 minutes or so just to
let you know that the system is up and running. The cron table entry
could look like this:
0,15,30,45 * * * * /bin/logger -t syslog_ping -p local7.info `/bin/uptime`
In this example, I also send the output from uptime, which logs the
number of users logged in and a brief summary of the system load.
A Logsurfer rule to detect when a host stops logging looks like this:
' ([^ ]+) syslog_ping:' - - - 0
open " $2 syslog_ping:" - - - 2100
exec "/usr/local/bin/start-mail operator@example.com \"Alert: syslog pings from $2 have stopped\""
The first line defines a rule that will match the "syslog_ping"
log message from hosts. The expression "([^ ]+)" simply
matches the host name from the log message and places it into $2.
The action from the rule creates a new context that monitors all future
syslog messages from the host and sends an email alert if there are
no messages seen in a span of 35 minutes. This means we are allowed
to drop one syslog message accidentally but, if we drop two, then
the monitored system may be having problems. Note that you need Logsurfer+
v1.6 to perform this properly -- the standard Logsurfer v1.5b
would store all matching lines in memory, possibly causing the system
to run out of memory over time.
The nice thing here is that we don't have to define a rule
for every host that is logging. The first rule will automatically
start up a monitoring context as soon as it sees a syslog_ping message
from a host that it is not already monitoring. In just a few lines
of configuration, we have created a relatively complex function.
Detecting Port Scans
Another feature of Logsurfer+ allows you to configure a minimum
number of lines to match in a context before the action is triggered.
If the minimum is not reached and the context times out, then no
action is performed. This can be used to alert when a serious security
event occurs, such as when a firewall system receives a large number
of attempted connections. You wouldn't really want to see alerts
if there were just 10 denied attempts, but you would want an alert
if there were 100 or more in rapid succession.
The following rule alerts the administrator if a Cisco router
logs a high number of denied packets from a single IP address:
' ([^ ]*) %SEC-6-IPACCESSLOGP: .* \
([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\([0-9]+\) ->' - - - 0
open " $2 %SEC-6-IPACCESSLOGP: .* $3\\([0-9]+\\) ->" - 10000 \
600 300 100
report "/usr/local/bin/start-mail operator@example.com
\"Port scan from $2\"" " $2 %SEC-6-IPACCESSLOGP: \
.* $3\\([0-9]+\\) ->"
This is a complex rule. Basically, for every denied packet, it
opens a context that begins tracking further denials from the same
source. If there are more than 100 denied packets (up to a limit
of 10,000) with an absolute time span of 600 seconds and a relative
timeout of 300 seconds, then an alert message containing the logged
messages, is emailed. Of course, this can be tuned to meet your
needs and can easily monitor other types of firewalls, such as iptables,
PIX, and CheckPoint Firewall-1.
Conclusion
Logsurfer is a simple, small, and efficient program that can monitor
log files in real time and intelligently send alerts when anomalies
occur. Unlike other monitoring systems, such as swatch, Logsurfer
can be tweaked and tuned to send only single alerts containing all
of the relevant information and not to send a deluge of emails to
the operator. Logsurfer can be used to detect faults and security
events before they develop into serious problems.
Here are some additional tips:
- Be careful how you process syslog messages. It is sometimes
possible for an attacker to influence the contents of syslog messages,
possibly inserting control and escape character sequences.
- It's a good idea to install and run Logsurfer under a
non-root userid, such as "logsurfer".
- Remember to kill -HUP the logsurfer process whenever
you rollover your syslog files, just as you need to kill -HUP
your syslog daemon.
- Of course, your alerts don't need to be emailed. You can
just as easily use a pager interface to send alerts.
- Logsurfer can be used to monitor any log file -- not just
syslog files. It is also often used to monitor Web server log
files and application logs.
Resources
Kerry Thompson's Logsurfer+ page, for Logsurfer+ download
and information -- http://www.crypt.gen.nz/logsurfer
emf's Logsurfer configuration page, containing many useful
Logsurfer configurations -- http://www.obfuscation.org/emf/logsurfer.html
DFN-CERT official Logsurfer site -- http://www.cert.dfn.de/eng/logsurf/
Kerry Thompson is a CISSP certified security consultant with
more than 12 years experience in systems administration and IT security.
He is based in Auckland, New Zealand and can be contacted at: kerry@crypt.gen.nz.
| 
