 FireWatch
-- A Firewall Monitoring Tool
Will Schroeder
When I switched from OpenBSD to Red Hat 7.2 on my machines, I
wanted an effective and simple way to see what was bouncing off
my servers. I looked for existing iptables-logging software and
found simple "write to a text file" loggers, software
that required a kernel recompile to work, or PHP programs. I like
PHP, but there have been some security issues with it recently.
I really didn't want to mess around with the kernel, and since
I'm a Perl diehard, I wanted something in Perl. Because I didn't
find what I wanted, and because I thought I could learn something
in the process, I wrote FireWatch, available for download at:
http://www.sysadminmag.com/code/
FireWatch provides administrators of publicly accessible machines
with the ability to see what is being rejected by the firewall of
each host. The application currently supports two interfaces configured
in a public and a private network configuration, and will log data
in and out on both. FireWatch will also work on ppp connections.
I wrote FireWatch on my Red Hat 7.2 servers using Perl, MySQL,
and Embedded Perl for the Web interface. My main goals during the
development of FireWatch were to make it as simple, light, easy
to administer, and secure as possible, and to lay the groundwork
for scalability. I also kept the principle of least privileges in
mind, so the FireWatch daemon runs as a non-privileged user and
the database users are limited to just inserting or selecting. Also,
all scripts that accept input have metacharacter filtering and environment
limits to prevent malicious users from manipulating the database.
How FireWatch Works
FireWatch serves two major functions -- data storage and data
analysis. All logged data is stored in a MySQL database. Data storage
in a relational database, as opposed to a flat file, has several
advantages that outweigh the additional complexity. A relational
database allows for easier analysis of data trends over time, simplified
reduction of complex data, the ability to save data remotely without
relying on NFS mounts, and the ability to analyze information from
multiple hosts in a single query. For example, if I had stored the
logged data into a flat file, it could be rather complicated to
find the top ten active ports on a host. Some simple SQL make this
kind of analysis easy. The code below does the math and creates
a temporary table of port information:
$sth = $dbh -> prepare ("create temporary table $temp_table select \
count(DST_Port) as Event_Count, Ports.Service_Name,
DST_Port,'${HostChoice}_${TableName}_Week$req->{weeknum}'.Protocol
from '${HostChoice}_${TableName}_Week$req->{weeknum}',Ports
where '${HostChoice}_${TableName}_Week$req->{weeknum}'.DST_Port = \
Ports.Port_Num and
'${HostChoice}_${TableName}_Week$req->{weeknum}'.Protocol = \
Ports.Protocol and Lan = '$LanChoice'
group by '${HostChoice}_${TableName}_Week$req->{weeknum}'.DST_Port");
A simple select displays the summarized information of top ten most
active ports:
$sth = $dbh -> prepare (" select * from $temp_table order by \
Event_Count desc limit 10");
The database is scripted to be self-maintaining via a roll script
that creates new tables for each host, each new week.
The second function of FireWatch is to provide report data in
two formats -- plain-text emails, and summary information provided
via dynamically generated HTML. The plain-text emails are generated
every night at midnight when cron runs the SendDailyReport.pl and
SendTopDomains.pl scripts. The SendDailyReport script queries each
host's table for the previous day for inbound and outbound
data on the external interface. The output is in plain text and
a short snippet is displayed below. The columns are date, number
of logged events, protocol, IP, source port, and destination port:
2003-2-22 home
Feb 22 1 Inbnd UDP Drp 4.47.98.83 1028 137
Feb 22 1 Inbnd UDP Drp 4.47.111.43 1027 137
Feb 22 1 Inbnd UDP Drp 12.15.238.75 1028 137
Feb 22 1 Inbnd UDP Drp 12.165.118.5 52421 137
Feb 22 1 Inbnd UDP Drp 24.35.65.37 1026 137
Feb 22 1 Inbnd UDP Drp 24.78.144.206 1032 137
Feb 22 2 Inbnd TCP Drp 24.90.93.37 2043 445
Feb 22 1 Inbnd TCP Drp 24.104.42.153 22773 1080
The second reporting script sends a text email of the top five domains.
The output is displayed below. The columns are event count, IP, hostname,
source port, destination port, and protocol:
Sent at 015 for home
12 64.81.148.234 dsl081-148-234.chi1.dsl.speakeasy.net. 1026 137 UDP
12 64.81.233.154 dsl081-233-154.lax1.dsl.speakeasy.net. 1044 137 UDP
12 213.96.115.211 213-96-115-211.uc.nombres.ttd.es. 4100 111 TCP
6 64.212.95.196 Failed to Resolve 4848 1433 TCP
5 203.73.199.48 Failed to Resolve 3105 20006 UDP
In addition to the email reporting, FireWatch has a Web interface
for monitoring dropped packets. I use Perl DBI to talk to the database
and Embedded Perl to dynamically generate the displayed HTML. The
Web display is useful for looking at real-time information without
querying the database directly. For example, to see all of the rejected
TCP traffic, you would click on TCP. The display is shown in Figure
1.
Queries may be added to the display by adding some code to head.html
and creating the new HTML page to display the queried data. The
different reporting methods are independent, so one or both may
be used.
The FireWatch application consists of modules that have specific
functions:
- Web Server -- All of the files necessary to display the
collected data in a browser window.
- Database Host -- All of the files necessary to install
and work with the FireWatch database.
- Firewall Host -- The FireWatch script and utility scripts
to write data to the database.
To use FireWatch the modules that relate to a particular machine's
functions are installed on each box. Since FireWatch is modular,
you can have any number of network configurations. My configuration
currently looks like Figure 2.
Firewatch uses syslog and a FIFO in /var/run to log dropped packets.
To log dropped packets add a rule to your file with -j LOG
to your firewall like the example below.
iptables -A INPUT -i eth0 -s $class_b -j LOG --log-prefix \
"Inbnd Spoofed B External"
iptables -A INPUT -i eth0 -s $class_b -j DROP
I also watch the internal interface like this:
iptables -A INPUT -i eth1 -j LOG --log-prefix "Inbnd Prot-X \
Drp Internal"
iptables -A INPUT -i eth1 -j DROP
Next, add the line in syslog.conf kern.info |/var/run/firefifo, then
create the FIFO so messages have somewhere to go, like this mknod
/var/run/firefifo p.
How this works is fairly simple. Line one appends logging to the
default input chain with the prefix of "Inbnd Spoofed B External"
and then line two does the actual work of dropping the packet. When
a packet that matches the criteria above arrives, a message is sent
to the FIFO from syslog. A full line actually looks like this:
Feb 12 15:53:34 home kernel: Inbnd UDP Drp External IN=eth0 OUT=
MAC=08:00:20:e5:88:f8:00:02:3b:00:4e:b4:08:00 SRC=24.232.38.109 \
DST=64.81.147.225
LEN=78 TOS=0x00 PREC=0x00 TTL=109 ID=45922 PROTO=UDP SPT=1029 \
DPT=137 LEN=58
Next, the FireWatch script opens the FIFO for reading:
while (1) {
open(FIFO, "</var/run/firefifo") || die "Where is the fifo $!";
$data = <FIFO>;
chomp $data;
if ($data =~ m/Inbnd/) {
While the fifo is open, it matches the Inbnd or Outbnd to determine
into which table the data should be inserted. It then removes extraneous
things like "Kernel:" and items not yet inserted into the
database (like TTL) with:
.$data =~ s/TTL=\S+//g;
It is important that I do a hex substitution on all IP addresses because
MySQL does not have a built-in IP data type, and it isn't possible
to sort numerically on an IP address. Therefore, I do this:
$data =~ s/SRC=(\d+\.\d+\.\d+\.\d+)/unpack("H8", inet_aton($1))/eg;
to convert the IP.
The resultant string is split into variables. I add my own date/time
stamp, quote everything. and insert it all into the table.
($Log_month,$day,$time,$host,$action1,$action2,$action3,
$lan,$nic,$mac,$src,$dst,$proto,$spt,$dpt) = split(/ /, $data);
$monthday = join(' ', $Log_month,$day);
$action = join(' ', $action1,$action2,$action3);
my $DateTimeStamp = sprintf("%04d-%02d-%02d %02d:%02d:%02d",
$year+1900,$mon+1,$mday,$hours,$min,$sec);
$statement = join('", "', ($DateTimeStamp,$monthday,$time,$host,
$action,$lan,$nic,$mac,$src,$dst,$proto,$spt,$dpt));
$statement =~ s/\b$/"/;
$statement =~ s/^\b/"/;
my $sth = $dbh->do (qq{ INSERT INTO '${myhost}_InBnd_Week$weeknum'
VALUES($statement) }) || die $dbh->errstr;
There are two "gotchas" here. First, iptables limit the
character length to 29 characters, so the phrase cannot be verbose.
Second, the lines that the Perl script reads are white-space delimited.
So, if "Inbnd Spoofed B External" gets changed to "Inbnd
Spoofed Class B External", the database insert will fail because
the number of columns to insert will be one more than the number of
actual columns in the table.
The process for outbound messages is identical to inbound messages
except the initial match occurs on the word Outbnd instead of Inbnd.
The Web Interface
The Web interface is written in Embedded Perl using EmbperlObject.
I chose Embedded Perl instead of cgi.pm because I find it easier
to format HTML output with it. The interface consists of a main
page that contains the frames to display the query pages and a header
that holds links to all the query pages. As mentioned previously,
FireWatch supports an internal and an external interface. How does
it know which interface and direction you want to look at? This
is accomplished in the main page of the Web interface.
When you first open the page, you are presented with a short form
where you can set the hostname, inbound or outbound traffic, and
internal or external interface. See Figure 3. When you click "choose",
the page is redisplayed with the parameters in the links to each
page. For example:
http://localhost/FW/TopDomains.html?LanChoice=External&HostChoice=mummer&Event=Inbnd
See Figure 4. The drop-down list for the hostname is retrieved by
a query of a hostname table. When new hosts are added to the database,
the Web page will automatically reflect the addition.
These parameters are passed into each used page as the base for
each query. The parameters are not reset until you select a new
set of data from the form.
An example query works like this:
[- $LanChoice = $fdat{LanChoice}; -] Process the information \
passed in.....
[- chomp $LanChoice; -]
[- $HostChoice = $fdat{HostChoice} -]
[- chomp $HostChoice; -]
[- $Event = $fdat{Event} -]
[- chomp $Event; -]
[$ if $Event eq "Inbnd" $]
[- $TableName = "InBnd" -]
[$ elsif $Event eq "Outbnd" $]
[- $TableName = "OutBnd" -]
[$ endif $]
.......and execute the query.
[- $sth = $dbh -> prepare ("SELECT count(*) as \
Event_Count,DateStamp,SRC_IP,SRC_Port,DST_Port,Protocol
from '${HostChoice}_${TableName}_Week$req->{weeknum}'
where Protocol = 'TCP' and Lan = '$LanChoice' \
and Event like '$Event%'
group by SRC_IP,DST_Port
order by DateStamp desc limit $LowerLimit, \
$ResultsPerPage"); -]
This example selects count(*) from the inbound table and calculates
the number of rows selected. Embedded Perl looks a lot like plain
old Perl, but what is $req->{weeknum}? It's EmbperlObject.
I used EmbperlObject to reduce the amount of repeated code
in each HTML file. When a request to the server to display a Web
page arrives, EmbPerl looks in the current directory for the file
specified by EMBPERL_OBJECT_BASE in httpd.conf. This file is executed
before the requested page is displayed. The key to this is the *
parameter in the line [- Execute ('*') -]. The asterisk literally
means "the filename that was actually requested". In addition
to the [- Execute ('*') -], base.epl can contain execute
statements for other epl files.
Global variables can be passed to each page as they are requested
via the "Request" object. This allows us to modularize
code and keep repeated code in each page to a minimum. For example,
the Web directory for FireWatch contains a file called time.epl.
This file holds all of the code for determining time, date, and
week number data. To make those variables available to each page,
use this Perl shift command ([- $req = shift; -])
near the top of each page. If you place a variable from time.epl
such as [+ $req->{TimeNow} +] in a page, the current time
will be displayed. This worked fine for time/date variables, but
I could not get MySQL and DBI/DBD to work in the modular fashion.
Installation
The installation of FireWatch consists of moving the appropriate
module or combination of modules onto a server. I built this with
Apache 1.3.27, mod_perl rpm version 1.26, and Embedded Perl 1.34.
I have experimented with Apache 2, mod_perl 1.99, and Embedded Perl
2, but the differences in Embedded Perl require some changes in
the Embedded Perl code (beyond the scope of this article).
Firewall Host Installation
I have included a firewall script in this package if you do not
have a firewall or if you are using ipchains. The script is configured
to allow SSH, SMTP, NTP, DNS, and Web services. To get it to work,
edit the script appropriately, copy it to /etc/rc.d/rc.firewall,
then run sh /etc/rc.d/rc.firewall. If there are no errors,
run:
service iptables save;
service iptables start
You now have a functioning firewall. You can verify by running iptables
-L -n to see the rules.
The next step is to test your services to make sure everything
still works. FireWatch needs MySQL (at a minimum) to log to the
database. The MySQL components that you'll need to install
depend on the function of the server. If the machine has only a
firewall on it, we will install the MySQL client, but if this machine
is also the database server, read the Database Host section of this
article for additional steps. Check whether MySQL is installed by
running:
rpm -qa | grep -i mysql
The minimum output should include:
mysql-3.23.41-1 -- This is the MySQL base application.
mysql-devel-3.23.41-1 -- This is the libraries and
include files.
mysqlclient9-3.23.xx -- This is the client application
for connecting to a remote server.
If any of these packages are missing, run up2date --nox "package"
to retrieve and install. If you have unresolved dependencies, it
is advantageous to use up2date because you can add the argument
--solvedeps and up2date will grab the dependencies.
Once the database client is installed, run the firewall-host-setup
script (found in the FireWatch-Setup directory). The last two items
are perl-DBI-xxx and perl-DBD-xxx. Check to see what
is installed by querying the rpm database:
rpm -qa | grep -i dbi and rpm -qa | grep -i dbd
for the DBD package. On my system, the output looked like this:
localhost $ rpm -qa |grep -i dbd
perl-DBD-MySQL-1.2216-4
localhost $ rpm -qa |grep -i dbi
perl-DBI-1.18-1
The perl-DBD-MySQL RPM's can be downloaded from rhn.redhat.com
under the packages section. The DBI and Msql-MySQL-modules (equivalent
to perl-DBD-MySQL RPM modules) can be found at CPAN.
To get the logger up and running, edit syslog.conf and add the
following snippet to the start section:
if [ ! -p /var/run/firefifo ]; then
'/bin/mknod /var/run/firefifo p'
'chmod 755 /var/run/firefifo'
fi
This little "if" statement makes sure that we have our fifo
up when syslog starts so we don't get an annoying error message
at boot. The last item, now that fifo is in syslog, is to add the
line to syslog.conf that feeds the fifo. Add this line to syslog:
kern.info |/var/run/firefifo
and then run service syslog restart.
Next, install the included Perl module Date::Calc:
Perl Makefile.PL
make
make test
make install
To conclude, add this to bounce FireWatch when the database server
creates the new week's tables:
00 00 * * * 1 /home/watcher/FW/roll.pl
Database Host
The database host will need the server version of MySQL installed,
so check for mysql-devel-3.23.54a-3.72, mysql-3.23.54a-3.72, mysql-server-3.23.54a-3.72,
and if necessary, find and install. If this is a new install, create
the initial access privileges by running mysql_install_db,
then service mysqld start. Now root can log in:
# mysql -u root mysql
The initial privileges are very open, so we need to restrict access.
Log into MySQL and at the MySQL prompt, set a password for root:
mysql> set password for root@localhost=PASSWORD('new_password');
mysql> set password for root@"hostname"=PASSWORD('new_password');
These commands set access restrictions for root on localhost and root
login to MySQL with the -h "hostname" option, which by default
does not have a password. From now on, you will need to use the password
that you chose to log into MySQL. Everything is ready, so restart
mysqld (service mysqld restart). As it starts, you should see
the green OK to the right.
We create the database by using a script called Create_it.pl.
Open the script and read what it is going to do. Note that on line
number 47 you must change my username (root) and password (test)
to whatever you have set up in the previous step. Also, the DBA
password can contain Perl special characters (like @ and $), but
remember to escape them (\@).
This creates the tables for the current week (that FireWatch will
start writing to) and creates three users: selector, creator, and
inserter. Having separate users for tasks minimizes risk by reducing
the privileges to only the necessary power to do each task. Run
the script and watch for errors. I tested this on several installs
of Red Hat 7.2 without any issues. If errors arise, check that mysqld
is running and verify that the DBA username and password have been
edited correctly in the Create_it.pl script.
If you choose not to run the Create_it.pl script, just open it
in a separate window and execute the commands as they are listed
in the script. Remember to run the Create_DB.pl script separately
if you do not run the Create_it.pl script. Finally, check that the
script did everything it was supposed to.
Install the included Perl module, Date::Calc:
perl Makefile.PL
make
make test
make install
Run crontab -e as root (you must be root to restart the service).
Add this line:
00 00 * * * 1 /home/watcher/FW/roll.pl
to create the new weekly table. Finally, add a crontab for the user
"watcher" to run the report scripts SendDailyReport.pl and
SendTop5Domains.pl every day around midnight.
Web Server Host
The Web server needs the following packages installed and functioning
in order to complete phase two:
1. Apache
2. Embedded Perl
3. Mod_Perl
4. Perl-DBI and Perl-DBD
5. MySQL client (if the database is remote)
Check for the Apache packages and make sure that you have the
apache-devel-version. The dev package will be needed for the Embedded
Perl install later.
Run apachectl start and open a browser to http://localhost
or your IP/DNS name and, if all is well, you will see the default
Apache page. Next, check whether Mod_Perl is installed:
rpm -qa | grep -i mod_perl
The output should be like mod_perl-1.2xx. If it is not, get it by
running up2date --nox mod_perl, or install from the CDs. Note
that the versions do not play well together. If you have Apache 2.0,
then you must have Mod_Perl version 2.x, and if you have Apache 1.3.22-x,
then stick with the Mod_Perl 1.x.
The last step is the most complicated, because we have to build
Embedded Perl from source and edit httpd.conf. You can get Embedded
Perl from:
perl.apache.org/embperl
To build this, run:
localhost> tar xzvf HTML-Embperl-1.3.4.tar.gz
and cd into the directory. Next, run:
perl Makefile.PL
The installer will ask a series of questions. If you have built Apache
from rpm, here is what the setup will look like:
[root@localhost HTML-Embperl-1.3.4]# perl Makefile.PL
Build with support for Apache mod_perl?(y/n) [y]
Use /usr/include/apache as Apache source(y/n) [y]
Will use /usr/include/apache for Apache Headers
Enter path and file to start as httpd \
[/usr/include/apache/httpd]/usr/sbin/httpd
Apache Version Server version: Apache/1.3.20 (Unix) (Red-Hat/Linux)
Library for mod_env.c not found,
please enter path to mod_env.so []/usr/lib/apache/mod_env.so.
Run make test, and then run make install. Once all of
the packages are installed and verified to work, edit httpd.conf to
add the parts that will make Embedded Perl function. Change directories
to /etc/httpd/conf and open httpd.conf. Find the <Directory>
section where the document root is set, and paste in this information
right above the <Directory /var/www/html/> so your conf file
will look like this:
<Directory "/var/www/html/FW/">
PerlSetEnv EMBPERL_ESCMODE 0
PerlSetEnv EMBPERL_OPTIONS 16
PerlSetEnv EMBPERL_OBJECT_BASE base.epl
PerlSetEnv EMBPERL_DEBUG 0
PerlSetEnv EMBPERL_SESSION_HANDLER_CLASS no
<FilesMatch ".*\.html$">
SetHandler perl-script
PerlHandler HTML::EmbperlObject
Options ExecCGI
</FilesMatch>
<FilesMatch ".*\.epl$">
Order allow,deny
Deny From all
</FilesMatch>
</Directory>
<Directory /var/www/html>
........
These lines of code set some environmental options and tell Apache
how to handle the Embedded Perl files in the FW directory. The first
FilesMatch sets up HTML files to be run as Perl files. The second
FilesMatch stops browsers from calling epl files directly. This is
a security-through-obscurity scheme and is useful for our purposes
because it stops wget from grabbing epl files with database username/password
and disguises our Embedded Perl files as plain old HTML. If everything
is configured correctly, the user will never know that Embedded Perl
dynamically generates the HTML that is viewed.
Lastly, install the included Perl module, Date::Calc, with perl
Makefile.PL, then make, make test, and then make
install. I isolated the FW Web directory on purpose so I would
not mess up an existing Web site. If you want to enable EmbperlObject
across all of your Web pages, just add a vanilla base.epl in the
document root that has one line [- Execute ('*') -] and
move the Embedded Perl directives into doc root. It is also a good
idea to password protect the FW directory after setup is finalized.
Now that Apache knows how to handle Embedded Perl, HUP Apache
by doing apachectl restart. The Web display should now be
functioning. Open a browser window to http://localhost/FW/mframe.html
or http://ip/dns name/FW/mframe.html and test it. If you
see weird Perl in brackets, then something went wrong with the Apache
configuration. Check the error_log and try again.
Test Test Test
Once all of the servers have been set up, you can go to any box
that has FireWatch installed and, as root, run service StartFire
start. Start mysqld on the database server. Next, open the Web
interface and see if anything has been logged. If not, you can move
things along by running a port scan against a firewalled machine.
Some Troubleshooting Tips
1. Check that the firewall machines can talk to the database.
For remote servers, run:
mysql -h hostname -u root -p
or, to verify connectivity:
local mysql -u root -p
2. Verify that FireWatch is logging data by logging into MySQL:
mysql -u selector -p
and typing select * from InBnd_Week (week number).
3. If StartFire fails to start FireWatch, check for database connectivity.
Conclusion
I have been using FireWatch since September 29, 2002. Since then,
I spotted the increase in scans to port 137 in October, a small
spike in port 80, and the slammer work recently. These events correlate
well with Internet Storm Center Statistics at http://isc.incidents.org.
Overall, I think that FireWatch is a good start in writing a firewall-logging
application, but it still needs some work. I am certain that there
are better ways to write the Perl, so I am posting this project
on Sourceforge.net so that everyone who is interested can contribute.
Will Schroeder has a B.S. in Geology from Northeastern Illinois
University. After working as a geologist for a few years, he changed
his career to UNIX administration. He currently supports the UNIX
side of electronic trading at the Chicago Mercantile Exchange. Red
Hat Linux is his primary desktop and server OS of choice. The author
thanks the Chicago Mercantile Exchange, H. Blevins, R. Davies, T.
Peterson, F. Lozano, D. Shea, and J. Cody for tons of support.
| 
