Article
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6

sep2005.tar

Monitoring PF Firewalls for Health and Performance

Ryan Matteson

The PF (packet filter) firewall package was introduced in OpenBSD 3.0, and has recently been ported to the FreeBSD and NetBSD operating systems. PF contains a stateful packet inspection engine, the ability to replicate state information to a backup firewall, a flexible self-optimizing rule engine, QOS support, and the ability to collect performance metrics. These metrics can be useful for gauging the performance of a firewall platform, and provide a way to trend firewall performance over time. This article will describe several utilities that can be used to monitor the health and performance of a PF firewall.

Viewing Performance Counters with pfctl

The pfctl utility provides a command-line interface to monitor and control the PF firewall module. Pfctl contains options to display firewall configuration information and a variety of performance and status counters. To display the status and performance counters, the "info" parameter can be passed to pfctl's "-s" (show values) option:

$ sudo pfctl -s info

Status: Enabled for 2 days 10:33:04           Debug: Urgent

Hostid: 0xdf1a2b73

Interface Stats for tun0              IPv4             IPv6
  Bytes In                       577197106                0
  Bytes Out                       30295580                0
  Packets In
    Passed                          564503                0
    Blocked                           1494                0
  Packets Out
    Passed                          386562                0
    Blocked                              0                0

State Table                          Total             Rate
  current entries                        1
  searches                         2164707           10.3/s
  inserts                            23601            0.1/s
  removals                           23600            0.1/s
Counters
  match                            1226533            5.8/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s

In addition to displaying the counters listed above, the show values option can be used to display firewall rulesets, NAT table entries, state table entries, and ALTQ queues. To get detailed usage data for each show values option, pfctl can be invoked with the "-v" (verbose) option. The following example shows how to get detailed usage data for each firewall rule:

$ sudo pfctl -v -s rules

@0 scrub in all fragment reassemble
  [ Evaluations: 3884714   Packets: 1917431  Bytes: 0        States: 0    ]

@1 block drop in log on tun0 all
  [ Evaluations: 1200737   Packets: 1368     Bytes: 170289   States: 0    ]

The pfctl(8) man page contains the full list of values that can be passed to the show values option, and covers pfctl's advanced options.

Viewing Active Connections with pftop

The pftop utility provides a "top" like view of the PF state table. Pftop displays source and destination IP addresses, TCP and UDP port numbers, packets and bytes transmitted, the age of a connection, and the time left until a connection will be removed from the state table. Pftop contains several options to control how data is displayed and is invoked by running the pftop executable from the command line:

$ sudo pftop

Pftop can be installed from the OpenBSD ports collection, or downloaded from the pftop Web site (http://www.eee.metu.edu.tr/~canacar/pftop/). A sample pftop display is included in Figure 1.

Monitoring Logged Packets with tcpdump

PF can record network packet headers and data when the log key word is used with a rule. Log files can be valuable for debugging rules, understanding traffic flows, and finding performance bottlenecks. When a packet matches a rule with the log key word, the headers and packet body are sent to the pflog pseudo-device. Once a packet is logged to the pflog pseudo-device, the tcpdump utility can be used to print the packet's contents in real time:

$ sudo tcpdump -i pflog0 -o -ttt -vv -e -n

Nov 26 00:58:32.001036 rule 4/0(match): block in on tun0: \
  1.2.3.4.4087 > 5.6.7.8.www: S [tcp sum ok] (src OS: Windows XP SP
1, Windows 2000 SP2+) 3615243359:3615243359(0) win 16384 \
  <mss 1440,nop,nop,sackOK> (DF) (ttl 119, id 7135)

[ ... ]

This example prints protocol headers, timestamps, and an OS fingerprint (using the signatures available in /etc/pf.os) for each packet logged to the pflog pseudo-device. Tcpdump's "-r" option can be used to process PF log files written to the file system by the pflogd daemon:

$ sudo tcpdump -r /var/log/pflog -o -ttt -vv -e -n dst port 80
Nov 26 00:58:32.001036 rule 4/0(match): block in on tun0: \
  1.2.3.4.4087 > 5.6.7.8.www: S [tcp sum ok] (src OS: Windows XP SP
1, Windows 2000 SP2+) 3615243359:3615243359(0) win 16384 \
  <mss 1440,nop,nop,sackOK> (DF) (ttl 119, id 7135)

[ ... ]

This example uses tcpdump's filtering capabilities to limit the results to connections with a destination port of 80. Tcpdump also allows connections to be filtered by IP address, hostname, Ethernet address, TCP flags, etc.

Summarizing Log Files with fwanalog

Fwanalog is a firewall log file analysis tool that can be used to parse and summarize the log files from several firewall packages (e.g., PF, Checkpoint, ipfilter, iptables, Cisco PIX ). Fwanalog uses the analog log file analysis program, and a simple straightforward configuration file to generate reports. The software is available as source code from the fwanalog Web site (http://tud.at/programm/fwanalog/).

The fwanalog package contains several sample configuration files, each used to represent a different firewall platform. The "fwanalog.opts.openbsd3" configuration file contains the definitions required to process PF log files on OpenBSD 3.X platforms. When the fwanalog.sh script is invoked, it will process the contents of the configuration file named "fwanalog.opts" by default. The fwanalog documentation recommends creating a symbolic link to point to the platform specific configuration file:

$ ln -s fwanalog.opts.openbsd3 fwanalog.opts

The fwanalog.opts configuration file contains a series of variables to define the log file locations, the format of each log file, and the location in which to store the reports. The customizable variables are summarized below:

outdir -- Location in which to store reports

logformat -- Format of the log file to analyze

inputfiles_mask -- The log file name (supports wildcards)

inputfiles_dir -- The directory with the log files

inputfiles_mtime -- The maximum age of the log files to parse

Once the configuration file is modified to match your desired layout and platform (a sample configuration file to parse PF log files on the OpenBSD platform is included in Figure 2), fwanalog.sh can be executed to generate a report:

$ fwanalog.sh

This will place the reports in the directory assigned to the "outdir" variable. The reports can be viewed with a Web browser or emailed as part of a routine reporting process.

Graphing PF Performance Data with pfstat

The pfstat utility can be used to collect and graph statistics exported through the /dev/pf pseudo-device. Pfstat can be installed from the OpenBSD ports collection, or downloaded from the pfstat Web site (http://www.benzedrine.cx/pfstat.html).

Pfstat requires the PF "loginterface" global configuration directive to be set in the pf.conf configuration file. This directive enables statistics collection for one of the physical interfaces in the firewall. The following pf.conf entry will collect statistics on the hme0 interface:

set loginterface hme0

Once statistics collecting is enabled, the pfstat utility can be invoked with the "-q" option. This will query the current value of each statistics counter and print the result to standard out:

$ pfstat -q
1101400143 1101219586 483226347 25637411 0 0 496899 3866 325988 \
  0 0 0 0 0 6 1692642 17030 17024 879499 0 2 0 0 0

Pfstat uses this data to generate historical utilization graphs, so the data should be collected at periodic intervals if graphs are desired. The following cron job will collect statistics every five minutes and write the results to "/var/log/pfstat/pfstat":

*/5 * * * * /usr/local/bin/pfstat -q >> /var/log/pfstat/pfstat

To graph the data that is collected, a pfstat configuration file needs to be created. This file describes the graphs to generate, how to display the data, and where to store the output. The following example shows the pfstat configuration required to graph state table data:

image "/var/www/htdocs/pfstat/state_table.jpg" {
     from 3 months to now
     width 800 height 300
     left
        graph states_entries  label "state table entries"  color 0 255 0,
        graph states_searches label "state table searches" color 255 0 0,
        graph states_inserts  label "state table insertions" color 0 0 255,
        graph states_removals label "state table removals" color 0 0 0
}

The pfstat configuration file contains one (or more) "image" directive. Each image directive is followed by the file name of the image to generate and a set of curly braces to control the attributes of the image. The "from" and "to" keywords select the time interval to graph. The value that follows the "from" keyword contains an integer value and a time frame (minutes, hours, days, weeks, months, years) to control how far back pfstat will go when processing data. The "to" keyword controls how pfstat processes new data elements, and the special key word "now" indicates the current time.

The "height" and "width" directives set the size, in pixels, for the height and width of the image to be output. Two directives determine the horizontal alignment of text descriptions: "left" aligns text on the left side of the graph, and "right" aligns text on the right side. The "graph" statements control which data is graphed, the label assigned to the graph, and the colors used to create the entries on the graph. As of pfstat version 1.7, pfstat can graph packets, bytes, state table information, and several miscellaneous packet counters. The full list of options that can be passed to the "graph" directive are described in the pfstat(8) man page.

Once the configuration file is created, we can execute pfstat, and pass the configuration and data file as arguments to the "-c" and "-d" options:

$ pfstat -c /etc/pfstat/pfstat.conf -d /var/log/pfstat/pfstat >/dev/null

Figure 3 contains a sample pfstat.conf configuration file to graph IPv4, IPv6, and state table information. Figures 4, 5, and 6 include sample pfstat graphs from a production firewall.

Conclusion

This article provided an introduction to several tools that can be used to monitor the health and performance of a PF firewall. Links to additional tools that can be used to monitor PF firewalls are included in the reference section. For additional details on advanced options and usage, consult the PF FAQ and application manual pages.

References

PF FAQ -- http://www.openbsd.org/faq/pf/

FW Analog -- http://tud.at/programm/fwanalog/

Analog -- http://www.analog.cx/

PF Top -- http://www.eee.metu.edu.tr/~canacar/pftop/

PF Stat -- http://www.benzedrine.cx/pfstat.html

PF Hatchet -- http://www.dixongroup.net/hatchet/

PF Flowd -- http://www.mindrot.org/pfflowd.html

PF Sysinfo -- http://team.gcu-squad.org/~aflab/projects/pfsysinfo/

Acknowledgements

Ryan would like to thank the OpenBSD, PF, pftop, pfstat, tcpdump, analog, and fwanalog developers for all of their awesome work!

Ryan Matteson works as a systems engineer for Earthlink and specializes in Web technologies, SANs, and the OpenBSD, Linux, and Solaris operating systems. Questions and comments about this article can be addressed to matty91@gmail.com.