Failover Firewalls with OpenBSD and CARP

Jason Dixon

Firewalls are a required component in commercial and residential computer networks. For many installations, the firewall is a single point of failure between client systems and external resources. It can also become a liability when hardware or applications fail, leaving potential customers unable to reach your servers. A properly designed and executed failover configuration for your primary firewall will address many of these concerns. This article introduces a proven method for installing redundant stateful firewalls using native OpenBSD features.

The OpenBSD project is known for creating a leading secure Unix-like operating system. They have always emphasized software robustness and security, while ensuring their code remains free for all purposes under the BSD license. A number of exciting features have been introduced to OpenBSD due to licensing disagreements. Many BSD users are familiar with the rift between Darren Reed, (the creator of IPFilter) and the OpenBSD developers. A change in the IPFilter license resulted in the rapid development of the OpenBSD PF firewall software. Not only is PF a competitor to expensive proprietary offerings, it is so successful that it has been ported to both FreeBSD and NetBSD distributions.

Within the past two years, OpenBSD recognized the need to support failover between OpenBSD firewalls. The pfsync protocol was completed, which sends state change messages via multicast over the pfsync interface. Using a secure connection (a crossover cable between systems is suggested), pfsync will notify other OpenBSD firewalls of changes to the local state table. If other firewalls are listening for the pfsync packets, they will update their own state tables with these announcements. This feature allows sessions to failover gracefully without losing connectivity or raising alerts in the firewall, providing the basic features required for stateful redundancy. However, the ability to dynamically failover to the stateful partners was still unavailable.

The Birth of CARP

The Virtual Router Redundancy Protocol (VRRP) eliminates the single point of failure in a static network by assigning a virtual gateway between multiple physical routers. This allows two or more routers to cooperate as a dynamic gateway; one will perform as the "master", while the other system waits as a "backup". If the master becomes unavailable, the backup will begin advertising itself as the master, allowing traffic to continue uninterrupted over the new physical path. Unfortunately, although VRRP is an IETF-standard protocol, it is also encumbered by a patent held by its author, Cisco Systems, Inc. They claim to have no intention of asserting patent claims against anyone implementing VRRP, but publicly reserve the right to assert patent claims defensively. OpenBSD needed this functionality to support failover between hosts, but the looming patent issue made VRRP a poor choice.

Based on their dedication to free software, the OpenBSD team went to work on creating a patent-free replacement for VRRP. This was released in the form of the Common Address Redundancy Protocol (CARP) in late 2003. CARP operates at the data-link and network OSI layers, using a virtual MAC and one or more virtual IP addresses. The master router of the CARP group responds to ARP requests for the virtual MAC with the shared IP address, allowing switches to quickly determine to which interface to forward traffic. CARP supports IPv4 and IPv6, load-balancing across the shared group, master preemption, and cryptographic hashing of the data-link announcements. Thanks to PF, pfsync, and CARP, users are now able to deploy truly redundant firewalls using free software and commodity hardware.

Installing OpenBSD is beyond the scope of this article, but should be familiar to NetBSD (and perhaps even Debian Linux) users. The media is available either via CD-ROM purchased from the OpenBSD store, or you can install via FTP by downloading one of the boot images. There are no CD-ROM ISO images available for download; CD-ROM and other project merchandise sales are used to support the ongoing development. The installation process usually takes no more than 10-20 minutes to complete, depending on media access and disk speed.

The scenarios presented in this article portray a typical dual-homed connection that you might encounter at any business or residence. Under most circumstances, it is suggested to incorporate a demilitarized-zone ("DMZ") network to segregate inbound traffic to your public servers. This helps keep unwanted intruders out of your LAN. We will utilize a third interface for this example, but it will only carry pfsync notifications. Adding a DMZ is as simple as creating an additional physical and CARP interface for the DMZ.

Basic Configuration

Both systems should be configured to use unique (not shared between CARP group members) IP addresses for the physical interfaces. As of the time of this writing, code has been imported to allow CARP devices to operate without the need for a network interface. However, this code is still under heavy testing at the time of writing and should not be considered production-ready until the OpenBSD 3.7 release (May 2005).

Figure 1 shows that each firewall has a publicly routable IPv4 address for fxp0, a private RFC1918 address on fxp1, and another private address on fxp2. Each of the routing interfaces can be configured to use multiple CARP interfaces, although we will only be creating a single CARP interface on fxp1 to operate as our LAN gateway. The network settings for each physical device are stored in the /etc/hostname.fxp* files; each CARP interface has its settings stored in the corresponding /etc/hostname.carp* files. Media options are considered optional and will not be shown in the examples below. This will result in the connection duplex auto-negotiating at 100baseTX.

Each CARP interface must be configured with a virtual host ID (vhid) and virtual host IP address. The vhid must be unique among CARP interfaces on the same network segment. The advbase and advskew parameters are optional and are used to control how frequently a master host sends advertisements. A custom advskew setting will result in fewer advertisements, forcing the host into the backup role. The pass parameter is recommended as it is used to authenticate CARP advertisements between group members. The commands to manually enable the CARP interfaces are:

test1# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 pass foo
test1# ifconfig carp1 10.0.0.1 netmask 255.255.255.0 vhid 1 pass bar

test2# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 pass foo
test2# ifconfig carp1 10.0.0.1 netmask 255.255.255.0 vhid 1 pass bar

To enable the pfsync interface, we only need to pass ifconfig the "up syncif" keywords and the associated interface. However, in those cases where a dedicated pair of interfaces is not available for crossover connectivity, the syncpeer keyword can be used to designate a peer network address. If this is the desired effect, all pfsync traffic should be encrypted across enc0, the OpenBSD IPSec virtual interface. For our purposes, we will simply rely on passing unencrypted messages across the dedicated crossover between fxp2 on each firewall:

test1# ifconfig pfsync0 syncif fxp2

test2# ifconfig pfsync0 syncif fxp2

In situations where a preferred master is wanted, preemption can be enabled. We need to raise the advskew for the intended backup host. Once that is complete, both systems must enable preemption via the net.inet.carp.preempt sysctl variable. The sysctl changes must be stored permanently in /etc/sysctl.conf, and the advskew updates should be made to the hostname.carp* files on the backup host:

test2# ifconfig carp0 advskew 100
test2# ifconfig carp1 advskew 100
test2# sysctl -w net.inet.carp.preempt=1
net.inet.carp.preempt 0 -> 1

test1# sysctl -w net.inet.carp.preempt=1
net.inet.carp.preempt 0 -> 1

Listing 1 reveals the basic PF ruleset that will allow our firewall pair to block unwanted traffic. This example only allows outbound Network Address Translation (NAT) connections bound to the external interface, as well as connections initiated from the firewall itself. We must also allow pfsync traffic on fxp2 and CARP traffic on fxp0 and fxp1. Once the pf.conf has been saved, the new configuration should be tested for syntax errors. A quiet return prompt indicates a successful ruleset parsing:

test1# pfctl -nf /etc/pf.conf
test1#

test2# pfctl -nf /etc/pf.conf
test2#

With no errors reported, we can enable the ruleset:

test1# pfctl -f /etc/pf.conf

test2# pfctl -f /etc/pf.conf

Please note that all of the aforementioned network settings will need to be preserved in static configuration files. The output of these files can be seen in Listing 2.

Testing the Configuration

Running tcpdump on each CARP-enabled physical interface should reveal the master host multicasting its CARP advertisements. A sample capture from the LAN segment is shown below. Note that we want to look for packets matching protocol number 112. Although this is recognized as VRRP on many systems, OpenBSD has updated the /etc/protocols file to reflect this as CARP traffic:

test1# tcpdump -nvi fxp0 -c3 proto 112
tcpdump: listening on fxp0
20:32:55.110102 carp 10.0.0.2 > 224.0.0.18: CARPv2-advertise 36: \
  vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] (ttl 255, id 15586)
20:32:56.120098 carp 10.0.0.2 > 224.0.0.18: CARPv2-advertise 36: \
  vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] (ttl 255, id 8283)
20:32:57.130095 carp 10.0.0.2 > 224.0.0.18: CARPv2-advertise 36: \
  vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] (ttl 255, id 18372)

The ifconfig command will convey the state of our physical, CARP, and pfsync interfaces. Passing the -A parameter to ifconfig will also show the state of all address aliases for each of our interfaces:

test1# ifconfig -A
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:d0:b7:bf:c6:95
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 66.77.24.2 netmask 0xffffff00 broadcast 66.77.24.255
        inet6 fe80::202:b3ff:fe0a:ce04%fxp0 prefixlen 64 scopeid 0x1
fxp1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:02:b3:0a:d1:28
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::202:b3ff:fe16:a957%fxp1 prefixlen 64 scopeid 0x2
fxp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:c0:4f:46:8d:ec
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.255.255.2 netmask 0xffffff00 broadcast 10.255.255.255
        inet6 fe80::2c0:4fff:fe46:9448%fxp2 prefixlen 64 scopeid 0x3
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=41<UP,RUNNING> mtu 1348
        pfsync: syncif: fxp2 syncpeer: 224.0.0.240 maxupd: 128
enc0: flags=0<> mtu 1536
carp0: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 66.77.24.5 netmask 0xffffff00
carp1: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 10.0.0.1 netmask 0xffffff00

The output clearly shows that both carp0 and carp1 on this system are serving as master of the group. Since preemption is enabled, this system will always retain the master role while it is able to advertise itself as available. If a designated "backup" firewall has accepted the master role, it will immediately relinquish that responsibility upon receiving advertisements from the system with the lower advskew. This behavior can also be monitored using the tcpdump techniques revealed earlier. Tcpdump can also be used to observe the pfsync state messages:

test1# tcpdump -nvvettti fxp2
Jan 31 21:18:22.135400 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 262: \
  10.255.255.2 > 224.0.0.240: PFSYNCv2 count 1: INS ST:
 (DF) [tos 0x10] (ttl 255, id 58487)
Jan 31 21:18:23.130035 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 302: \
  10.255.255.2 > 224.0.0.240: PFSYNCv2 count 3: UPD ST COMP:
 (DF) [tos 0x10] (ttl 255, id 64429)
Jan 31 21:18:25.940527 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 70: \
  10.255.255.2 > 224.0.0.240: PFSYNCv2 count 2: DEL ST COMP:
        id: 41f3a32500007f5b creatorid: d0491513
        id: 41f3a32500007fe1 creatorid: d0491513
 (DF) [tos 0x10] (ttl 255, id 59638)
^C

All LAN workstations should be configured to use the internal CARP address as their network gateway. Then if the master firewall becomes unavailable, any existing connections will be immediately resumed by the next possible failover member. Simple tests -- such as Web browsing and ping -- should be performed to verify that clients are able to traverse the gateway.

Once basic connectivity and routing are confirmed, more advanced tests should be initiated to confirm stateful failover capabilities. SCP is an excellent tool for this test. Make sure to use a sufficiently large test file such that it will still be transferring when you unplug the cable. Immediately following the physical disconnect, you may experience a brief 1-2 second delay as the failover occurs. The session should abruptly resume, revealing that the connection has been preserved. After you plug the cable back in, traffic should immediately "bounce" to the preemptive master and continue unassumingly. By running pftop from the ports collection, you can watch the traffic "move" from one machine to the other.

Advanced Configuration

The next design introduces a method for load balancing connections between the redundant pair and a pool of internal servers. To distribute the load across both firewalls, a second CARP interface must be created on each firewall's external and internal interfaces. Each firewall will serve as master and one as backup for each CARP interface. The net.inet.carp.arpbalance sysctl variable must also be enabled. An address alias is also being added to support additional services:

test1# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 pass foo
test1# ifconfig carp0 alias 66.77.24.10 netmask 255.255.255.0 vhid 1 pass foo
test1# ifconfig carp1 66.77.24.5 netmask 255.255.255.0 vhid 2 advskew \
  100 pass foo
test1# ifconfig carp1 alias 66.77.24.10 netmask 255.255.255.0 vhid 2 \
  advskew 100 pass foo
test1# ifconfig carp2 10.0.0.1 netmask 255.255.255.0 vhid 1 pass bar
test1# ifconfig carp3 10.0.0.1 netmask 255.255.255.0 vhid 2 advskew 100 \
  pass bar
test1# sysctl -w net.inet.carp.arpbalance=1
net.inet.carp.arpbalance: 0 -> 1

test2# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 advskew \
  100 pass foo
test2# ifconfig carp0 alias 66.77.24.10 netmask 255.255.255.0 vhid \
  1 advskew 100 pass foo
test2# ifconfig carp1 66.77.24.5 netmask 255.255.255.0 vhid 2 pass foo
test2# ifconfig carp1 alias 66.77.24.10 netmask 255.255.255.0 vhid 2 pass foo
test2# ifconfig carp2 10.0.0.1 netmask 255.255.255.0 vhid 1 advskew \
  100 pass bar
test2# ifconfig carp3 10.0.0.1 netmask 255.255.255.0 vhid 2 pass bar
test2# sysctl -w net.inet.carp.arpbalance=1
net.inet.carp.arpbalance: 0 -> 1

Reviewing the output of ifconfig, we are able to confirm the presence of the new interfaces and aliases:

test1# ifconfig -A
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:d0:b7:bf:c6:95
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 66.77.24.2 netmask 0xffffff00 broadcast 66.77.24.255
        inet6 fe80::2d0:b7ff:febf:c695%fxp0 prefixlen 64 scopeid 0x1
fxp1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:02:b3:0a:d1:28
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::202:b3ff:fe0a:d128%fxp1 prefixlen 64 scopeid 0x2
fxp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:c0:4f:46:8d:ec
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.255.255.1 netmask 0xffffff00 broadcast 10.255.255.255
        inet6 fe80::2c0:4fff:fe46:8dec%fxp2 prefixlen 64 scopeid 0x3
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=41<UP,RUNNING> mtu 1348
        pfsync: syncif: fxp2 syncpeer: 224.0.0.240 maxupd: 128
enc0: flags=0<> mtu 1536
carp0: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 66.77.24.5 netmask 0xffffff00
        inet 66.77.24.10 netmask 0xffffff00
carp1: flags=41<UP,RUNNING> mtu 1500
        carp: BACKUP vhid 2 advbase 1 advskew 100
        inet 66.77.24.5 netmask 0xffffff00
        inet 66.77.24.10 netmask 0xffffff00
carp2: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 10.0.0.1 netmask 0xffffff00
carp3: flags=41<UP,RUNNING> mtu 1500
        carp: BACKUP vhid 2 advbase 1 advskew 100
        inet 10.0.0.1 netmask 0xffffff00

The diagram in Figure 2 shows that we have added three HTTP servers and one SMTP server. Each of the Web servers has a private IP address, but they all use the same public address via NAT. Outbound connections will cause the source address to be translated; inbound connections will be redirected to one of the destination addresses, mapping packets based on a hash of the source address. This will allow the firewalls to persistently send traffic from one client to the same server. The SMTP server also has a private address, but its traffic is translated via bidirectional mapping (binat) to a new external carp interface. The revised pf.conf can be viewed in Listing 3. A compilation of all the revised network settings are detailed in Listing 4.

With arpbalance enabled, incoming packets will be distributed between the two firewalls in a round-robin fashion. New connections are tracked via the source-hash option to the rdr translation rule. This ensures that future packets originating from the same client are passed on to the same internal server. Additional load-balancing techniques can be used on the internal HTTP pool by incorporating CARP on each of the pool members. This way, we not only allow traffic to be distributed between all three hosts, but we provide failover capabilities as well, just like that of the firewall pair.

Summary

Through the use of native OpenBSD utilities, we have a high-availability firewall solution based on commodity hardware that competes favorably with commercial offerings costing thousands of dollars more. The practical applications for CARP are limitless, as it can be used anywhere that load-balancing needs exist. It has been ported to userland on Linux kernels 2.4 and 2.6, NetBSD and FreeBSD, making it an ideal redundancy tool no matter what Unix-like system you use. But combine CARP with the enhanced security of OpenBSD, PF and pfsync, and you have a stateful firewall that won't let you or your customers down.

Resources

OpenBSD -- http://www.openbsd.org/

OpenBSD FAQ -- http://www.openbsd.org/faq/index.html

PF -- http://www.benzedrine.cx/pf.html

PF User's Guide -- http://www.openbsd.org/faq/pf/index.html

Userland CARP -- http://www.ucarp.org/

VRRP RFC 3768 -- http://www.benzedrine.cx/pf.html

Jason Dixon is the owner and primary consultant of DixonGroup Consulting LLC, in Westminster, Maryland. He specializes in secure, robust IT solutions using free and open source software. Outside of work, he spends his time with his family and Apple PowerBook, in that order. Jason can be reached at: jason@dixongroup.net.