Embedded Linux Router

Tom Erjavec

Embedding Linux to create a networking device has long been a desired project of mine. With the advent of many micro-Linux distributions, it has become an easy task that can be done by any Linux lover. My goal was a high-performance secure Internet gateway, so I built a PC/Linux device that could offer full speed on an Ethernet port and provide a firewalled connection to the Internet.

It would be easy to run just one of the Linux Router Project solutions off a floppy disk. But the 1.44 or 1.68 MB floppy space that most LRP solutions provided was not sufficient for the router software configuration I wanted. One possibility was to move the software to a hard disk or to burn the software image onto a CD and run the router off the CD drive. But I decided to go for a solution with no moving mechanical parts. I wanted a flash-disk solution without any drives, connected devices, or even a cooling fan in order to provide an "install-and-forget" and completely silent operation.

The goal of my project is sketched in Figure 1. The routing device must support NAT, DHCP, DNS, and packet filtering with stateful inspection. More than two network segments can be configured, thereby creating one or more demilitarized zones. Inbound NAT enables packet forwarding to virtually provide services on the router while they are really running on a machine in the demilitarized segment. All this and more (using additional LRP packages) is possible with an LRP distribution.

Hardware

The hardware I found suitable for my purpose was an older Cyrix MediaGX 233 set-top box, Allwell STB3036. It's a slim-line, small footprint PC with power consumption low enough to survive without a power-supply fan, and a processor slow enough not to need a cooling fan (Figures 2 and 3). The STB3036 is equipped with M-Systems' DiskOnChip (DoC) device providing 16 MB of solid-state disk space in a flash onboard, but it lacks both a floppy drive and a CD drive. The DoC module is situated in a corner of the motherboard, making it easy to extract and insert the module into the socket (Figure 4).

Software

My software selection was Jacques Nilo's and Eric Wolzak's LEAF/Bering, a derivative of Charles Steinkuehler's Dachstein, itself a derivative of Dave Cinege's original Linux Router Project. The underlying Linux is a dwarfed Debian distribution. I chose Bering because it included Tom Eastep's Shoreline Firewall, a handy configuration utility around iptables. Additionally, Jacques Nilo's archive offers a number of fine software packages that are compatible with Bering. The documentation that comes with Bering is very well written, and it is not difficult to set up a customized environment. The prefixed acronym LEAF (Linux Embedded Appliance Firewall) suggests the purpose itself.

The first obstacle I encountered was booting my hardware platform without a floppy drive from a Bering startup floppy disk. I didn't have an external floppy drive so I used my desktop PC instead. I disconnected the floppy data cable from my desktop PC's motherboard and connected it to the STB3036 motherboard. After powering desktop PC first and STB3036 second, it looked like a blood transfusion but it worked fine (Figure 5). STB3036 woke up in the micro-Linux environment of Bering. As a candidate for embedding, it is clear that Bering does not run off the boot media but rather generates a RAM disk in the system's memory, transfers all the software from the boot media into the RAM disk, and runs from there.

Once my black box with micro-Linux was running, it was still far from operational as a router. Because a micro-Linux kernel supports very few devices straight out of the box, it is necessary to provide all the drivers for the specific hardware being used. The kernel that comes with Bering 1.2 is 2.4.20 and is modular. Jacques Nilo provides a broad range of various modular device drivers in his repository:

http://leaf.sourceforge.net/devel/jnilo/bering/latest/modules/

mii.o, 8193too.o and 3c59x.o

mtdcore.o, docecc.o, doc2000.o, docprobe.o and nftl.o

I activated the modules with the insmode command manually to test whether everything worked. It did. The docprobe module revealed the segments of my flash memory, and ntfl module gave life to the /dev/nftla device. However, the flash was still unusable without partitions and a file system.

Configuring Modules

So far, all the work has been done manually. To make Bering set the customized environment automatically, the system must be configured. This can easily be done with lrcfg, the Bering configuration menu system. Modules can be added at two points in the micro-Linux file system. The modules that do not need to be active at the beginning of the boot procedure (e.g., Ethernet drivers) are placed in /lib/modules using the "Packages configuration -- modules" menu selection in lrcfg. However, the drivers that are necessary for reading from the boot device (e.g., flash memory drivers in my case) must be loaded prior to the boot procedure. These are placed in /boot/lib/modules using the "Packages configuration -- initrd" menu selection. When the flash image is saved, these will be put into the initial RAM disk for startup purposes.

Because Bering runs off the RAM disk, all the configuration changes done in RAM disk must be saved back to the target boot media before restarting the system. This can be done using the backup options in the lrcfg menu (Figure 6).

Configuring Services

The future services of the routing platform are defined and configured by adding software packages to the bare-bone routing platform. Many of them are already included in the distribution, such as:

• IPtables packet filtering
• Pump (DHCP)
• Shorewall configuration tool for iptables
• Logging utility
• DNS cache

But there are some that may be added. I used the following additional packages:

• libz.lrp
• sftp.lrp
• sshd.lrp
• sshkey.lrp

These provided remote management through SSH, although a simple Web-based monitoring is provided with the initial system (Figure 7). Some additional services you may want to consider include:

• A tiny FreeSWAN package for IPsec VPN tunneling
• A tiny HTTP server (a question of security)
• A tiny SMTP server (again, a question of security)

My selection of additional packages was too big to fit on the distribution floppy. I added them onto the RAM-disk file system after my machine was booting off the flash and then reconfigured the boot image.

Note that a .lrp package is nothing but a gzip archive. All LRP packages have a common structure and as long as they are compiled for the same kernel, they can be interchanged among various derivatives of the original LRP project. On Bering, I used some from Charles Steinkuehler's Dachstein and they worked perfectly well.

Besides the services you want to run, you must also acquire the packages needed for customizing the system. These will be used only once and, after the finished setup, they can be deleted from the system. These packages are:

• fdisk.lrp
• mkdosfs.lrp
• syslinux.lrp

They will provide target disk partitioning, creation of the file system, and setting up the boot procedure, respectively.

The services can be configured individually through the lrcfg menu once they have been successfully installed in the RAM disk. The installation can be done manually by running the lrpkg command or automatically by defining the service in the syslinux.cfg file, as described in the next section.

I will not describe here how to configure the networking services, Shorewall firewall, and other individual packages. That process is very well described in the Bering documentation.

Starting Services

The system startup is provided by Peter Anvin's syslinux. It can boot Linux from DOS FAT-12 filesystem, which is used in LRP. To load a service from the target disk, the service package must be copied to the root of the filesystem and the service must be included in the LRP= parameter of the syslinux.cfg file. My line for loading packages is:

LRP=root,etc,local,modules,iptables,pump,keyboard,shorwall, 
ulogd,dnscache,weblet,libz,sshkey,sshd,sftp

Embedding

With flash modules in /boot/lib/modules, the system reset will bring up all software needed for accessing the flash in DoC. The first tasks are to create a partition table in the master boot record of the flash disk and to create a file system. The tools for this are installed with:

lrpkg -i fdisk.lrp
lrpkg -i mkdosfs.lrp

fdisk /dev/nftla

Now a FAT-12 partition exists on the/dev/nftla1, but it is empty. A FAT-12 filesystem is created in it by running:

mkdosfs -F 12 /dev/nftla1

mkdir /doc
mount -t msdos /dev/nftla1 /doc

mount -t msdos /dev/fd0u1680 /mnt
cp /mnt/* /doc

display syslinux.dpy
timeout 0
default linux initrd=initrd.lrp init=/linuxrc rw root=/dev/ram0 \
  boot=/dev/fd0u1680:msdos PKGPATH=/dev/fd0u1680
LRP=root,etc,local,modules,iptables,pump,keyboard,shorwall, \
  ulogd,dnscache,weblet,libz,sshkey,sshd,sftp

boot=/dev/nftla1:msdos
PKGPATH=/dev/nftla1

umount /doc
syslinux -s /dev/nftla1

The big moment has arrived. We do one last check: the partition table is OK; the file system is OK; the bootstrap code should be fine; the modules configuration is OK; the packages are OK. Power down. Power up. Change the boot device order in BIOS from floppy drive to flash. Reset. The screen flickers after the BIOS test and then comes the infinite darkness...

Debugging the Bootstrap Procedure

When trying to determine what went wrong, I suspected syslinux. I tried three different releases without success. When all reasonable resources had been tried, nothing was left but a HEX editor. I zeroed the Master Boot Record of the flash, restarted the setup from the beginning for the fourth time, and took snapshots of the MBR with dd after each step:

dd if=/dev/nftla of=/mnt/<snapshot-name> count=1

I looked around and found a 16-year-old floppy disk with MS-DOS 6.2. The floppy still worked (I am not sure about MS-DOS). Microsoft did implement a non-documented switch with their FDISK.COM to provide the Step 1 bootstrap code in the MBR by issuing the command:

FDISK /MBR

I made FDISK /MBR on a spare disk of another machine, but any other healthy disk with an MBR should be fine. Then I ripped the bootstrap code from the MBR off that hard drive by using:

dd if=/dev/hda of=/mnt/mbr.bin bs=512 count=1

mount -t msdos /dev/fd0 /mnt
dd if=/mnt/mbr.bin of=/dev/nftla bs=512 count=1

For those that don't keep old floppies around or don't want to dd MBRs of healthy (usually non-backuped) hard drives, I must add that, obviously, I did this the hard way. If I had thoroughly read the documentation of syslinux, I'd have known that Peter Anvin provided a generic MBR, including the assembly language source code that contains the Step 1 bootstrap code. However, the Bering step-by-step instructions keep this fact as a secret, and you must know the bootstrap procedure pretty well to locate the problem. I did not experiment with this generic MBR code since I trusted my patchwork was flawless.

This time the reboot didn't fail. In comparison to a floppy disk, the flash provided a lightning-fast bootstrap.

Performance Evaluation

I installed a commercial SOHO router and my embedded Linux router side-by-side to operate in the same environment. My evaluation test was feeding traffic in increments of 1 Mb/s on the internal side through the router to the external side. A probe measured the achieved throughput. The generated packets had the maximum size of 64 KB and were fragmented by the source machine to the largest acceptable framesize on Ethernet -- 1518 bytes. In real life, the packets would be smaller and the routers would be busier with NAT address translation on additional smaller-sized packets. Still, the measurement can give us a feeling for performance comparison.

Figure 9 shows the throughput graph for the commercial router. It was incrementally loaded with eight 1-Mb/s streams. In the beginning, the increments can be seen from the graph, but after the fourth increment (at 4 Mb/s), the linear response of the router is broken, the graph starts oscillating, and the router starts dropping IP packets. The packet loss went as high as 86% with router saturation reached at just over 4 Mb/s.

The situation is much different in Figure 10, which shows the response of the embedded Linux router. Note that the scales of the two graphs are different -- the graph of the commercial router scales to 7 Mb/s, while the scale of the Linux router graph is almost 10 times higher (60 Mb/s). The Linux router was incrementally loaded with 40 simultaneous streams of 1 Mb/s. The graph shows a nice linear response without any sign of saturation. At 42 Mb/s, ten times exceeding the threshold of the commercial router, the packet loss of the Linux router was 0% while the packet delay increased from 1 ms to 75 ms, which is still a quite acceptable response time.

Conclusion

An embedded LRP Linux device can be an appealing security gateway. This solid-state box with no moving parts and Linux hidden in a flash is an "install-and-forget" solution with automatic log rotation that needs very little or no maintenance. Superior performance, compared to commercial devices, and an availability of various configurable networking services make it ideal for an advanced SOHO user.

The Bering distribution, a flavor of LRP, is one of the more advanced LRP branches. However, Bering already has a descendant called Lince, developed by Juan Jesus Prieto. It includes a downsized Squid proxy server and Dansguardian Web content filtering package in the standard distribution. The sacrifice (or rather evolution) is that Lince no longer fits on a floppy but on a CD-ROM.

FreeSWAN IPsec VPN package is another logical choice for an embedded Linux router. But, it should be noted that 3-DES algorithm implemented in FreeSWAN drains the processor power significantly. Loaded with encrypted IPSec traffic, the performance of a system would be expected to drop by a factor of 10.

References

1. LEAF project (Bering, Dachstein, Lince, Oxygen, etc.) -- http://leaf.sourceforge.net

2. Tom Eastep's firewall project Shorewall -- http://www.shorewall.net

3. Repository of the Bering project -- http://sourceforge.net/project/showfiles.php?group_id=13751

4. Compiled modules for Bering, Linux kernel 2.4.20 -- http://leaf.sourceforge.net/devel/jnilo/bering/latest/modules/

5. Peter Anvin's Syslinux -- http://syslinux.zytor.com/

6. Repository of Syslinux releases -- http://www.kernel.org/pub/linux/utils/boot/syslinux/

7. Charles Steinkuehler's home page -- http://lrp.steinkuehler.net/

8. Original LRP homepage -- http://www.linuxrouter.org/

Tom Erjavec was awarded his M.Sc. from computer science at the University of Ljubljana, Slovenia. In the past, he was an assisting teacher at Faculty of computer science in Ljubljana. He then worked with IBM Slovenia as a software developer, with SRC as an electronic meeting systems manager. He co-founded a network integration company MIBO Integra, and is now a marketing director at Iskra Transmission, a microwave telecommunications equipment vendor. Linux remains his hobby.