Maintaining Patch Levels with Open Source BSDs

Michael Lucas

The open source BSDs are based on 20-year-old UNIX code released by the University of California Berkeley in the early 1990s. Today, open source BSD has broken into three branches: FreeBSD, NetBSD, and OpenBSD. Each has its own focus. NetBSD runs on almost any hardware. OpenBSD's focus is security on a wide hardware base. FreeBSD focuses on being the most stable, high-performance OS possible on Intel CPUs. Versions of the open source BSDs are used on several major ISP Web sites and in many commercial organizations. Besides time-tested code and top-notch performance, however, all the BSDs offer three main features.

The emulation functions are an important feature. Most open source BSDs can run binaries for SVR4, SCO UNIX, BSDI, character-mode DOS, and Linux. The Linux emulation alone is remarkable; applications from WordPerfect to Informix run perfectly. The second advantage (for corporate users, at least) is the license. The BSD license boils down to: "Give us credit for our work, and don't sue us if something breaks." There is no obligation to make changes public or to propagate source code. The last advantage is the very easy upgrade method. This article was written on a FreeBSD machine, installed in November of 1997. It's running a version of FreeBSD less than a week old. I upgrade the operating system every week, and it takes less than fifteen minutes each time. This is due to the BSD development model and tools like sup and CVSup.

This article focuses on upgrading FreeBSD systems, but the concepts are applicable in both OpenBSD and NetBSD. Some of the procedures are slightly different, but pointers to detailed information for each BSD are provided.

BSD Development Model

The open source BSDs have several versions available at any time: snapshots/releases, current, and (in FreeBSD) stable. Current is the very latest version, containing code being tested publicly for the first time. If you can't read C, or don't feel like debugging your OS, or don't like parts of an OS failing during an initial public test, current might not be your best choice. The developers do their best to keep the current version reliable, but there are times when this simply isn't possible. For example, this spring, FreeBSD-current changed compilers from gcc to egcs. Some pieces of third-party software stopped working until they could be debugged. Although the developers fixed these issues quickly, this is indicative of the types of problems a current user can have.

Under FreeBSD, once a piece of code is thoroughly tested in current mode, it is merged into "stable". Stable is expected to be solidly tested and relatively bug-free. Keep in mind that "stable" is a description of the codebase, not of the OS itself. For example, FreeBSD 2.2.8-stable is considered more "reliable" than FreeBSD 3.0-stable, although both have the stable tag. As with any software, you need to be leery of .0 releases.

A "snapshot" or "release" is generally made for a CD image and reflects the state of current or stable at a particular date. The advantage to this system is quality control. You can use the quality given by a stable or release version, or contribute to the BSD of your choice by running and debugging the current version. By the time a piece of code makes it into a release, it has been tested by thousands of users with countless different hardware configurations.

When a developer changes code for any BSD, the change is publicly available from the project's CVS servers within hours. You can synchronize your machine's source code with a master CVS server and recompile the system at any time. The source code for the entire system, including kernel and utilities, is maintained in an easily compilable manner. The entire operating system can be recompiled and reinstalled at any time. All the user needs to do is keep a copy of the system source code available and get the latest patches via CVS.

Before you can update your source code, however, be sure you have a full source tree for your BSD. The source code is usually under /usr/src. You can install the source code when you install the machine. If you're running FreeBSD, CVSup will automatically fetch and install the source code for you. Otherwise, you can generally ftp the source code from the 'Net, or copy it from the installation CD-ROM and uncompress it under /usr/src. A full source tree uses about 50 MB of disk space.

If you're unsure whether you need to upgrade, check out the online sidebar to this article (www.sysadminmag.com). This sidebar is a list of all BSD-related CERT advisories and pointers to security alerts for the various BSDs.

What is CVS?

CVS, or the Concurrent Version System, is a further development of RCS. CVS allows developers to keep a central repository of all code. Files can be checked out across the 'Net, edited and tested locally, and then committed back to the central repository for everyone's use. CVS is used by a variety of open source projects, such as Nessus. For more information on CVS, see "The Linux Kernel: A Case Study for CVS" in Sys Admin, Vol. 8, No. 6, or the CVS home page at:

http://www.csc.calpoly.edu/~dbutler/tutorials/winter96/cvs/

Sites lacking an IP Internet access, such as a UUCP feed, can look at CTM. CTM allows you to receive easily applicable source diffs via email. For more information on using CTM with a BSD, look at:

http://www.freebsd.org/handbook/synching.html#CTM
http://www.openbsd.org/ctm.html
http://www.nemeton.com.au/netbsd/ctm.html

The base distribution of all BSDs includes CVS. FreeBSD has an improved CVS called "CVSup." (CVSup is a successor to sup, or "Software Update Protocol," that includes CVS integration.) Both are used for sweeping CVS updates of large sections of source code. CVSup uses network resources far more efficiently than CVS does, however, and if you have FreeBSD it is highly recommended that you use CVSup. CVS is part of the base distribution of NetBSD and OpenBSD. You will need to install CVSup on a FreeBSD machine, however.

Installing CVSup on FreeBSD

The simplest way to install CVSup is by package. If you have a FreeBSD CD-ROM, just do:

mount -t cd9660 /dev/wcd0c /cdrom
cd /cdrom/packages/net/
pkg_add cvsup-bin-16.0.tgz

Your version number might differ, depending on the age of your CD-ROM. If you don't have the CD-ROM, you can get CVSup for FreeBSD 3.x with:

pkg_add
ftp://ftp.freebsd.org/pub/FreeBSD/packages/net/cvsup-bin-16.0.tgz

If you're a fan of rolling your own binaries from source, you can install the CVSup port. This takes quite a while to build. (Technically, CVSup itself builds rapidly. It is written in Modula-3, however, which does take time to build.) You can build the port by:

cd /usr/ports/net/cvsup
make install

If you have the CD-ROM containing the source code for CVSup mounted under /cdrom, it will copy the source tarballs automatically. Otherwise, it will fetch the sources automatically from a master ftp site.

Selecting your supfile

CVSup uses a config file, or "supfile". This tells CVSup exactly which files to update. Various supfiles are in /usr/share/examples/cvsup/ (or, on NetBSD and OpenBSD in /usr/share/examples/supfiles/). The supfile you use varies with the version of your particular BSD you want. If you want to track stable, use the stable-supfile. If you want to track current, use the standard-supfile. The files are well commented, with all the options described.

Looking at the uncommented lines of a recent FreeBSD stable-supfile, we see:

*default host=CHANGE_THIS.FreeBSD.org
*default release=cvs tag=RELENG_3
*default base=/usr
*default prefix=/usr
*default delete use-rel-suffix
*default compress
src-all

You'll need to edit the file. First, choose a CVSup server. A full list of FreeBSD CVSup servers is available at: http://www.freebsd.org/handbook/mirrors-cvsup.html. (NetBSD just uses sup.netbsd.org, while a list of mirrors for OpenBSD can be found at http://www.openbsd.org/anoncvs.html.) Use ping and traceroute to choose a mirror close to you.

The default release is a tag for the version, or collection, you've chosen. RELENG_3 is the tag for 3-stable. A full list of available tags for updating FreeBSD is available at:

http://www.freebsd.org/handbook/synching.html#CVSUP-COLLEC

The example supfiles for each BSD list all available tags. The default prefix is where the collection you've chosen will go. In this particular case, /usr is correct. The default base is where sup/CVSup will keep its status files. CVSup will keep a list of which files it has updated, allowing quicker future updates.

Both sup and CVSup have a variety of other options. Those normally specified for CVSup are delete, use-rel-suffix, and compress. If your connection is a T1 or faster, you can remove the compress setting. delete gives CVS permission to remove obsoleted source files. use-rel-suffix allows CVSup to share a common base directory among several versions of the source, without confusing them. These options cover almost all situations.

The src-all tells CVSup which part of the stable tree you are synchronizing with. A close look at the standard-supfile shows a whole list of available sub-collections, such as usr.bin, contrib, sys, and so on. You can choose to update only one section of your system (i.e., the who program, or the contents of /usr/sbin). This is not recommended. No BSD guarantees that random mixes of different versions of utilities will work together.

If you're interested in the ports collection, you might also add the line ports-all tag=. to your supfile. This will also update your ports tree. Likewise, if you want to update your system's cryptography and are in the United States or Canada, the tags src-crypto, src-kerberosIV, and src-secure might be useful. If you're outside the United States or Canada, you can fetch these collections from: cvsup.internat.freebsd.org. South Africans can fetch them from: cvsup2.internat.freebsd.org.

You will need to examine example supfiles for your particular OS; subtle differences from BSD to BSD make broad generalizations difficult. Once you've created a supfile, however, you can continue to use it forever.

Blocking CVSup Updates

You can choose to refuse to allow updates of certain programs. For example, you might not be interested in the ports for Vietnamese, Chinese, and German software. To tell CVSup not to bother updating these directories, make a file /usr/sup/refuse that looks like this:

ports/vietnamese
ports/chinese
ports/german

You can choose to do this with any section of your OS; however, you must know what you're doing. If you refuse updates to a program, that program might become incompatible with your system at some point.

Upgrade Preparation

If you intend to maintain your system at stable or current patch levels, you need to read the appropriate mailing list for your OS. All of the BSD teams announce major changes that require special handling. Device drivers, bootstrap methods, compilers, even binary types are subject to change through the lifetime of the OS.

For example, if you're tracking freebsd-stable, you should read stable@freebsd.org. Warnings of major changes are announced with a subject beginning "HEADS UP". OpenBSD and NetBSD maintain similar mailing lists. Watch for those messages, and take whatever steps they recommend.

Upgrading Source Code

Once you have a supfile, running CVSup is easy. Become root and do:

cvsup supfile

If you're running X, CVSup will open a GUI. Otherwise, it will just start upgrading your source files. Under OpenBSD, you will need to run:

setenv CVSROOT anoncvs@anoncvs.ca.openbsd.org:/cvs
cd /usr
cvs -q get -PA src

to install your initial source tree. To grab the latest patches, you need to run:

cd /usr/src
cvs -q up -PAd

NetBSD is more complicated, but the NetBSD project has made a script available to ease upgrades. For details, take a look at:

http://www.netbsd.org/Documentation/current/index.html#using_cvs

Building New Kernel and Userland

Once CVSup is complete, you can rebuild your system. Become root, cd /usr/src, and type:

make world (FreeBSD)
make build (NetBSD, OpenBSD)

Your machine will begin compiling all system utilities. On my desktop AMD K6 350 running FreeBSD-current, this takes about 2 hours. On my office firewall, a 25-Mhz 486 running FreeBSD 2.2.8-stable, it's about 12 hours. When make world finishes, you need to reconfig and recompile your kernel. The exact method varies from BSD to BSD, and from platform to platform. On an Intel FreeBSD box, do the following:

cd /sys/i386/conf
config KERNELNAME
cd ../../KERNELNAME
make depend && make all install && reboot

When your new kernel is complete, the system will reboot. You will now be running a fully upgraded system. This process can even be run out of cron, and updates will be taken care of automatically.

What Won't CVS and make world Do?

CVS only takes care of the base operating system. If you have separate programs such as shells, Web servers, or editors, you need to upgrade them manually. You can do this by fetching the latest package/port, running pkg_delete oldpackagename, and building the new package. CVS won't handle rewriting your personal kernel config file to accommodate system changes. Lastly, make world will not change the contents of your /etc directory. This is left to the user, for good reason.

/etc Changes

You also need to handle any changes in /etc. System startup scripts change from time to time as new functions are added. Samples of all files and directories in /etc can be found under /usr/src/etc. Compare any files there (particularly rc files) to those in your /etc directory, and see which changes you need to merge. The FreeBSD utility "mergemaster" can cut merge time considerably. Mergemaster is not part of the base system, but can be installed as a package or port in a manner much like CVSup.

Partial Updates

If you don't want to update your entire system, but only a piece of it, you can use CVS to grab the appropriate files. For example, if a new version of a utility comes out, and you wish to upgrade that utility without upgrading the entire OS, you can do that with CVS. Do this at your own risk; programs that are closely dependent on other parts of the system (top, vmstat, ps, etc.) will be very unlikely to work.

Before using CVS, you need to set an environment variable so that your CVS program will know which server to talk to. You can do this with:

setenv CVSROOT anoncvs@anoncvs.freebsd.org:/cvs

This is perfectly safe to set in your login script. Once you have your CVSROOT configured, you can check out the latest version of a program with:

cvs co <program name>

For example, to get the latest version of more, use cvs co more. A directory called more will be created in your current directory. You can cd more && make install. You can also use this to get the latest skeleton for a port, without upgrading your entire ports collection.

A cvs co modules will give you a complete list of all possible targets for a CVS operation. CVSup also supports partial updates with the -i pattern option. A cvsup my-supfile -i src/etc will update the /usr/src/etc directory using the default settings in my-supfile.

Conclusion

The BSD upgrade system makes it easy to maintain a large number of servers at the latest patch levels. Using the procedures outlined here, you can keep servers at the optimum OS reliability level, based on the use of the particular machine, and do so with relative ease. The ease with which you can upgrade open source BSDs using the tools described here, however, may create a desire to migrate from other Intel-based UNIX-like systems. Thus, you may want to be cautious about how much time you save, keeping practicality and popularity in the appropriate perspective. n

About the Author

Michael Lucas is an independent networking, security, and FreeBSD consultant. He previously worked for Verio, AGIS, and Oakland University. He lives in Detroit, Michigan with his wife Liz, four gerbils, and assorted fish. He can be reached at mwlucas@exceptionet.com.