Cover V13, i05
may2004.tar

Fanout -- Tools for Simultaneous Administration of Multiple Systems

J.D. Baldwin

Systems administrators frequently need to execute the same commands, edit a configuration file in the same way, or apply the same set of patches to several (or several dozen) machines at once. Of course, writing scripts to execute commands or using tools like sed and awk for file editing can lighten the load, but these methods have drawbacks: you do not get immediate visual feedback as when using vi directly, and you may not immediately see when a command does something unexpected. Also, some operations (e.g., the passwd command) require a terminal and, thus, cannot be run from a script at all. Too often, the complexities and risks of the scripting approach lead us to take the path of least resistance and simply visit each machine in sequence, doing the same work ten or fifty -- or even more -- times.

In this article, I will describe a set of free open source tools, collectively called "fanout", that Bill Stearns has written to address just this problem. The most basic tool in the suite, fanout, loops through a list of machines and, using ssh, executes a command or a script on each of them, displaying the output for your use. Fanout is a bash script and should run on any Unix variant supporting that shell (though some slight modifications may be required).

Fanterm is a more powerful tool for controlling multiple terminal sessions, all to different machines, at one time. Unfortunately, as of this writing it only runs from a Linux host (though the target machines can be any platform running ssh). I tested it on several Linux variants with no problems, but running it on Solaris or other commercial Unix variants is, at the moment, out of reach.

Installing Fanout

Fanout and fanterm are shell scripts and require no compiling or other building. Simply download the tarball from stearns.org (or one of its mirrors), extract it, and put the relevant scripts in one of the local bin directories in your PATH. The current version as of this writing is 0.6.1. The installation is as simple as the following (putting the relevant files in /opt/adm/bin):

[jd@dace] $ cd /tmp
[jd@dace] $ wget http://www.stearns.org/fanout/fanout-0.6.1.tar.gz
[download output]
[jd@dace] $ gzip -c fanout-0.6.1.tar.gz | tar xf -
[jd@dace] $ cd fanout-0.6.1
[jd@dace] $ cp fanout fanterm fanmux fanmux.sh /opt/adm/bin
There are also README, INSTALL, and other documentation files provided with fanout, which you should read and understand before proceeding.

SSH Key-Caching Review

Fanout and fanterm depend on ssh for their operation, and they will not be useful tools unless ssh keys and a key-caching agent are set up. For the purposes of this article, I will assume that the machines on which you need to run commands are running sshd, and that the machine from which you wish to connect to them has ssh and its associated utilities. If you need to install ssh, you can get a free, open version of OpenSSH (along with documentation) from openssh.org.

The following is a brief review of the basics of configuring a set of DSA keys so that one machine will "trust" another for the purposes of establishing ssh sessions. The ssh, ssh-keygen, and ssh-agent man pages, as well as the excellent book, SSH, The Secure Shell: The Definitive Guide by Daniel J. Barrett and Richard Silverman (O'Reilly & Associates), can provide much more detail.

There are two ways to use keys to authenticate to a host running sshd: by simply indicating the key file (or using the default), or by using the ssh-agent to cache your keys in memory during your login session. The former would require either that you establish key files with no passphrase or that you enter the passphrase every time you wish to establish a session, so I will deal with only the ssh-agent method here.

To begin, you will need to establish a key, which is accomplished by running the ssh-keygen command. To establish a 2048-bit DSA key (a highly secure configuration), simply run:

$ ssh-keygen -b 2048 -t dsa
The host will then prompt you for a file in which to save the private key:

Generating public/private dsa key pair.
Enter file in which to save the key (/home/jd/.ssh/id_dsa):
Hit Enter to accept the indicated default.

Next, you will be prompted for a passphrase. In almost any configuration, using ssh keys with empty passphrases is an extremely poor security practice, so we will provide (and confirm) our passphrase for this key. Of course, passphrases should be selected according to good password selection criteria (i.e., no dictionary words, etc.) and protected carefully. If you forget your passphrase, you can forget about ever recovering it from the private key file and you will have to start from scratch. Passphrases can be changed with the -p option of ssh-keygen (see the man page for details):

Enter passphrase (empty for no passphrase):  [.....]
Enter same passphrase again:  [.....]
Your identification has been saved in /home/jd/.ssh/id_dsa.
Your public key has been saved in /home/jd/.ssh/id_dsa.pub.
The key fingerprint is:
90:af:75:56:51:bb:91:fa:e9:ad:0f:d8:3a:93:dc:c9 jd@dace
"dace" here is the name of the host.

This operation will create two files in ~/.ssh: id_dsa (the private key file), and id_dsa.pub (the public key file). id_dsa should never be distributed, but id_dsa.pub (as the name implies) may safely be published even to untrusted hosts. If you examine these files, you will see that they are ASCII-fied versions of binary data, all on one line:

$ cat .ssh/id_dsa.pub
ssh-dss AAAAB3NzaC1kc3MAAAEBAPFKzjg19k1A+gCwGQ1NUa0z0v5oq/Z9MTECZWuT\
BlXHFpT/6McLtK9Kx//Qi+Irs2FeN4UUUGcOcmK/AwAlqmrBlFGxv5Q6U18MFmdCWNjg\
[and so forth]
The next step is to visit each of the hosts to which you will be logging in and appending the contents of id_dsa.pub to the .ssh/authorized_keys file. (This can be tedious, but you will only need to do it once per host.) To transfer the file, use the scp command:

[jd@dace] $ scp .ssh/id_dsa.pub parche:
The authenticity of host 'parche' can't be established.
RSA key fingerprint is
     c5:97:c2:5e:14:60:4d:b0:2c:9a:a2:3f:6e:0c:d1:8f.
Are you sure you want to continue connecting (yes/no)? yes
You will only get this "authenticity can't be established" message the first time you connect to a new host. Just answer "yes" if you are confident that you are truly connecting to the intended host; once you do so, you will be informed that the host's key has been added to your account's list for future authentication. You will then be prompted for your account password and the file will be transferred. At that point, log in (again, using your password) and append the key to the authorized_keys file:

[jd@dace] $ ssh parche
jd@parche's password:
Last login: Tue Feb  3 05:45:57 2004 from skate
Sun Microsystems Inc.   SunOS 5.9       Generic May 2002
[jd@parche] $ cat id_dsa.pub >> ~/.ssh/authorized_keys
[jd@parche] $ rm id_dsa.pub
[jd@parche] $ exit
Next, go to all of the hosts to which you would like to connect and perform the same operation: transfer the key with scp, then log in and append id_dsa.pub to .ssh/authorized_keys. For a lot of hosts, this will be a lengthy and boring operation, but you will only have to do it once.

Note that the most recent implementations of ssh use the authorized_keys file for all connections, but earlier versions may check authorized_keys or authorized_keys2 for ssh protocol 1 or 2 connections, respectively. If in doubt, just append id_dsa.pub to both authorized_keys and authorized_keys2. Also keep in mind that key-based authentication will work in default sshd configurations, but a given setup may not accept keys for authentication due to settings in the host's sshd_config file. See the sshd_config man page or Barrett and Silverman for help.

At this point, you may connect from dace to parche (in case you are wondering, all hostnames in this article refer to famous U.S. Navy submarines) without a password, using only the id_dsa key you generated above. However, you will still need to enter your key's passphrase every time you connect, so you have gained lots of security but no additional convenience. That is where the ssh-agent comes in.

ssh-agent and ssh-add

Think of ssh-agent as a daemon that runs once per login session. You start it once, and it generates output that sets some variables in your environment. ssh-add (despite its name) both adds and deletes keys from the cache maintained by ssh-agent. I will give only a rudimentary overview here, so I recommend consulting all of the aforementioned documentation sources to appreciate the full power of these utilities.

Note that, like all really powerful tools, ssh-agent entails some risk. Using it to cache your keys could result in compromise of critical hosts if an untrusted person should acquire root on a host while your keys are cached. Exercise caution when deciding where and when to make use of these utilities.

To begin this process, execute ssh-agent and use its output to set up your environment with the command:

[jd@dace] $ eval `ssh-agent`
Agent pid 17368
Next, use ssh-add with the private key filename to cache that key (you will, of course, need to enter the passphrase):

[jd@dace] $ ssh-add -T 3600 .ssh/id_dsa
Enter passphrase for .ssh/id_dsa:  [.....]
Identity added: .ssh/id_dsa (.ssh/id_dsa)
The -T 3600 option tells ssh-agent to cache the key temporarily, only for 3600 seconds (one hour). This is an important (but optional) security feature, as it is generally a bad idea to let a host cache your keys indefinitely. Unfortunately, not every ssh implementation (e.g., Solaris 9) supports the -T option.

Ssh-agent is capable of caching many keys simultaneously, so if you have different keys for different sets of hosts, you may load all of them at once into the same ssh-agent instance, and you may use ssh-add with the -d option to delete individual keys from the cache.

Now check that the key you think is cached really is the one cached with the -l ("list") option:

[jd@dace] $ ssh-add -l
2048 90:af:75:56:51:bb:91:fa:e9:ad:0f:d8:3a:93:dc:c9 .ssh/id_dsa (DSA)
This command reports the key length (2048 bits), the fingerprint (check it against the fingerprint reported when you generated the key), the filename, and the key type (DSA) of the cached key.

Keys may be deleted from the cache with the -d option of ssh-add (or -D to delete all cached keys). The agent itself can be killed by running eval on the output of ssh-agent -k (which will also unset the appropriate environment variables).

At this point, you should be able to log in to any host that recognizes this key as an authentication key without supplying a password or a passphrase. Just ssh to a host to test it:

[jd@dace] $ ssh parche
Last login: Tue Feb  3 06:22:38 2004 from dace
Sun Microsystems Inc.   SunOS 5.9       Generic May 2002
[jd@parche] $
And you are in! At this point, you have set up multiple hosts (viz., parche, houston, darter, growler, nautilus, and triton) in the same way, and you can begin using fanout and fanterm.

Fanout

Fanout simply runs a single command or simple set of commands, without interaction, on a series of hosts. Its syntax is simple:

fanout "<list of hosts>" "<command(s) to run>"
The list of hosts is simply a space-delimited list of hostnames; the list of commands (if more than one) must be separated by semicolons, just as for a single command line. Fanout will then send the commands to sessions on all named hosts in parallel (i.e., they will run simultaneously in the background). When the last session finishes, fanout will display the time taken to run the commands and present the outputs from each host sequentially.

Here is a simple example, assuming that all six hosts in the list accept the key generated and cached on dace:

[jd@dace] $ fanout "parche darter growler houston nautilus triton" "uptime"
Starting parche
Starting darter
Starting growler
Starting houston
Starting nautilus
Starting triton
Fanout executing "uptime"
Start time Tue Feb 3 13:58:24 EST 2004 , \
  End time Tue Feb 3 13:58:37 EST 2004
==== On parche ====
1:58pm  up 224 day(s), 1:30, 4 users, load average: 0.38, 0.20, 0.14

==== On darter ====
12:58pm  up 30 day(s), 23:59, 2 users, load average: 0.03, 0.05, 0.01
 
==== On growler ====
1:58pm  up 23 day(s), 5:50, 1 user, load average: 0.01, 0.02, 0.02
 
==== On houston ====
1:58pm  up 23 day(s), 2:41, 12 users, load average: 1.22, 2.24, 2.20
 
==== On nautilus ====
1:58pm  up 17 day(s), 18:59, 4 users, load average: 0.01, 0.01, 0.02

==== On triton ====
1:58pm  up 86 day(s), 18:39, 9 users, load average: 3.18, 3.01, 3.06

Exiting fanout, cleaning up...done.
That is pretty much all there is to fanout: a list of hosts, a set of non-interactive commands, and sequentially formatted output. It is powerful enough for many operations, but has obvious limitations. To launch multiple interactive sessions using ssh, use fanterm.

Fanterm

Fanterm is even simpler to use than fanout. From within a terminal window, simply execute the fanout script in a terminal window with a series of hostnames as the arguments. As with fanout, you must ensure that your public key has been propagated to all of the remote hosts concerned, and that your private key is cached on the local host. Figure 1 shows the result after the following command has been typed in the large window in the lower left of the screen:

[jd@dace] $ fanout parche darter growler houston nautilus triton
Fanterm automatically creates windows for each resulting login session, which you can see in Figure 1. To use fanterm, simply keep the main window (the one from which you launched fanterm) active and type your commands there. They will not echo in the main window, but will be echoed in each of the login session windows. Typing "uptime" followed by the Enter key in the main window yields the results in Figure 2 -- the command is executed on each of the remote machines and the results are displayed individually.

Each window may be scrolled back individually to examine previously generated output, but it is not possible to highlight just one of the remote session windows for the purpose of entering keystrokes into that session only.

The possibilities with fanterm are endless. For example, suppose we want to know which machines have ftp enabled or disabled in /etc/inetd.conf. Figure 3 shows the result of the appropriate grep command; ftp services are commented out on all hosts except growler and darter. If we wanted to comment them out on all hosts, it would be a simple matter to use vi with search and insert commands to prepend a # to the ftp lines on all of these hosts, then issue a pkill -HUP inetd on all of them simultaneously.

Figure 4 shows the result of the date command run on all hosts at the same time, just to verify that they are all in sync (since they all run the NTP daemon, they are all correct). It would be possible (if sub-optimal from a systems administration viewpoint) to set them all to the same time manually using the date command, but operations like this must be undertaken with care -- note that darter is on CST rather than EST, and factors like this must be taken into account when making changes across a large number of hosts.

Conclusion

This has been only the simplest introduction to the capabilities of fanterm and fanout. There are more options to be found in the fanout docs (supplied with the software) and, of course, systems administrators at any experience level can easily imagine further applications of these tools -- account maintenance, password resets, patch application, and many others that even Bill Stearns has not envisioned. The hardest part by far is the setup of ssh authentication and key caching, but that only needs to be done once.

As you use these powerful (and therefore dangerous) tools, keep in mind the pitfalls: the security implications of key caching, the care that must be exercised to ensure that commands entered with fanout and fanterm are truly appropriate and non-harmful to all hosts on which they are executed, and so forth. Properly used, these tools can increase productivity, both in the sense of more work accomplished in the same time, and in terms of reduced errors and greater consistency of administration across hosts.

I administer more than 75 hosts in locations around the world and estimate that in one year (around 2,000 work hours) of using fanout and other scripts depending on remote command execution over ssh, I've saved around 400 hours of repetitive work, and I've gained the benefit of absolute assurance that tasks are performed identically across diverse hosts. This article was written in the hope that you, too, might realize similar benefits.

J.D. Baldwin is a senior network security engineer for a large U.S. pharmaceutical corporation. He is a graduate and former faculty member of the U.S. Naval Academy; he earned an M.S. in Computer Science from the University of Maryland and graduated from the U.S. Naval War College. He lives in Michigan with his wife, children, and cats.