Fanout
-- Tools for Simultaneous Administration of Multiple Systems
J.D. Baldwin
Systems administrators frequently need to execute the same commands,
edit a configuration file in the same way, or apply the same set
of patches to several (or several dozen) machines at once. Of course,
writing scripts to execute commands or using tools like sed and
awk for file editing can lighten the load, but these methods have
drawbacks: you do not get immediate visual feedback as when using
vi directly, and you may not immediately see when a command does
something unexpected. Also, some operations (e.g., the passwd
command) require a terminal and, thus, cannot be run from a script
at all. Too often, the complexities and risks of the scripting approach
lead us to take the path of least resistance and simply visit each
machine in sequence, doing the same work ten or fifty -- or even
more -- times.
In this article, I will describe a set of free open source tools,
collectively called "fanout", that Bill Stearns has written to address
just this problem. The most basic tool in the suite, fanout, loops
through a list of machines and, using ssh, executes a command or
a script on each of them, displaying the output for your use. Fanout
is a bash script and should run on any Unix variant supporting that
shell (though some slight modifications may be required).
Fanterm is a more powerful tool for controlling multiple terminal
sessions, all to different machines, at one time. Unfortunately,
as of this writing it only runs from a Linux host (though the target
machines can be any platform running ssh). I tested it on several
Linux variants with no problems, but running it on Solaris or other
commercial Unix variants is, at the moment, out of reach.
Installing Fanout
Fanout and fanterm are shell scripts and require no compiling
or other building. Simply download the tarball from stearns.org
(or one of its mirrors), extract it, and put the relevant scripts
in one of the local bin directories in your PATH. The current version
as of this writing is 0.6.1. The installation is as simple as the
following (putting the relevant files in /opt/adm/bin):
[jd@dace] $ cd /tmp
[jd@dace] $ wget http://www.stearns.org/fanout/fanout-0.6.1.tar.gz
[download output]
[jd@dace] $ gzip -c fanout-0.6.1.tar.gz | tar xf -
[jd@dace] $ cd fanout-0.6.1
[jd@dace] $ cp fanout fanterm fanmux fanmux.sh /opt/adm/bin
There are also README, INSTALL, and other documentation files provided
with fanout, which you should read and understand before proceeding.
SSH Key-Caching Review
Fanout and fanterm depend on ssh for their operation, and they
will not be useful tools unless ssh keys and a key-caching agent
are set up. For the purposes of this article, I will assume that
the machines on which you need to run commands are running sshd,
and that the machine from which you wish to connect to them has
ssh and its associated utilities. If you need to install ssh, you
can get a free, open version of OpenSSH (along with documentation)
from openssh.org.
The following is a brief review of the basics of configuring a
set of DSA keys so that one machine will "trust" another for the
purposes of establishing ssh sessions. The ssh, ssh-keygen, and
ssh-agent man pages, as well as the excellent book, SSH, The
Secure Shell: The Definitive Guide by Daniel J. Barrett and
Richard Silverman (O'Reilly & Associates), can provide much
more detail.
There are two ways to use keys to authenticate to a host running
sshd: by simply indicating the key file (or using the default),
or by using the ssh-agent to cache your keys in memory during your
login session. The former would require either that you establish
key files with no passphrase or that you enter the passphrase every
time you wish to establish a session, so I will deal with only the
ssh-agent method here.
To begin, you will need to establish a key, which is accomplished
by running the ssh-keygen command. To establish a 2048-bit
DSA key (a highly secure configuration), simply run:
$ ssh-keygen -b 2048 -t dsa
The host will then prompt you for a file in which to save the private
key:
Generating public/private dsa key pair.
Enter file in which to save the key (/home/jd/.ssh/id_dsa):
Hit Enter to accept the indicated default.
Next, you will be prompted for a passphrase. In almost any configuration,
using ssh keys with empty passphrases is an extremely poor security
practice, so we will provide (and confirm) our passphrase for this
key. Of course, passphrases should be selected according to good
password selection criteria (i.e., no dictionary words, etc.) and
protected carefully. If you forget your passphrase, you can forget
about ever recovering it from the private key file and you will
have to start from scratch. Passphrases can be changed with the
-p option of ssh-keygen (see the man page for details):
Enter passphrase (empty for no passphrase): [.....]
Enter same passphrase again: [.....]
Your identification has been saved in /home/jd/.ssh/id_dsa.
Your public key has been saved in /home/jd/.ssh/id_dsa.pub.
The key fingerprint is:
90:af:75:56:51:bb:91:fa:e9:ad:0f:d8:3a:93:dc:c9 jd@dace
"dace" here is the name of the host.
This operation will create two files in ~/.ssh: id_dsa (the private
key file), and id_dsa.pub (the public key file). id_dsa should never
be distributed, but id_dsa.pub (as the name implies) may safely
be published even to untrusted hosts. If you examine these files,
you will see that they are ASCII-fied versions of binary data, all
on one line:
$ cat .ssh/id_dsa.pub
ssh-dss AAAAB3NzaC1kc3MAAAEBAPFKzjg19k1A+gCwGQ1NUa0z0v5oq/Z9MTECZWuT\
BlXHFpT/6McLtK9Kx//Qi+Irs2FeN4UUUGcOcmK/AwAlqmrBlFGxv5Q6U18MFmdCWNjg\
[and so forth]
The next step is to visit each of the hosts to which you will be logging
in and appending the contents of id_dsa.pub to the .ssh/authorized_keys
file. (This can be tedious, but you will only need to do it once per
host.) To transfer the file, use the scp command:
[jd@dace] $ scp .ssh/id_dsa.pub parche:
The authenticity of host 'parche' can't be established.
RSA key fingerprint is
c5:97:c2:5e:14:60:4d:b0:2c:9a:a2:3f:6e:0c:d1:8f.
Are you sure you want to continue connecting (yes/no)? yes
You will only get this "authenticity can't be established" message
the first time you connect to a new host. Just answer "yes" if you
are confident that you are truly connecting to the intended host;
once you do so, you will be informed that the host's key has been
added to your account's list for future authentication. You will then
be prompted for your account password and the file will be transferred.
At that point, log in (again, using your password) and append the
key to the authorized_keys file:
[jd@dace] $ ssh parche
jd@parche's password:
Last login: Tue Feb 3 05:45:57 2004 from skate
Sun Microsystems Inc. SunOS 5.9 Generic May 2002
[jd@parche] $ cat id_dsa.pub >> ~/.ssh/authorized_keys
[jd@parche] $ rm id_dsa.pub
[jd@parche] $ exit
Next, go to all of the hosts to which you would like to connect and
perform the same operation: transfer the key with scp, then log in
and append id_dsa.pub to .ssh/authorized_keys. For a lot of hosts,
this will be a lengthy and boring operation, but you will only have
to do it once.
Note that the most recent implementations of ssh use the authorized_keys
file for all connections, but earlier versions may check authorized_keys
or authorized_keys2 for ssh protocol 1 or 2 connections, respectively.
If in doubt, just append id_dsa.pub to both authorized_keys and
authorized_keys2. Also keep in mind that key-based authentication
will work in default sshd configurations, but a given setup may
not accept keys for authentication due to settings in the host's
sshd_config file. See the sshd_config man page or Barrett and Silverman
for help.
At this point, you may connect from dace to parche (in case you
are wondering, all hostnames in this article refer to famous U.S.
Navy submarines) without a password, using only the id_dsa key you
generated above. However, you will still need to enter your key's
passphrase every time you connect, so you have gained lots of security
but no additional convenience. That is where the ssh-agent comes
in.
ssh-agent and ssh-add
Think of ssh-agent as a daemon that runs once per login session.
You start it once, and it generates output that sets some variables
in your environment. ssh-add (despite its name) both adds and deletes
keys from the cache maintained by ssh-agent. I will give only a
rudimentary overview here, so I recommend consulting all of the
aforementioned documentation sources to appreciate the full power
of these utilities.
Note that, like all really powerful tools, ssh-agent entails some
risk. Using it to cache your keys could result in compromise of
critical hosts if an untrusted person should acquire root on a host
while your keys are cached. Exercise caution when deciding where
and when to make use of these utilities.
To begin this process, execute ssh-agent and use its output to
set up your environment with the command:
[jd@dace] $ eval `ssh-agent`
Agent pid 17368
Next, use ssh-add with the private key filename to cache that key
(you will, of course, need to enter the passphrase):
[jd@dace] $ ssh-add -T 3600 .ssh/id_dsa
Enter passphrase for .ssh/id_dsa: [.....]
Identity added: .ssh/id_dsa (.ssh/id_dsa)
The -T 3600 option tells ssh-agent to cache the key temporarily,
only for 3600 seconds (one hour). This is an important (but optional)
security feature, as it is generally a bad idea to let a host cache
your keys indefinitely. Unfortunately, not every ssh implementation
(e.g., Solaris 9) supports the -T option.
Ssh-agent is capable of caching many keys simultaneously, so if
you have different keys for different sets of hosts, you may load
all of them at once into the same ssh-agent instance, and you may
use ssh-add with the -d option to delete individual keys
from the cache.
Now check that the key you think is cached really is the one cached
with the -l ("list") option:
[jd@dace] $ ssh-add -l
2048 90:af:75:56:51:bb:91:fa:e9:ad:0f:d8:3a:93:dc:c9 .ssh/id_dsa (DSA)
This command reports the key length (2048 bits), the fingerprint (check
it against the fingerprint reported when you generated the key), the
filename, and the key type (DSA) of the cached key.
Keys may be deleted from the cache with the -d option of
ssh-add (or -D to delete all cached keys). The agent
itself can be killed by running eval on the output of ssh-agent
-k (which will also unset the appropriate environment variables).
At this point, you should be able to log in to any host that recognizes
this key as an authentication key without supplying a password or
a passphrase. Just ssh to a host to test it:
[jd@dace] $ ssh parche
Last login: Tue Feb 3 06:22:38 2004 from dace
Sun Microsystems Inc. SunOS 5.9 Generic May 2002
[jd@parche] $
And you are in! At this point, you have set up multiple hosts (viz.,
parche, houston, darter, growler, nautilus, and triton) in the same
way, and you can begin using fanout and fanterm.
Fanout
Fanout simply runs a single command or simple set of commands,
without interaction, on a series of hosts. Its syntax is simple:
fanout "<list of hosts>" "<command(s) to run>"
The list of hosts is simply a space-delimited list of hostnames; the
list of commands (if more than one) must be separated by semicolons,
just as for a single command line. Fanout will then send the commands
to sessions on all named hosts in parallel (i.e., they will run simultaneously
in the background). When the last session finishes, fanout will display
the time taken to run the commands and present the outputs from each
host sequentially.
Here is a simple example, assuming that all six hosts in the list
accept the key generated and cached on dace:
[jd@dace] $ fanout "parche darter growler houston nautilus triton" "uptime"
Starting parche
Starting darter
Starting growler
Starting houston
Starting nautilus
Starting triton
Fanout executing "uptime"
Start time Tue Feb 3 13:58:24 EST 2004 , \
End time Tue Feb 3 13:58:37 EST 2004
==== On parche ====
1:58pm up 224 day(s), 1:30, 4 users, load average: 0.38, 0.20, 0.14
==== On darter ====
12:58pm up 30 day(s), 23:59, 2 users, load average: 0.03, 0.05, 0.01
==== On growler ====
1:58pm up 23 day(s), 5:50, 1 user, load average: 0.01, 0.02, 0.02
==== On houston ====
1:58pm up 23 day(s), 2:41, 12 users, load average: 1.22, 2.24, 2.20
==== On nautilus ====
1:58pm up 17 day(s), 18:59, 4 users, load average: 0.01, 0.01, 0.02
==== On triton ====
1:58pm up 86 day(s), 18:39, 9 users, load average: 3.18, 3.01, 3.06
Exiting fanout, cleaning up...done.
That is pretty much all there is to fanout: a list of hosts, a set
of non-interactive commands, and sequentially formatted output. It
is powerful enough for many operations, but has obvious limitations.
To launch multiple interactive sessions using ssh, use fanterm.
Fanterm
Fanterm is even simpler to use than fanout. From within a terminal
window, simply execute the fanout script in a terminal window with
a series of hostnames as the arguments. As with fanout, you must
ensure that your public key has been propagated to all of the remote
hosts concerned, and that your private key is cached on the local
host. Figure 1 shows the result after the following command has
been typed in the large window in the lower left of the screen:
[jd@dace] $ fanout parche darter growler houston nautilus triton
Fanterm automatically creates windows for each resulting login session,
which you can see in Figure 1. To use fanterm, simply keep the main
window (the one from which you launched fanterm) active and type your
commands there. They will not echo in the main window, but will be
echoed in each of the login session windows. Typing "uptime" followed
by the Enter key in the main window yields the results in Figure 2
-- the command is executed on each of the remote machines and the
results are displayed individually.
Each window may be scrolled back individually to examine previously
generated output, but it is not possible to highlight just one of
the remote session windows for the purpose of entering keystrokes
into that session only.
The possibilities with fanterm are endless. For example, suppose
we want to know which machines have ftp enabled or disabled in /etc/inetd.conf.
Figure 3 shows the result of the appropriate grep command;
ftp services are commented out on all hosts except growler and darter.
If we wanted to comment them out on all hosts, it would be a simple
matter to use vi with search and insert commands to prepend a #
to the ftp lines on all of these hosts, then issue a pkill -HUP
inetd on all of them simultaneously.
Figure 4 shows the result of the date command run on all
hosts at the same time, just to verify that they are all in sync
(since they all run the NTP daemon, they are all correct). It would
be possible (if sub-optimal from a systems administration viewpoint)
to set them all to the same time manually using the date
command, but operations like this must be undertaken with care --
note that darter is on CST rather than EST, and factors like this
must be taken into account when making changes across a large number
of hosts.
Conclusion
This has been only the simplest introduction to the capabilities
of fanterm and fanout. There are more options to be found in the
fanout docs (supplied with the software) and, of course, systems
administrators at any experience level can easily imagine further
applications of these tools -- account maintenance, password resets,
patch application, and many others that even Bill Stearns has not
envisioned. The hardest part by far is the setup of ssh authentication
and key caching, but that only needs to be done once.
As you use these powerful (and therefore dangerous) tools, keep
in mind the pitfalls: the security implications of key caching,
the care that must be exercised to ensure that commands entered
with fanout and fanterm are truly appropriate and non-harmful to
all hosts on which they are executed, and so forth. Properly used,
these tools can increase productivity, both in the sense of more
work accomplished in the same time, and in terms of reduced errors
and greater consistency of administration across hosts.
I administer more than 75 hosts in locations around the world
and estimate that in one year (around 2,000 work hours) of using
fanout and other scripts depending on remote command execution over
ssh, I've saved around 400 hours of repetitive work, and I've gained
the benefit of absolute assurance that tasks are performed identically
across diverse hosts. This article was written in the hope that
you, too, might realize similar benefits.
J.D. Baldwin is a senior network security engineer for a large
U.S. pharmaceutical corporation. He is a graduate and former faculty
member of the U.S. Naval Academy; he earned an M.S. in Computer
Science from the University of Maryland and graduated from the U.S.
Naval War College. He lives in Michigan with his wife, children,
and cats.
|