Cover V12, I10

Article

oct2003.tar

Using a Distributed Shell to Manage Deployments of Linux and Unix Servers

Jeff McElroy

Linux and Unix servers have proliferated across many corporate networks. Some deployments have been well organized, but many have happened slowly and haphazardly. Unsurprisingly, many of these deployments lack organization and standardization. In smaller deployments, an administrator can manage each server individually, but as the number of servers increases this approach becomes awkward and additional tools must be utilized.

There is no standard way to maintain middle- to large-sized deployments. Instead, an administrator puts together a hodgepodge of solutions including SNMP, syslog, shell scripts, webmin, or more recently, the Red Hat Network. This article introduces distributed shell commands to this collection. A distributed shell is a good first step in standardizing your deployment because it provides a mechanism for you to view and manage your deployment as a single system.

Distributed Shell Description

A distributed shell allows a user to have a shell command run simultaneously on a large number of servers. The output from these commands is returned to the user's shell with the name of his source login and server pre-pended to each line. For instance, the command dssh hostname would give the following output:

root@japan.radux.com   japan
root@italy.radux.com   italy
root@sweden.radux.com  sweden
root@germany.radux.com germany
Other, more useful commands can be issued just as readily. For instance, if you want to locate any server with a user currently logged in, you could issue the command dssh users. This is the same in function as the command rusers, although the output is a bit different. The commands executed by dssh do not need to return immediately. For instance, the command dssh tail -f /opt/apache.instance/logs/access_logs would never exit, but its output could be used to monitor the logs of many Web servers in a simple way.

The best use of a distributed shell is often as a building block for more powerful scripts. For example, you could quickly write a script that would check and restart any frozen or malfunctioning mail server process. If you added that script to your crontab, you would have a centralized form of automated recovery. A side benefit to using a distributed shell through other scripts is that you are somewhat protected from typos that occur when performing ad hoc commands.

The DCMD Package

There are many implementations of a distributed shell. The implementation used for this article can be found at http://dcmd.sourceforge.net. This is a simple (less than 200 lines of scripts and no binaries) implementation based on OpenSSH. In addition to the dssh command referred to above, dcmd provides several other commands and utilities. For the transfer of files or the maintaining of identical directory structures across your servers, dcmd provides the commands dscp and drsync. The command dscp is to scp what dssh is to ssh; it allows you to easily transfer a file to or from all of your servers at once. For example, if you want to transfer the subdirectory /opt/newApplication to all of your servers, you could do so by issuing the command dscp -rp /opt/newApplication []:/opt/newApplication. The [] characters will expand to the login@hostname used to contact each server. The above example might expand into the following scp commands:

scp -rp /opt/newApplication root@japan.radux.com:/opt/newApplication
scp -rp /opt/newApplication root@italy.radux.com:/opt/newApplication
scp -rp /opt/newApplication root@sweden.radux.com:/opt/newApplication
scp -rp /opt/newApplication root@germany.radux.com:/opt/newApplication
The dscp command can also transfer files from a group of servers back to your workstation. This can be done with the command dscp -rp []:/var/log []. After running this command, a copy of the /var/log directory from each server will be located in your current working directory. Each retrieved /var/log directory would be named after the login and hostname used to contact the server (i.e., login@hostname).

The drsync command is similar in function to dscp, but unlike scp, rsync attempts to transfer only the differences in the source and destination file sets. This makes it a better command for maintaining large or remote file sets. See the Web site http://rsync.samba.org for more details.

The dcmd package also provides several other commands. They are dps, which runs the ps command on each server; dfind, which runs the find command on each server; and dusers, which is a synonym for the rusers example above. These commands are all one-line wrappers for the command dssh. The commands dssh, dscp, and drsync are in turn one-line wrappers for the command dcmd, after which this package is named.

Setup of dcmd

Download the dcmd tarball from http://dcmd.sourceforge.net and extract it into a convenient destination directory on your workstation. For this article, I will put it into "/opt/dcmd". Add the directory "/opt/dcmd/bin" to your path. If you execute the command which dssh, it should report a path of /opt/dcmd/bin/dssh. Do not run any of the dcmd commands until after you have set up OpenSSH.

Next, edit the /opt/dcmd/etc/dcmd.hosts file. Put the login and hostname for each server you want to manage into this file. The format for this file is "login@hostname". Comments are allowed in this file and should begin with a pound sign (#).

The name and location of the dssh.hosts file is configurable by setting the environment variable DCMD_HOSTS. Examine the /opt/dcmd/bin/dcmd_getHostsFile script for details. By creating several host files and setting this environment variable appropriately, you can easily manage different groups of servers.

Setup of OpenSSH

OpenSSH should be installed and running on all the servers you want to manage. If you already have OpenSSH configured to allow you to log into each server without requesting a password, you can safely skip this section. If you are unfamiliar with OpenSSH, visit the Web site (http://www.OpenSSH.org) for background information and installation instructions. You will need to generate and distribute encryption keys.

There are two sets of keys with which to be concerned: the host keys and the user keys. Both sets of keys are asymmetric, which means that they consist of a private key and public key. The public keys are the ones that will be distributed.

Host keys are used by OpenSSH to validate that you are connecting to the proper machine. They provide protection from man-in-the-middle attacks. These keys are usually generated when OpenSSH is first installed (see the OpenSSH documentation). Whenever your OpenSSH client connects to an OpenSSH server, it verifies this key. If you are connecting to the server for the first time, your client will tell you that this is a new key and ask you whether it is valid. You will not be able to respond to this query while dcmd is running because dcmd runs OpenSSH in the background, and OpenSSH is not in control of your terminal. Therefore, these host keys must be verified and saved on your workstation prior to running any dcmd commands.

User keys are used by OpenSSH to authenticate your client to the remote server. Unlike the host keys, you must generate these keys manually, which can be done with the command ssh-keygen -t dsa. This command will first prompt you for a filename. Use the default ~/.ssh/id_dsa. The next two prompts will ask you to enter and verify a password. You have two options. The first is to just press enter twice and leave your private key un-encrypted. Use this option only if you are sure your home directory is safe from prying eyes. If you choose to encrypt your private keys, you will need to use ssh-agent (see the ssh-agent man page for details). The easiest way to do this is to define the environment variable DCMD_USE_SSH_AGENT. When this is defined, the dcmd script will re-launch itself under ssh-agent and execute the command ssh-add. ssh-add will ask for your user key's password and manage your keys for you. If you already use ssh-agent and don't want dcmd to request a password every time you run a dcmd command, leave the variable undefined and OpenSSH should operate fine.

The dcmd package contains a script that can simplify the distribution of public keys. This script appends your user public key to the end of the "authorized_keys" file on your servers, which allows the server to authenticate who you are, without requesting a password. This script will also collect each server's host key for you. To use this command, type "dcmd_distributeKeys ~/.ssh/id_dsa.pub /root/.ssh/authorized_keys" where "~/.ssh/id_dsa.pub" is the location of your public key and "/root/.ssh/authorized_keys" is the location of the authorized_keys file on your servers. You will be prompted for the password to each server listed in your dcmd.hosts file as dcmd_distributeKeys works.

Caveats

There are a few possible issues to be aware of. As the number of servers listed in dcmd.hosts increases, so does the utilization of system resources on your workstation. You will have many run-able OpenSSH processes waiting for CPU resources and will probably see a load average in the double digits. dcmd does not mind this, but the performance of other applications will suffer. If dcmd has 30 run-able processes and your Web server only has 3, dcmd will take up 10 times more of your CPU resources than your Web server. It is best not to run a dcmd command from a machine on which you run other production applications, especially ones that are response-time sensitive. If you are managing several hundred servers at once, you may also run into other system resource issues, such as the number of file descriptors available to the kernel. These limits can be easily increased, but you may need to do so manually.

Conclusion

Implementing a distributed shell can simplify an environment and save time for an administrator who is tasked with maintaining many servers. It is easy to set up, and builds upon the scripting skills that most administrators already possess.

Jeff McElroy is a consultant based in Sioux Falls, South Dakota. He specializes in Java, C/C++, Perl, AIX, and Linux and has more than 10 years working experience. Jeff has a bachelor's degree with majors in Chemistry, Math, and Psychology. He recently started focusing on providing single sign-on and unified directory services for heterogeneous networks. Jeff can be contacted at: jmcelroy11@sio.midco.net.