Tracking System Changes with Inventory Scripts

Ryan Ordway

Hardware changes, configuration changes, operating system patches, and software installations can radically alter the operation of a system. In any environment, documentation of system changes is not just a good idea -- it is a necessity. Tracking system and hardware information for Unix systems can be complex, however, especially when you are dealing with a large number of systems.

To overcome this challenge, I have written a small suite of inventory scripts to capture system configuration data programmatically. Why spend hours or more each week doing something that your systems can do for you?

The inventory suite consists of three individual scripts:

• do_inventory.sh -- This script determines which systems to query and generates the HTML data from the information returned by each of the other scripts.
• client_info.sh -- The client info script runs a handful of commands to output information about the system to be captured by the do_inventory.sh script.
• index_builder.sh -- The index builder script also runs a handful of commands, but outputs them as an HTML table row to be included in the inventory index page.

The do_inventory.sh script (Listing 1) is the master script (all Listings for this article can be found on the Sys Admin ftp site at: http://www.sysadminmag.com). The other two scripts are executed by the master script on each system via SSH to generate the inventory information as HTML output. The client scripts obtain the inventory information by running several informational commands. This data is captured by the master script and saved into HTML files.

The following sections describe these scripts in more detail and then give a sample of the output using an example network.

Initial Setup

The easiest way to use these scripts is to create a pseudo-user on each host named "inv". On each host in your network, run the following command:

useradd -c "Inventory User" -d /usr/local/inventory \
-s /bin/sh inv

ssh-keygen -t rsa

Of course, you can use any "master" user or "remote" user that you choose; just copy the "master" user's RSA key into the "remote" user's authorized_keys file for the RSA authentication to work properly. The "master" user on the central system will have unlimited shell access to the "remote" user on every host in your network; you should not run these scripts as root.

After the RSA keys have been propagated to all of your systems, you will need to create a hosts file that lists each host in the same format as /etc/hosts. This will be used to determine the hosts to connect to when you run the master script. Put this file in /usr/local/inventory/etc/hosts. Below is an example of this file:

# Hosts file for inventory suite
192.168.42.12     admin-server
192.168.42.14     ftp-server
192.168.43.76     web-server

ln -s /usr/local/inventory/html /usr/local/apache/html/inventory

http://your-server.your-domain.com/inventory/

The master script is responsible for determining which hosts are in the network, generating the inventory index page, and starting the client scripts on all of the hosts in the network. It then gathers the data that is returned by the hosts and places it in the appropriate location.

There are a few things that will need to be configured before running the do_inventory.sh script (Listing 1):

• SCRIPTDIR -- This is the location of the client script files. The default is /usr/local/inventory/bin.
• INVENTORY_BASE -- This is the directory where the HTML files will be stored. This should be the directory that was linked into your DocumentRoot. The default is /usr/local/inventory/html.
• HOSTS -- This is the location of your hosts file for the inventory. The default is /usr/local/inventory/etc/hosts.
• REMOTE_USER -- This is the user that you chose to use on the client hosts. The default is inventory.
• DO_MAIL -- This variable determines whether the contents of the index page should be embedded into an email message and emailed to you when the script completes. The default is 1, which enables this feature. Setting this to 0 disables the feature.
• MAIL_RCPT -- The To: address that will be used for sending the email message if DO_MAIL is enabled.
• MAIL_FROM -- The From: address that will be used for sending the email message if DO_MAIL is enabled.
• MAIL_SUBJECT -- The subject line that will be used if DO_MAIL is enabled.

The structure of the do_inventory.sh script is quite simple. If there is an existing index page, it is backed up before the script starts. The first portion of the index page is generated as a temporary index file. A list of hosts to use is generated from the hosts file and stored in the variable $NODES. For each entry in $NODES, the script cycles through the following steps:

• Create backup directory for system-specific HTML files.
• Back up an existing HTML file for that host.
• Run the client information script and store its output.
• Run the client index script and append its output to the temporary index file.

The first client script, client_info.sh (Listing 2), outputs comprehensive system information in HTML format. The data captured by this script is obtained by executing a number of diagnostic and other information commands. The standard output of this script is then captured by the master script and included in the system-specific client information HTML file.

One option that exists for the client script is for a list of third party packages that are not in the system package database. The script will output the contents of the file /var/sadm/softinfo/PACKAGES if it exists and include it under the heading "Third Party Packages".

The other option for this script is the level of patch information that you want included. The variable FULL_PATCH, when set to "yes", will include all of the showrev -p output. Otherwise, only the patch numbers themselves are printed.

Index Builder Script

After collecting the system-specific HTML data, the master script runs another script on the client, index_builder.sh (Listing 3). It then grabs some general data about the system and outputs it to stdout as an HTML table row. This is added to an HTML table in the main inventory index page. The hostname of the system is then linked to the system-specific HTML file generated by the first script.

One variable that might need to be configured is the URL_BASE variable. This is used to generate the link to the system-specific HTML file. If you linked the inventory html directory to something other than DocumentRoot/inventory, then you will need to update this variable to reflect the real location.

If you would like the kernel version information included with the operating system version, you can change the variable DO_KERNEL to yes.

There are two files that you can choose to create in the directory /var/sadm/system/admin. One file, named PURPOSE, can contain a single line with a short description of the purpose of the system. For example, a Web server might have "Web Server" in its PURPOSE file. The other file should be named SERIALNUMBER and contain, oddly enough, the serial number of the system.

A few of the functions in this script may need to be tweaked to fit your network. For example, the function get_hw_info() takes the output of uname -i and prints it in a friendlier format. If your system is not listed, or if you would like to change the names, just change the string that is returned.

Example

Our example network will contain three hosts: a master host named fred, and two client hosts named barney and wilma. After configuring the master host and pushing out the RSA keys and client scripts to both of our client hosts, we run the do_inventory.sh script on the master host. Figure 1 shows the hosts file, and Figure 2 shows the output from the do_inventory.sh script.

In the html directory, there will now be four HTML files, an index containing general information about all three of our systems, and a system-specific HTML file for each of our systems. Figure 3 shows a rendered version of our index page, and Figure 4 shows a rendered version of wilma's system-specific page.

Conclusion and Summary

Managing system configuration information for a large number of systems can be quite a chore. With a minimum of setup time, and the right software, your systems can do the hard work for you.

Besides juggling 100+ Sun servers for the Portland-based @Once, Corp., Ryan enjoys spending time with his wife, Stacy, and son, Andrew, and writing programs to allow him to spend more time with them. Ryan has been a Unix systems administrator for 5 years since his time lurking around Lewis & Clark College in Portland, Oregon.