Article
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Sidebar
Table 1
Table 2
Table 3
Table 4
Table 5
Table 6

sep2004.tar

Using LSD to Sync a Solaris Environment

Justin Buhler

Currently, my team and I manage more than 400 Sun servers for the Athens 2004 Olympic Summer Games. From previous events (see Problem Profile sidebar), I learned that we had a definite need for a centralized Unix software distribution utility. In this article, I will show how we have used the versatility of an LDAP directory, vendor's standard packaging, and a little Perl to achieve centralized and secure software distribution.

Architecture Overview

We often forget to keep things simple when developing a solution to solve a particular problem. But, keeping it simple is the philosophy behind the LSD (LDAP Software Distribution) framework we created for our environment. This framework is made of the five components shown in Table 1. There are some additional pieces that provide a little more security in the overall process, such as MD5sum(s) and the SSH framework for transferring files (SCP).

Let's start with an example of how LSD works before jumping into the details of the LSD framework. Figure 1 shows two examples of the framework in action. These examples are based on listing (auditing) missing packages and then fixing (correcting) the missing packages.

Now let's look at some of the nuts and bolts of the LSD framework and see how you can use the tools to sync your Solaris environment.

LSD Packages

The Solaris environment has a standard package format, which comes with its own software management framework, which includes version control and auditing. Richard Elling has written a great series of articles that cover the management of Solaris systems using Solaris packaging. (See the references.)

The LSD framework is built completely around the Solaris package format and its pkginfo file. Therefore, you must have a very good understanding of the Solaris packaging format and requirements in order to really master the concept of LSD.

Table 2 shows the key points that are required by LSD framework regarding the use of the pkginfo file when building software to be managed by LSD. The package factory is where all packages are built and tested. All of our packages have standard directory structure and consistent build process using a Master Makefile. The result of the build process is a compressed software package with the following "mandatory" naming convention.

LSD Software Package: LSDtop-1.0.pkg

The software factory and structure are shown in Figure 2.

The package depot is the only important element for the LSD framework; however, the package factory should be managed diligently since you must trust the package built from it. For further information on building Solaris packages, see the Application Developers' Guide. See Figure 3 for an example of the LSD package factory.

LSD Agent

The LSD Agent is where all the "magic" occurs and was developed using Perl and some additional Perl modules. I have tried to make it as functional and reusable as possible. Below is a table of all the Perl modules required by the LSD framework, and it should be installed on all of the client servers. All the Perl modules not already included with the standard Perl distribution listed in Table 3 or the LSD framework are available for download from CPAN. The Perl version I used is Perl-5.8.3 from www.sunfreeware.com.

The Agent script, lsdagent.pl (see Listings 1 and 2), contains the options shown in Figure 4. The definition to each option of the lsdagent.pl script is shown in Table 4.

The Agent checks the LDAP Registry and then the Agents Registry in /var/sadm. The Agent considers the registry the baseline for all software that should be installed on the client server. It then reports the status of the Agent if the -list option has been selected (see Figure 5).

Agent Configuration Overview

To make the Agent a little more portable, I have tried to keep it from being environment-specific. This is because the tool needs to be used for other projects, which may have different naming conventions. Thus, there is a configuration section at the beginning of the script, which allows for customization of the Agent.

The Agent uses the operating systems commands /usr/sbin/pkgadd and /usr/sbin/pkgrm to perform all of the installations. The Agent will only install software, which is identified by the registry as being approved for the host based on the values in the [lsdhost] attribute. The comparison of the rules is done by the Agent, so you can determine the identifier the Agent uses to approve software to be installed. See the lsdagent.pl configuration section for details.

The following are guidelines for the Agent to successfully interact within the LSD framework:

1. You must have a local account configured on the Agent and Depot servers for downloading software from the registry. This account is used to retrieve software across the LAN or WAN using SCP and Public-Key Authentication.

2. You must generate SSH public and private keys without a passphrase, using the locally created account, which will be used to download software from the registry. The private key must be installed on all systems, whereas the public key will only need to be installed into the depot server:

## LSD Agent Depot Account
my $lsduser = 'softdist'; # Local Account
my $lsdkey = "/home/$lsduser/.ssh/id_rsa"; # Private Key
my $lsdlocaldepot = "/home/softdist/data"; # Local Package spool directory

3. You must have permission to read the packages on the depot server. This is achieved by creating the identical local account on the depot server, using the same UID and GID.

4. You must configure the correct CATEGORY the agent should look for. You need to separate each category with a space:

## LSD Package PKGINFO Categories
## Package category used to find
## software locally in /var/sadm.
####################################
my @lsdcategories = qw(LSD);

LSD Registry

The Registry is an LDAP directory with each entry representing a software package. The LDAP server that we are using is SUNONE 5.2, which allows for easy customization of the schema. The schema consists of the attributes/value pairs found in a standard Solaris package. The schema is also configured to include some additional attribute/value pairs for the LSD application. Table 5 contains the additional LSD value names and descriptions.

Each LDAPAttribute value is populated by the information found within the Solaris package (pkginfo -l ). All of this additional information is gathered during the software registration process using the lsdregsitry.pl script (Listing 3).

Registration is the process in which you take a tested package and update the registry with the packages information (see Figure 6):

-pkg [/path/to/the/software/pkg-x.x.pkg] -- Package to be registered or modified in the LSD registry

-version -- Version of the script

-debug -- Print debugging information for troubleshooting

I have made the lsdregistry.pl command portable by including a configuration section at the beginning of the script, which allows for customization of the registry software. This must be completed before using the script to register software.

Here are some guidelines for successfully registering software within the LSD framework:

1. Review and complete configuration section of the lsdregistry.pl command.

2. You must have an account with add/modify permission on the LDAP directory. The lsdregistry.pl command prompts for your user name and password.

3. The software package "must" have all fields completed correctly.

Figure 7 provides an example of registering the SLBtop-2.0.pkg software package. Figure 8 is the schema definition, which is used to create our LSD registry, and Figure 9 shows an example LDAP entry.

Security Note

One of the principle reasons for having the registry is that it is considered a global trusted reference. We trust the registry to provide correct information regarding the version, location, and integrity (md5) for the packages that are destined for the client system. Therefore, the registry must be secured correctly. LDAP security is outside the scope of this document. However, here are some keys points regarding to the LSD framework:

lsdagent.pl requires LDAPDN and Password to BIND/SEARCH.
lsdregistry.pl requires LDAPDN and Password -- BIND/ADD/MODIFY.

Optional items that could further enhance the security:

Disable anonymous access to listing the contents of packages.
Add TLS authentication to lsdagent.pl.

LSD Depot

The Depot consists of "ready" or "tested" versions of the LSD software that are located centrally on the network for the LSD agents to download using SCP. Our current Depot server is a Sun Fire 280R configured with some external storage for additional capacity.

A package must be registered into the LSD registry in order to be made available for the environment. The Agent uses the "lsdlocation" field from the software entry to locate the package on the network first. It tries to verify the IP address is up by pinging it. If successful, the Agent uses SCP to download the software package (see Figure 10).

Guidelines for the Agent to successfully interact with the Depot:

You must have the same software distribution account locally configured.
You must have SSHD listening.
You must configure the local account with the agent's public key.
The Agent/Local account must have permission to read the packages.

LSD Security Issues

Now we have this great framework, but is it secure? The answer is yes, but you have options. Table 6 lists some of the items we have implemented and some options that could be implemented depending on your own level of acceptable risk. Everyone's security requirements are different, but the options let you customize the framework for your needs.

Configuring Restricted Shell on the Depot Server

In the /etc/passwd file on the Depot Server, the following must be configured for the softdist account:

HOMEDIR -- /Software
SHELL -- /usr/lib/rsh (restricted shell)

Next, move the /home/softdist/.ssh directory containing the public key of the softdist account to the new HOMEDIR location and remove the old location.

Create the .profile with owner and group matching the softdist account in the HOMEDIR location. Set and export the PATH environment variable to PATH=/Software/bin.

Create the /Software/bin directory. If you want to add some functionality to the "softdist" account, create the necessary links to the commands you want to allow.

Everything under /Software must have the correct owner and group permissions:

Directories -- find /Software -type d -exec chmod 750 {} \;
Files (Packages) -- find /Software -type f -exec chmod 640 {} \;
BaseDir (/Software) -- chmod 755 /Software
Ownership -- chown -R root:softdist

This allows the softdist account using public key to collect the software, but have access to do anything.

Conclusion

The following are some keys points that we worked through when evaluating our software distribution framework:

1. Our main objective was to keep the solution simple.

2. We needed a centrally available and easily maintainable registry for storing information regarding packages.

3. The system needed to be a pull process and not a push process because environments are not guaranteed to be online when updates occur.

4. The framework needed to use the vendor's software format.

5. The installation process needed to be clearly understood and simplified. For example, operating system software on the installation CDs and LSD packages would equate to a complete image.

I hope this article has shown how, with the power and versatility of LDAP and the magic of Perl, you can easily build solutions to meet everyday issues, which in the end improves your productivity and manageability.

Wish List and References

I hope to include the following items in the next version of the tool:

Web-based management and report
Rollback functionality on the Agent using the perversion attributes
Allowing agent to have an alternate version for the host using altversion attributes
Add a -generate option to create lid file for LDAP server installation
Place configuration parameters into a configuration file such as lsd.properties
Convert to a Java application

References

Elling, Richard. Rapid Recovery Techniques: Auditing Custom Software Configurations. Sun BluePrints. February, 2000.

Elling, Richard. Rapid Recovery Techniques for the Solaris Operating Environment. Sun BluePrints. January, 2000.

Elling, Richard. Rapid Recovery Techniques: Repairing File Ownership and Mode. Sun BluePrints. December, 1999.

Elling, Richard. Rapid Recovery Techniques: Exploring the Solaris Software. Sun BluePrints. October, 1999.

Application Packaging Developer's Guide -- http://docs.sun.com/db/doc/817-1778

Sun ONE Directory Server 5.2 Ref. Manual -- http://docs.sun.com/source/816-6699-10/schemaov.html

CPAN -- http://search.cpan.org/

SunFreeware -- http://www.sunfreeware.com

Justin Buhler is the Senior Unix Administrator for Major Events, Atos Origin. His current projects include Athens 2004 Summer Olympics and Torino 2006 Winter Olympics. You can contact him at: jbuhler@torino2006.it or jpbuhler@athens2004.com.