Conserver: A Flexible, Mature Console Management System

Bryan Stansell

How would you like the flexibility to troubleshoot system problems from anywhere? How would you like the ability to have others watch what you are doing, and even help? And how would you like to have everything logged, so you can "go back in time" and review past events?

These are just some of the things conserver can do for you. Conserver is a daemon that runs on a Unix-like system to facilitate remote management of devices. The only prerequisite to taking advantage of conserver is the ability to connect to the "console" of the device you wish to manage. This "connection" can be via a direct serial connection, a TCP socket, or any command conserver can invoke. A TCP socket is traditionally used to connect to a terminal server, whereas random commands allow you to do anything -- from connecting to proprietary systems to integrating into an IPMI environment. The definition of "console" is really left up to your imagination.

Once connected, conserver logs everything displayed on the console of each device. This allows administrators, for example, to go back and see any error messages sent to a console immediately before a system crash, which is an enormous value when debugging problems remotely. Conserver also manages administrators accessing the consoles, entering commands, sending break signals, etc. It even supports multiple administrators accessing the same console simultaneously.

A Little History

Conserver was first publicly announced at LISA IV in 1990 [1]. Back then, it could only attach to local serial ports, had no concept of distributed conserver hosts, and had only the most basic command set. This simple, yet powerful, concept attracted the attention and help of others, and conserver grew into a fully distributed, command-rich, flexible piece of software by the end of 1993 [2].

It wasn't until 1995 that I actually used conserver for the first time. I was immediately hooked. Once I had the ability to go through console logs, share console connections with others (to mentor as well as be mentored), and respond to issues without running to the data center or driving in from home, I could never go back.

Like many pieces of software at that time, there was no central distribution of conserver. Conserver itself was actually becoming stagnant. The code we were running was a combination of the original work, patches posted to Usenet, and our own bug fixes and enhancements. Because of my interest in the code, I became the in-house maintainer of conserver. I took that responsibility to my next job and continued distributing the code to my circle of friends.

In 1999 I registered conserver.com, not only to make it easier for the folks I knew to get the software but also to offer it to the world. There had been a lot of work done on the code and since it didn't look like anyone else was actively maintaining it, I happily took on the task. Additionally, I created mailing lists for announcements and user support, which have drawn a great community of helpful and creative individuals.

The "original" (but heavily modified) version of the code continued until the 7.2.7 release in April, 2003. Then the "big rewrite" happened and, in September of 2003, version 8.0.0 was released.

It wasn't a total rewrite of code, but major portions were reworked. The configuration file parser was replaced, a generalized I/O subsystem was created (specifically to support SSL and buffer failed writes), the POSIX standard was adopted, and the client-server protocol extended, just to name a few. The result was what everyone had wanted for years: a tool that was flexible, reliable, and able to grow with the future.

What Can Conserver Do?

For me, any console management system should be able to do two basic things: log the console stream and allow multi-user access to the console. Anything else is just bells and whistles (although many things will become a personal requirement once you use them). Conserver is full of bells and whistles. After 15 years, it should be.

There are three major aspects of conserver to look at. The first is building the software with optional capabilities, since that determines how conserver will behave in your environment. The second thing is the configuration file, since most capabilities are enabled or disabled at runtime based on that file. The third, and perhaps the most important aspect, is the end-user experience -- using the conserver client.

Building Conserver

Conserver has been run successfully in a variety of Unix-like environments. This includes the Cygwin environment for Windows. My goal is to be able to run in any Unix-like environment that the GNU configure utility supports. I'm not sure if that will ever happen, but I do know all the common vendors work just fine.

The first step (aside from reading the documentation) is to run configure. You can use all the standard GNU configure options as well conserver-specific ones. Aside from overriding file locations, you can tell conserver to compile in support for PAM (Pluggable Authentication Module), tcp_wrappers (for further access restrictions), OpenSSL (to encrypt and authenticate client-server communication), and Unix domain sockets (to have all client-server communication contained on a single host).

There are also a couple of compile-time tuning options. The first (--with-maxmemb) tells conserver how many consoles to manage per-process. Because you could have thousands of consoles managed by conserver, it splits up the workload into groups and hands each group to a sub-process. This was mainly done to prevent a single process from running out of file descriptors, but it also allows for a certain level of parallelism. The second tunable option is the TCP connection timeout value (--with-timeout). If a console is defined as a direct TCP connection and the remote host is not responding, the system default timeout can be much longer than you would like. Conserver will abort any attempt that reaches the timeout and mark the console as "down". This happens without blocking, so all other consoles managed by that process can still be used.

The Configuration File

Up through version 7.2.7 the configuration file is very basic. It is similar to a standard password file -- a colon-separated list of items, one per console, with common information replicated throughout the file.

With version 8.0.0, the configuration file format became very powerful, especially for such a simple concept as a console server. It has been formatted into blocks of information, with an "include" facility to allow for inheritance of information and elimination of redundant data. These blocks of information are console definitions (of course), sets of named defaults, runtime settings, access lists, groups of users, and break strings (similar to macros). All blocks of information are of the form:

type name {
  key value;
  key value;
  ...
}

console simple {
  master localhost;
  type exec;
  rw *;
}

Another console option is initcmd, which specifies which command should be attached to the console when it first comes "up" (i.e., when conserver successfully connects to the console). This allows you to authenticate to a terminal server or do any other activity required to get to the point where you are fully connected to the console. There is also a timer (initspintimer) and counter (initspinmax) to catch rapid successive failures of initcmd and stop conserver from thrashing.

Aside from protecting itself, conserver can also protect you by doing port calculations internally. This allows you to do things like specify the physical port number in the console definition and let conserver calculate the TCP port number or even local serial port name (using the devicesubst feature, which is described below). That can be quite handy when you are on the phone with a data center technician and want to say what physical port a machine is plugged into. Conserver uses the portbase, portinc, and port options to do the calculation.

Terminal servers use various TCP port number sequences for exposing their serial ports. Cisco devices typically use the formula 2000 + port, while old Xyplex devices use 2000 + 100 * port. Vendors can also vary this depending on whether you are connecting to telnet-protocol ports, ssh-protocol ports, or "raw" ports. Conserver uses the formula portbase + portinc * port to determine the final port number, where portbase defaults to 0 and portinc defaults to 1. For example, say you wanted to connect to port #4 of a Xyplex terminal server. If you set portbase to 2000 and portinc to 100, conserver would calculate the appropriate port when you used port 4; in the console definition.

Console items, when fully specified, end up having a lot of common information. This is where sets of defaults come in handy. They allow you to take all the common information and combine it into logical groups, which you can then use in a console definition (or other groups). You can group defaults in default items, which take the same key-value pairs as console items.

All default items are given a name. The default named * is special and automatically applied to all further console definitions. Consoles (or other default items) can explicitly reference a default by using the include option. Here's an example of a named default (ts-ssh) being included in the console definition for foo:

default ts-ssh {
  type exec; portbase 2000; portinc 1;
  exec /usr/local/bin/ssh -p P -l tsuser H;
  execsubst H=hs,P=Pd;
}
console foo { include ts-ssh; host ts1.conserver.com; port 4; }

The above example substitutes all H characters with the host option (which is set in the console definition), and all P characters with decimal representation of the calculated port value. Thus you can define an abstract connection in a named default and only put console-specific information in the console definition. If the default was named * instead of ts-ssh, the include ts-ssh; could have been removed since the * default is automatically included in the console definition.

You can also change the runtime behavior of conserver via config items. These match the command-line options you can give conserver. Some things you can do are set file locations, change SSL settings (if you compiled in SSL support), disable the client redirection feature, and set the default access type.

With managed consoles, you must also think about access restrictions. Access to a console is governed by two levels of restrictions: server-specific and console-specific. A client must first be allowed to connect to the server, and then it must be allowed to connect to the particular console.

The first server-specific access restriction is governed by tcp_wrappers (if built into the code). The next access check is against the lists built with access configuration items. If no access list matches, the default access type (defined with the defaultaccess option in the config item) is used.

To build an access list, the name of the item should match the conserver host it applies to or * for all hosts. There are three types of access checks, as shown in this access list for conserver01.conserver.com:

access conserver01.conserver.com {
  trusted 127.0.0.1;
  allowed 192.168.0.0/16, 172.0.0.0/8;
  denied 192.0.0.0/8;
}

Once past the server-level access list, the client must then be given access to the console via the rw and ro console options. As I showed previously, the rw option specifies who is allowed access to a console in read-write mode. ro specifies who is granted read-only access. Both options take a list of users, which can be user names, conserver group names, or system group names (/etc/group, etc).

Conserver group names are created with the group item. You give it a list of users, just like the rw and ro items, and can then use them wherever a list of users is needed. Here's an example:

group admin {
  users bryan, todd, @sysadmin;
}

If a client is to have access and a password check is necessary, conserver looks for a conserver.passwd file. That file should have user names and encrypted passwords of the form username:passwd. If the user exists and the password matches, access is granted. Two special keywords are allowed in the file. For username, you can use "*any*", to match all users. And for passwd you can use "*passwd*", which tells conserver to look up the password via PAM (if compiled in) or the standard system library. This way you can hard-code passwords for some users, allow others to use their system password, restrict access by locking an account, or do a variety of other combinations. If the conserver.passwd file doesn't exist, conserver will just query PAM or the system library.

One final aspect of the configuration file is its ability to be split into parts. This allows you to edit pieces individually, generate a portion based on external data, or organize things however you like. You simply specify a file that you'd like to include via the #include feature:

#include /etc/conserver-globals.cf
console one { include ts3.conserver.com; port 4; }

These are just some of the ways in which you can define defaults, consoles, users, and access lists. There are more examples of basic to complex configurations in the conserver.cf/samples directory of the distribution.

Distributed Conserver Hosts

Conserver also supports the idea of a distributed conserver configuration. The idea is that you would have multiple hosts running conserver, sharing a single configuration file. Each server would then redirect the client requests to the appropriate conserver host. That is what the master console option states -- which host should manage the console. One way to set this up would be to override the automatic default (*) as you define consoles:

default global { ... } # set generic defaults here

default * { include global; master conserver1.conserver.com; }

console foo { include ts1.conserver.com; port 2; }
console bar { include ts2.conserver.com; port 3; }

default * { include global; master conserver2.conserver.com; }

console zip { include ts1.conserver.com; port 4; }
console zap { include ts2.conserver.com; port 5; }

There really is no difficulty in setting up a distributed configuration. You could even take independent conserver.cf files from multiple conserver hosts, concatenate them, and distribute them (assuming your master values were unique hostnames and not localhost). But using things such as named defaults and sharing the information between console definitions would, of course, make things much cleaner.

The Conserver Client

The conserver client can be used in two ways: to do global operations across all consoles, and to connect to a specific console. All global operations return you to your shell, whereas connecting to a console virtually links your terminal to the console.

Most of the global operations you can do are querying functions. You can ask who is attached to each console, if the consoles are up, how the consoles are connected (host and port numbers, local devices, etc.), and even get a formatted data dump of the state of the consoles for scripting purposes.

Aside from the querying functions, you can also do "bulk" operations. You can send a broadcast message to all users, send a message to specific users, disconnect users, and tell the conserver processes to shut down. The destructive functions require administrator rights (which are set with the admin option in config items), but the rest are available to all valid users.

When connecting directly to a console, the client does nothing but send all key sequences to the server and display all characters the server returns. There is an escape sequence, processed on the server, that allows you do other things, like bring the console up or down, send messages to other users on the console, take over read-write mode (only one user is allowed read-write access at a time), send break sequences, etc. To give you an idea of all the things you can do, here's the help screen you get when connected in read-write mode:

$ console simple
[Enter '^Ec?' for help]
[help]
 .    disconnect                      ;    move to another console
 a    attach read/write               b    send broadcast message
 c    toggle flow control             d    down a console
 e    change escape sequence          f    force attach read/write
 g    group info                      i    information dump
 L    toggle logging on/off           l?   break sequence list
 l0   send break per config file      l1-9 send specific break sequence
 m    display the message of the day  o    (re)open the tty and log file
 p    replay the last 60 lines        r    replay the last 20 lines
 s    spy read only                   u    show host status
 v    show version info               w    who is on this console
 x    show console baud info          z    suspend the connection
 |    attach local command            ?    print this message
 <cr> ignore/abort command            ^R   replay the last line
 \ooo send character by octal code

The break sequences deserve a little explanation. You can think of them as macros, with the ability to specify delays and serial breaks (which was the original need, hence the name). They are defined in the configuration file and available to all consoles. One of them is then defined to be the primary break sequence for the console and accessed via the "^Ecl0" escape sequence. This allows you to have different types of hardware, with different definitions of a "break" and all accessible via a single escape sequence. To illustrate these concepts, here are a couple of examples:

break 1 { string "\z"; }
break 2 { string "\r\d~\d^b"; delay 600; }

As you can see, using the client is straightforward and provides a wide array of capabilities and information, which is exactly the way it should be.

Conclusion

By no means was this a complete description of everything conserver can do. I hope, however, that I've provided a good overview of why conserver is a robust, mature console management system.

The latest version of conserver can be found at http://www.conserver.com/ or one of its mirrors. The mailing lists and archives are available to help you make conserver a successful part of your environment.

Notes

1. Fine, Thomas A. and Romig, Steven M. "A Console Server". Ohio State University -- http://hea-www.harvard.edu/~fine/Tech/cs1.1/paper.asc

2. Kevin Braunsdorf of Purdue University and Robert Olson of Argonne National Laboratory were the significant contributors.

Bryan Stansell has been part of the system administration community in the San Francisco bay area for more than 10 years. When not dealing with pesky computers or hacking on open source software, he likes visiting strange, exotic lands, like Canada! Bryan can be contacted at: bryan@conserver.com.