The Value Proposition for Using Console Servers

David K. Z. Harris

Imagine yourself in this scenario -- you are carrying the primary on-call pager, and it's after normal business hours. You are in your car, nearing the end of an hour-long commute, when the pager sounds. An important device isn't responding to polling. Which would you rather do?

1. Turn around at the next off-ramp and face a certain 45-minute return to the office to get to the keyboard of the device's console to check its health.

2. Continue home, turn on your computer, put dinner in the microwave on medium power, and then use the VPN to securely access the serial console of the device, find out what the problem might be, and try to restore it to service.

Using serial ports to configure network gear and computing devices is not a new idea. However, using serial ports to help deploy and then manage in-service equipment has been steadily gaining new converts in the system, network, and security administration communities, because the "value" of implementing a remote serial console access service has been steadily improving.

The value of any deployment, or project, is determined not only by considering the hard and soft costs incurred, but also by taking into account the cost-to-benefit equation, which is sometimes forgotten during the initial project discussions. The cost of the hardware needed for secure, remote access to serial consoles is still expensive, but when the hardware is supplemented with additional software services, the value of the access multiplies many times. This article will discuss many points that should be considered if you are considering the deployment of a remote serial console access service.

Before delving into the issues of costs for the access hardware, let's review the benefits that the console service application will add to a console server deployment.

What Are the Benefits?

For clarification, a remote serial console access service has two tiers. The first tier uses terminal server or console server hardware devices to allow for network-based access to physical serial ports. The second tier uses a console service application to make the primary network connections (reverse-TCP, SSL, or SSH) to the console servers to establish connections to the serial ports.

Adding a console service application on top of the hardware deployment mitigates most of the pitfalls associated with terminal server and console server hardware-only deployments, and adds significant usefulness and added features, thereby improving the overall value of that initial high hardware cost.

Most of the console service applications available today share some common features:

• Separate log file for each monitored port
• Centralized logging for all monitored ports
• Centralized management of access to the consoles
• Audit trails of who accessed which port, when
• Multi-user simultaneous access to a console
• Ability to pass the write access between those users
• Ability to script access (chat) to certain ports

In the scenario at the beginning of this article, we were only using tier 1 (the console server hardware for access to the serial port). Let's continue with this scenario. When the pager sounded, you continued driving home, started your computer, and started warming up dinner. Once you log into your corporate network, you can access the console service application and access the console for the host that isn't responding, but then what?

If you are only using a console server device, you press the [return] key a couple times. Maybe you find the device at the "ok" prompt, or maybe you get a system prompt and start to debug why the application isn't responding. But what happens if the host crashed? You press [return], and nothing happens. You may need to go to the office after all. Maybe can you ask someone on-site to power-cycle the device, and see whether it comes to life (although serial-controlled power distribution units are also an option; see below).

However, if you are using a console service application, and the host doesn't respond, you can simply replay the last entries of the logfile for the host. You get to see the last gasps of a dying machine. Now you have some clues as to what caused the problem (failing CPU, memory error, maybe a full hard disk). Maybe you will need to return to the office, but you will be prepared to fix the problem. Or maybe the solution is to have someone power-cycle the machine, but you'll interrupt the boot and do that voodoo that you do so well. If the problem actually belongs to a different admin, you can also involve that person quickly.

If you do decide to involve another sys admin, he or she can connect to the same port simultaneously, via the console service application, and view the latest log entries and watch as you work. (Or the other sys admin could take the read/write control while you watch and learn!) This is an excellent mentoring tool, especially when coupled with a phone call to allow questions to be asked and answered as all parties are watching the same data on their respective terminals. This multi-user access, and the ability to give/take the write access to a shared port, is one of the main benefits where a console service application can mitigate the one-user-at-a-time restrictions of many console server devices.

The second major feature of the console service application is that the application now makes all of the initial "reverse-TCP" connections (to all of the console server device ports that you care about) when the application starts. As a result, the application is "always watching" and maintaining a logfile for each of the monitored devices. Because the admin's access to each console session will be made by the admin authenticating himself on the console service application host, you can configure all of the console server devices to only accept connections from a few machines (including the console service application host), and only by a few accounts, thus increasing the security around the console server devices.

In many cases, you can also access these active logfiles (e.g., using tail -f) to pass the output to a Perl handler that could be watching for specific strings, or perhaps triggering an SNMP trap, or sending email when certain strings are seen. The logfiles for each host can also be an invaluable source of forensic data, after an event has occurred. You can use the logs to look at many devices, all being monitored at once, including some devices that don't have a network interface and devices that cannot be SNMP managed.

In summary, a console service application provides sys admins with better access to the information they need to quickly diagnose a problem and (in most cases) improves the time needed to restore service. Allowing multiple sys admins to work on the same port at once will help minimize mistakes and can encourage mentoring among your technical staff. The logs also provide the opportunity to review failures after the fact, allowing you to help prevent future outages. By letting other tools comb through your log files, the logs can supplement other system and network monitoring tools you may be using.

It's Not Just for Servers

Most network gear is configurable using a serial console. You can even set most gear so that changes can only be made using the serial console. And, most manageable network gear can log useful information to that serial console. In larger networks, logging the data from consoles can provide valuable forensic data as well as help the network administrator watch for small problems that could become large problems if left unchecked. See:

http://www.greatcircle.com/blog/2005/03/10/new_paper_rigor.html

TIP: If you have a Cisco device with an unused Network Module (NM) slot, you should consider adding an NM-16A, or NM-32A module to that device and allowing it to also be a console server for the other nearby devices! This can be a great solution in a remote office environment or for adding console services at an external data center, because the ports can be protected by the secure access provided by the network gear, and it won't cost you an additional rack unit.

How Much Does It Cost?

You can start cheaply using free or open source console service applications, such as RTTY or Conserver. If you want a commercial application, perhaps with a support agreement, you can check out ASP Technologies Vantage application or check with Carlo Gavazzi for the latest ControlTower package. TECSys Development offers a Web-based console management tool called ConsoleWorks:

http://www.conserver.com
http://www.asptech.com
http://www.gavazzi-computing.com
http://www.tditx.com

To determine the value of tier 1 and tier 2 together, you need to determine the cost of the hardware deployment, which is the total of the "hard costs" (paid to vendors and contractors) and the soft costs (cost of finite resources that you already own but that you will consume for this deployment). I have an Excel spreadsheet available to help you do the math at:

http://www.conserver.com/consoles/DoTheMath.xls

You must also figure in the cost of the console service application, as well as the server on which it will run. If you choose an open source option or leverage an existing server, you may still need to add RAM, and possibly add disk capacity for the logfiles. If you choose the commercial path, you will have the application cost, possibly the server cost, and possibly a service/support contract cost. These would all be hard costs.

Are you going to install the console servers yourself, or will you pay a contractor or consultant to install them? Perhaps you want to add the cost of some training on the application server for your staff? A contractor or consultant could help you get the deployment running more smoothly, and quickly, but the money that you budget for that becomes a hard cost.

On the other hand, if you want your staff to do all of the work, possibly learning along the way, you will be committing some amount of staff time to this project. Many of the installation and configuration tasks can be done by a junior-level admin or a technician. In this case, you were already going to pay that staffer anyway, but you still need to estimate the number of hours they will need to spend and multiply that by their hourly wage. This is a soft cost, since you might not need to spend it all. But it is a cost, because that staffer won't be performing other tasks while they are deploying the console servers.

Other soft costs include the cost of the power outlets and the Ethernet ports needed to connect the devices. You probably already have some available, but these are a finite resource. The value of these ports will be the per-port cost to buy your next power strip or network switch. If you use cheap power strips, and cheap, unmanaged hubs, this cost can be less than $30 per console server, or it could exceed $50 if you use expensive PDUs and managed network gear. (This is the item that usually gets missed when people are trying to deploy many cheap, one-port console servers.)

You will also have some senior admin time, for adding DNS entries and IP address allocations. While this is not a significant cost, it does add up based on the number of devices to be installed. Here is a summary of the costs:

False Economies

You cannot build a large number of adapters (consistently wired, including labeling) more cheaply than buying them from a vendor. While it's a good plan to keep a few ready-to-build adapters handy, buy the pre-built ones in bulk, and plan to keep some spares in stock. You will want some extra pre-built adapters for debugging later.

A cheap terminal server on eBay may seem too good to pass up, but maybe you should pass it up anyway. Consider these issues, which all add hard or soft costs. Let the buyer beware:

• It may not have boot Flash, so you may need a net-boot environment for it.
• It may not come with its boot media (flash or floppy, or program on CD), so you will need to search the Internet to find it.
• You may need to buy the software from a vendor, for additional cost.
• It may not have its RAM, or it may not have enough RAM to run the software that you find.
• It may not come with documentation, so you may need to search the Internet to find instructions or configuration help.

It is not cheaper to go with many single-port console server devices if you need more than 6-8 ports in a single location. As an example, you can buy a single-port device for $165, but by the time you add the soft costs, each device costs between $250-275. Using this more complete cost, you can easily buy a single, full-featured console server, with 16 ports or more, instead of the eight basic ports. (All of the soft costs, including power and network connections and staff time, add up fast when you are buying the "cheap" one-port devices.)

While the cost of console servers is still high, the vendors want your business. Console servers are now easier to deploy and to manage. Many also incorporate scripts that help administrators through the setup of some of the more common configurations, making it easier to learn about these new devices. In many cases, the configuration documentation has improved, and there are useful Web resources with helpful information. All of these things reduce the amount of admin time needed to learn about, and deploy, new console server devices.

One of the keys to justifying the initial hardware costs for remote serial access hardware is to add a console service application to your hardware deployment. These applications can add integrated user authentication and access control, provide multi-user simultaneous access (useful for mentoring), and implement logging to a central machine (handy for real-time monitoring and forensics).

David K. Z. Harris is a Network Plumber, with more than 15 years of serial communications experience ranging from circuit design through communication application hacking. He maintains an open Web site with serial console clues at: http://www.conserver.com/consoles/.