WebJob -- Virtual Evidence Locker and Administration Tool

Dave Mangot

WebJob is a program written by Klayton Monroe of KoreLogic (http://www.korelogic.com) that allows you to download and execute code from a remote server in an easy and secure manner. The original intent of WebJob was to aid in the performance of incident response and triage on potentially compromised, or otherwise "hostile" systems. What Klayton created, however, is an extremely versatile and powerful tool that has many applications.

WebJob was created in part to aid in the creation of a Virtual Evidence Locker (VEL) during forensics investigations. The idea behind a VEL is that all the tools that are used for the investigation are stored in a central location. Additionally, all the output of running these tools is stored in the same central location, thus an evidence locker.

With WebJob, all the tools and logs are transferred over the network in an encrypted manner and can be strongly authenticated. If multiple investigators are working on the same project, or if one investigator is working from multiple locations, all the data is collected into a central location where it can be reported on, shared, and examined. It may be possible to extrapolate how WebJob could be used for management and monitoring, which will be covered later in the article.

Advantages of Using WebJob

Simplicity

WebJob consists of two readily available components -- Apache and the WebJob client. The Apache server, like the client, needs to be able to support SSL if you want your downloads and logging to be encrypted. This also provides the option of doing certificate-based authentication in addition to, or instead of, Apache basic authentication. The amount of configuration necessary on the Apache side is not much more than installing a script in a cgi-bin directory of your choosing and creating a few directories. On the client side, WebJob can be run from a single binary and configuration file that can easily fit onto a floppy or CD-ROM (or other immutable media) or be downloaded to the client in a matter of seconds, even over a modem.

Forensics Grade

WebJob was written with security in mind. It was designed to be used in incident response situations where the client environment should not be trusted. If you don't trust the libraries on the client, then you can bring trusted libraries or statically compiled binaries with you. If you suspect the machine you are investigating has an LKM rootkit, then you should mount the disk in another machine and use your tools through WebJob to investigate it. In fact, there are many reasons to mount the disk read-only on another machine either physically or through NFS. In either case, your WebJob toolbox is still available to you. Great care was taken in writing WebJob to make sure that all standard secure coding practices were followed.

Works Behind Firewalls

WebJob operates in a "pull" rather than a "push" manner, which means if you are at a remote location doing an investigation (whether physically or via a network), the only firewall permission you need is to be allowed to connect out via port 443 (or whatever port on which your Apache server runs). There is no requirement to open any additional inbound or outbound ports, and you still get to use all your standard tools and procedures. This is also handy if you are using WebJob to administer remote machines. You could actually have your firewall block all inbound traffic that is not part of an established connection from the client machines, and you would still retain your ability to have complete control over the clients.

Trusted Binaries

With WebJob, you get to keep all your forensics tools in one centralized location. It is certainly possible to burn a CD-ROM and take the tools you like with you, but then you would need to burn a different CD-ROM every time your tools preference changed. If you are working as part of a team, you might have to burn multiple copies. With WebJob you can keep your (or your team's) tools in one central location. If your needs change, you change them on the WebJob server and everyone gets the new changes, everywhere, instantly. If you don't necessarily trust the integrity of your WebJob server, then you probably shouldn't keep evidence there, but you can always cryptographically check the integrity of your binaries with GetHook, as is discussed later.

Good Logging

Because WebJob was designed to be used in the collection of evidence, logging was written in as a crucial component. When you run a WebJob, the output of stdout, stderr, as well as internal WebJob records are all logged to a central location. You can collate jobs by job id (JID), and you can examine the results any way you want; they are written as text files by default. If you want to load your logfiles into a database, it is not difficult to implement. There are example recipes for how this is done on the SourceForge Web site (http://webjob.sourceforget.net/WebJob).

Multiplatform

WebJob is designed to run on AIX, Cygwin, FreeBSD, HP-UX, MacOS X, NetBSD, OpenBSD, Linux, Solaris, and Windows NT/2K, which gives WebJob multiple advantages. If you are using WebJob for forensics, be assured that you can use your familiar WebJob tools on almost any system you need to investigate. If you are using WebJob for remote administration, you don't have to come up with a different solution for every platform. WebJob can probably run on whatever kind of system you have in-house. You still need to have the different binaries for the different platforms available on the WebJob server, of course, but if you often deal in something portable like the Unix shell, you will likely be able to use the same source on multiple systems, which can save you considerable time and effort.

Encryption and Mutual Authentication

As was mentioned previously, WebJob is designed to run over SSL. This way, all the communication of both your tools and crucial logs traverse the network encrypted. It can be configured so that both the client and server mutually authenticate, using certificates. This means that the client must verify/trust the certificate presented by the server, and the server must likewise verify/trust the certificate presented by the client. If either of these conditions is not met, the transaction will fail.

Centralized Management

One of the key benefits of using WebJob is centralization. If you use WebJob to do forensics, your binaries and logs are kept in a central location, which becomes your VEL. If you use WebJob for administration, you can think of the WebJob server as the master location for a centralized crontab. If you make changes to this crontab, they take effect the next time WebJob is run, and you don't need to go to each remote client and make the change. Additionally, it is a centralized crontab for which you don't need anything messy like NFS, and it works over the Internet, even from behind locked-down firewalls. If it is run from cron, you can schedule it to call in during your company's maintenance window, and all changes will begin during that time.

Not Another Daemon

Often as a systems administrator, when you want to gain the advantages of some software that is going to make your life easier -- either to administer your servers, or perform some good monitoring -- you need to install "another piece of software". With WebJob, there is no other software to install. You can use the tools already at your disposal like cron and Apache. Much time and effort has been spent installing the latest "management application" or trying to work out the key exchanges between SSH hosts. With WebJob, the tools you use are the tools you have been using, and you can use them any way you want. When you need to change the tools, you can do so from a central location.

Scalability

Because WebJob is a pull technology, and uses existing infrastructure, it scales very well. If you need to add another WebJob client, you simply have the client "phone in". There are no scripts to edit that tell your management system that there is yet another host to which it needs to SSH within your maintenance window. If you start to overload the Apache server, you can build another one, put the two behind a load balancer, and keep adding clients or servers as you deem necessary.

Compilation, Installation, and Configuration

Compilation

WebJob is available from the SourceForge Web site (http://webjob.sourceforget.net/WebJob). Because WebJob comes in two components -- a server-side CGI, and a client-side C program -- only the client needs to be compiled. WebJob is also part of the FreeBSD ports system.

Once you have downloaded the tar file, compilation follows the standard procedures:

$ tar -zxf webjob-1.4.0.tgz
$ cd webjob-1.4.0
$ ./configure && make
checking build system type... sparc-unknown-openbsd3.5
checking host system type... sparc-unknown-openbsd3.5
checking target system type... sparc-unknown-openbsd3.5
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
etc...

./configure --with-server-side --with-all-tools && make
Installation

On the server side, you will need to perform a few steps. For the purposes of this example, Apache is installed under /usr/local/apache. First, you must create directories for WebJob to run and store its configuration files if they don't already exist:

mkdir -p -m 755  /usr/local/apache/webjob /usr/local/apache/etc
chown 0:0  /usr/local/apache/webjob /usr/local/apache/etc

Next, we need to move the scripts into the appropriate directory. Find nph-webjob.cgi in the WebJob source tree you untarred and:

# cp nph-webjob.cgi /usr/local/apache/webjob
# chmod 755 /usr/local/apache/webjob/nph-webjob.cgi
# chown 0:0 /usr/local/apache/webjob/nph-webjob.cgi

Then find the nph-webjob.cfg file in the source tree and:

# cp nph-webjob.cfg /usr/local/apache/etc
# chmod 644 /usr/local/apache/etc/nph-webjob.cfg
# chown 0:0 /usr/local/apache/etc/nph-webjob.cfg

# mkdir -m 755 /var/integrity; chown -R 0:0 /var/integrity
# mkdir -m 755 /var/integrity/incoming /var/integrity/logfiles \
  /var/integrity/profiles
# chown www:0 /var/integrity/incoming
# touch /var/integrity/logfiles/nph-webjob.log; chown www:0 \
  /var/integrity/logfiles/nph-webjob.log; \
  chmod 644 /var/integrity/logfiles/nph-webjob.log

Configuration

Before you can configure WebJob, you must determine the requirements of your environment. Will you use SSL or plaintext? Will you use http basic auth or certificates? For the purposes of this example, you will use http basic auth over SSL. These decisions will affect the configuration of your webjob.conf file.

If you examine the webjob.conf file below, you will see the connection is made over https, and you need to fill in a username and password. In this example, you will save the configuration file in /tmp/webjob.conf:

ClientId=client1
URLGetURL=https://192.168.1.123/webjob/nph-webjob.cgi
URLPutURL=https://192.168.1.123/webjob/nph-webjob.cgi
URLUsername=client1
URLPassword=foo
URLAuthType=basic
RunType=snapshot
TempDirectory=/tmp # Use \temp for NT/2K
OverwriteExecutable=Y
UnlinkOutput=N
UnlinkExecutable=N

You are now ready to see WebJob in action. On the server, create a simple shell script in /var/integrity/profiles:

# mkdir -p -m 755 /var//integrity/profiles/client1/commands
# chown -R 0:0 /var/integrity/profiles/client1
# cat >/var/integrity/profiles/client1/host.sh
#!/bin/sh
hostname
^D
# chmod 644 /var/integrity/profiles/client1/host.sh

$ ./webjob --execute --file /tmp/webjob.conf host.sh
$

/var/integrity/logfiles/nph-webjob.log:
2004-05-23 16:32:06 server_1_1085355126_00723 client1 192.168.1.120 \
  GET client1 host.sh 0 19 0 200 -- Success
2004-05-23 16:32:10 server_1_1085355126_00723 client1 192.168.1.120 \
  PUT client1 host.sh 624 0 0 200 -- Success

-rw-r--r--  1 www  wheel  606 May 23 16:32 \
  client1_20040523163210_00724_host.sh.env
-rw-r--r--  1 www  wheel    0 May 23 16:32 \
  client1_20040523163210_00724_host.sh.err
-rw-r--r--  1 www  wheel   18 May 23 16:32 \
  client1_20040523163210_00724_host.sh.out
-rw-r--r--  1 www  wheel   30 May 23 16:32 \
  client1_20040523163210_00724_host.sh.rdy

The .env file provides information about the entire transaction:

Version=webjob 1.4.0 ssl 32 bit
Hostname=testmachine.example.com
SystemOS=sparc OpenBSD 3.5
ClientId=client1
GetRequest=host.sh
Command=host.sh
CommandLine=host.sh
Jid=server_1_1085355126_00723
Pid=20008
KidPid=27818
KidStatus=0
KidSignal=0
KidReason=The kid exited cleanly.
JobEpoch=2004-05-23 17:14:41 PDT (1085357681.245070)
GetEpoch=2004-05-23 17:14:41 PDT (1085357681.267157)
RunEpoch=2004-05-23 17:14:45 PDT (1085357685.027765)
PutEpoch=2004-05-23 17:14:45 PDT (1085357685.151000)
HashType=MD5
StdOutHash=cc69e3a7b641705ed798b7069e2aea6e
StdErrHash=d41d8cd98f00b204e9800998ecf8427e
GetError=NA
RunError=NA
Advanced Features in WebJob
Payload and Delivery (PaD)

#!/bin/sh
wget http://servername.example.com/myprograms.rpm
rpm -i myrpm.rpm

webjob -e -f /usr/local/etc/webjob.conf myprograms.rpm.pad rpm \
  -i /var/tmp/%payload

GetHook

GetHook allows you to perform a manipulation on the binary you are downloading. This command will change the binary into the form you want to use. An example will explain it best. You could define a hook where WebJob knows that your binaries are stored on the server as .gpg files. Then, when you make a call for host.sh, WebJob will actually download host.sh.gpg. Before host.sh is executed, it will have its signature checked with GPG. If the check did not return an error code, the binary would be executed as host.sh. If the check were to fail, WebJob would simply log the error.

Job ID

Job ID (JID) is a standard part of WebJob. Each job run through WebJob is assigned a unique identifier, the JID, which can be used to match GET and PUT requests in the nph-webjob.log file with each other, and with the resulting logfiles in /var/integrity/incoming.

Uses for WebJob

Forensics investigations

An investigator could have a standard procedure they follow on every investigation that includes taking MD5 checksums of certain files as well as collecting information from specific logs. This could be put in the form of a script like:

#!/bin/sh
ftimes --mapfull testbinaries.cfg
cat /var/log/messages /var/log/maillog

Remote Management in a Batch Manner

Systems administrators could use WebJob to manage their systems. In one scenario, you could set up the servers to run a command to call a specific script on the WebJob server every night at the beginning of the maintenance window. The script they call can be thought of as a batch queue. The batch queue itself may simply be a list of WebJob commands for the client to call:

#!/bin/sh

# if we don't have tcpdump installed, install it
if [ 'rpm -qa | grep tcpdump' -lt 1 ]; then
    webjob -e -f /usr/local/etc/webjob.conf tcpdump.rpm.pad rpm \
      -i /var/tmp/%payload
fi

if [ 'ls /var/spool/mqueue | wc -l ' -gt 1000 ]; then
    echo "too many messages in the mailqueue" | /usr/ucb/mail \
      -s "Message Warning" postmaster@mydomain.com
fi

Remote Monitoring

Because you know that the output of WebJob commands are stored on the Apache server in a centralized location, you can collect monitoring statistics from remote machines in a centralized, secure manner. In this example, you would set up a cronjob that runs every hour. In the profiles directory on the WebJob server, you would set up a script for your clients to execute, which calls the various monitoring commands you want (lsof, netstat, iostat, etc.). Whenever the cronjob executed, the results of these commands would be sent to the WebJob servers where they could be collected, parsed, manipulated, reported on, or maybe just stored.

Conclusion

WebJob is a very versatile tool to have in your systems administration toolbox. The architecture for its deployment is a simple client-server model. It allows you to execute binaries you trust, anywhere, in a secure manner. When you combine that with the ability to examine the results of your actions in a centralized location, it becomes a very powerful tool. There are more examples of ways to use WebJob, many of which are provided in the Cookbook at the homepage on SourceForge. I invite you to try it out, and let Klayton know if you've come up with a great new way to use WebJob!

Dave Mangot is a systems administrator in San Francisco. When not hacking on his OpenBSD SparcStation or obsessing about the Steelers, he can be found in the mountains snowboarding or hiking with his wife, Peggy. Dave can be contacted at sysad@mangot.com.