Article

jul2005.tar

Implementing an Effective Abuse Management Process -- Part II

Luis E. Muñoz

A good Abuse Management Process (AMP) combines policy enforcement with incident response in a way that is scalable, reliable, cheap, and swift. In this series of three articles on abuse management, I will share some tips and tricks about building an AMP. Last month, I described the Mail::Abuse package and focused on receiving and analyzing abuse reports.

This second article covers configuring Mail::Abuse to your network, so that various information elements and access logs can be used to automatically pinpoint the source of security incidents you want to know about.

You can download Mail::Abuse from the Comprehensive Perl Archive Network (CPAN) using this Perl command:

$ perl -MCPAN -e shell
cpan> install Mail::Abuse

Configuring the Abuse Script for the Network

A good place to start is to gather the information about the network for which you'll be handling abuse reports. If you're in charge of the incident response or the AMP, you probably already have this. But let's quickly review.

You should start with your IP address space. We will be creating a text table, where each entry is a subnet. You can provide any data item you see fit. That data will be attached to an incident when a match occurs, courtesy of Mail::Abuse::Processor::Table.

Let's say that 192.168.0.0/16 is the IP space you received from your RIR and that those IP addresses are happily routable through the Internet. Now, assume you have the following address allocation scheme in your network:

192.168.0/24 -- Internal servers, including inbound and outbound mail servers. These are managed by you.

192.168.1/24 -- Workstations located in the first floor of your main office. These machines are managed by Fred.

192.168.2/24 -- Workstations located in the second floor of your main office. These machines are managed by Robert.

192.168.3/24 -- Workstations located in the third floor of your main office. These machines are managed by Mike.

192.168.4/24 -- Workstations located in the fourth floor of your main office. These machines are managed by Lou.

192.168.5/24 -- Addresses allocated to the corporate VPN. You have two VPN aggregators, each allocated one /25. This is managed by you.

192.168.6/24 -- Addresses allocated to the corporate dial-up pool. This is also managed by you.

We will build a table with this data. The order of the entries is not that important and, in fact, we can have overlapping entries. This is fine if we want to provide more detailed information for some entries. For example, we could end up with something along these lines:

192.168.0/24 corp.manager=me@my.domain;corp.purpose=Internal Servers
192.168.1/24 corp.manager=fred@my.domain;corp.purpose=1st floor
workstations
192.168.2/24 corp.manager=robert@my.domain;corp.purpose=2nd floor
workstations
192.168.3/24 corp.manager=mike@my.domain;corp.purpose=3rd floor
workstations
192.168.4/24 corp.manager=lou@my.domain;corp.purpose=4th floor
workstations
192.168.1.36/32 corp.manager=fred@my.domain;corp.device=Main print
server
192.168.5.0/25 corp.manager=me@my.domain;corp.device=VPN 01 aggregator
192.168.5.128/25 corp.manager=me@my.domain;corp.device=VPN 02 aggregator
192.168.6/24 corp.manager=me@my.domain;corp.device=Corporate Dial-up Pool

Building Tables

The table is built by placing the IP subnet we want to match, followed by the variables or placeholders we want to define for that subnet. The IP subnet can be written in any conventional notation.

Note that this file does not have a fixed structure for the arguments we want to use. We can define as many variables or placeholders to store our data as we see fit. We can also change them or use different names for different parts.

It is a very good idea to have separate tables and merge them periodically. This allows each sys admin to manage the information for his or her own devices. Also, part of the information can be extracted from many systems and databases. You could, for instance, use SNMP to spider your devices and extract data from them to populate your tables.

With this setup, each incident would be matched with useful information about what type of device is the source of the problem and to whom to escalate the case. But with the VPN and the dial-up pool, we have lots to do yet, so let's get busy.

Managing Details

Most RADIUS servers produce a log of the accesses, often called a "detail" file. This name comes from the days of the old Livingston Radius. A detail file has entries that usually look similar to:

Fri Jan 21 10:05:56 2005
        Acct-Session-Id = "35000004"
        User-Name = "bob"
        NAS-IP-Address =  192.168.6.1
        NAS-Port = 1
        NAS-Port-Type = Async
        Acct-Status-Type = Stop
        Acct-Session-Time = 144000
        Acct-Input-Octets = 22
        Acct-Output-Octets = 187
        Acct-Terminate-Cause = Host-Request
        Service-Type = Login-User
        Login-Service = Telnet
        Login-IP-Host = 192.168.6.91
        Acct-Delay-Time = 0

Each particular RADIUS server has its own configuration directives and conventions but can often be coerced into generating a file with this format. Depending on the details of your particular server, you will need to rotate the detail file periodically and store it in a directory. Another approach is to generate a single, large, detail log with the entries for all your devices. This is a valid approach when your RADIUS server stores its detail data in a database.

We will assume that the RADIUS detail files are kept under /var/logs/radius/details/current and that there is a script run from cron that gets rid of (or moves away) files that are older than a few days. This is important because Mail::Abuse::Processor::Radius will scan all files when matching incidents. Keeping old detail files around will make this process slower.

Something similar to the following script will do. You can invoke it from cron a few times a day, and it will take care of the file management. Note that you must decide what to do with the old detail files, after archiving them.

#!/bin/sh
#
# Rotate the RADIUS detail logs and move old details out of the way
#
# (c) 2005, Luis E. Muñoz
# This is free software, subject to the same terms as Perl itself.

DETAIL_SRC=/var/log/radius/details/
DETAIL_DST=/var/log/radius/details/current
DETAIL_OLD=/var/log/old-details/
MAX_AGE=+15

# Move current files into our archive and re-touch them
find ${DETAIL_SRC} -type f -maxdepth 1 | while read i
do
  mv $i ${DETAIL_DST}/'basename $i'-'date +%Y%m%d%H%M'
  touch $i
done

# Move old files into the historic archive

find ${DETAIL_DST} -type f -mtime ${MAX_AGE} | while read i
do
  mv $i ${DETAIL_OLD}
done

If you're also a Perl user, you will find that Log::Rotate will help you do this with fewer keystrokes.

At this point, we're ready to start configuring the abuso script included with the Mail::Abuse package. We will start with a blank file and add entries to it. We'll be commenting them shortly. Fire up your favorite editor and open /usr/local/etc/abuso.conf (or any name that suits both your site's policy and your taste for configuration file naming).

As you might expect, comments can be added by preceding them with a #, just like in the shell.

Configuring Modules

To begin, let's configure which modules to use for the process. To accomplish this, we'll add the following configuration lines:

abuso reader: Stdin
abuso parsers: Normalize Log
abuso filters: Time IP
abuso processors: Table Radius Store Mailer

This tells abuso to slurp abuse reports using Mail::Abuse::Reader::Stdin, parse the incidents using Mail::Abuse::Incident::Normalize and Mail::Abuse::Incident::Log, filter them with Mail::Abuse::Filter::Time and Mail::Abuse::Filter::IP, and use the Mail::Abuse::Processor::* modules listed there. That was simple.

For performance reasons, you might want to feed many reports to the same invocation of abuso. To do this, Mail::Abuse::Reader::Stdin supports a delimiter between successive abuse reports. By default, this delimiter is creatively set to "___END_OF_REPORT___", a very uncommon string in the abuse reports. You may set this to any value to want in the configuration file using the "stdin separator" directive. We will leave the default as is. I've never had to change it.

Incident Parsing

Next, we'll configure the incident parsers. This is easy enough, as they have very few knobs to turn. In fact, the only setting you can tweak from the configuration file is the number of lines that Mail::Abuse::Incident::Log will see at a time, which defaults to 5. Setting this to a larger value will allow the recognition of more incidents in the abuse reports, although it will also increase server load, processing time, and the chance of false positives.

Setting this to a lower value will reduce the "window" in which an incident must fit to be recognized. If you feel daring, you can change this by adding it to the configuration file, although my recommendation is to tune this after your setup is complete and running:

log lines: 3

Now let's tell abuso how to filter out which incidents we do not want. Let's assume that we will reject incidents more than 10 days old or from outside our network. This is easy enough:

filter before: 10 days ago
source ip within: 192.168/16

The filters allow for the specification of more complex criteria. If your operation needs this, check the documentation that comes with the modules.

The last step is configuring the processors so that they accomplish what we need. These are the required lines:

mailer success message: /usr/local/etc/good-response.txt
mailer fail message: /usr/local/etc/bad-response.txt
mailer subject: Abuse complaint confirmation
mailer errors to: devnul@my.domain
mailer reply to: me@my.domain
mailer from: abuse@my.domain
mailer type: sendmail

As you can infer from the above entries, Mail::Abuse::Processor::Mailer is quite configurable. The first two lines tell it about our response templates, which must be readable by the user under which we will be running abuso. Next, we define the subject line we want on our responses. Using a fixed subject line makes it easier for you, and others, to filter. You can use this to get rid of dummy auto-responders that might cause loops by stripping headers and ignoring the Reply-To headers in the produced email. Yes, there are still some of those around.

The Errors-To, Reply-To, and From headers are set with our custom values. As set in this example, likely we do not want to hear from the bounces to our communications, we want replies to our message sent to our own account and our responses should come from abuse@my.domain.

The mailer type tells the module how to send an email. The value shown causes the invocation of the local sendmail program to do the delivery, which is a bit more reliable if you have configured it properly.

Routing Responses

It is worthwhile to discuss the routing of these answers and their replies. Mail::Abuse::Processor::Mailer routes its replies using the Reply-To headers, the Return-Path and, if all that fails, it uses the From address. Many auto-responders are not smart enough to do the same, so they send their own auto-ack to abuse@my.domain, and the loop begins.

Setting the Reply-To address to your own account will also land many confirmation messages into your mailbox. This may or may not be what you want. Think about quantity here, bearing in mind that more than half of the replies that abuso sends will receive an auto-reply or a bounce.

Since we're assuming that we will be archiving the processed reports, which I recommend, we'll tell it where to do so with the Mail::Abuse::Processor::Store module:

store root path: /var/log/abuse-reports

Note that the user under which you plan to run abuso and friends needs write permission in that directory. Also, for testing, we will use ./arch, so create both if needed.

The next processor, Mail::Abuse::Processor::Table, has a much simpler configuration, as shown:

table location: /usr/local/etc/data.table

It simply needs to be told where we placed the data table we constructed previously, which of course must also be readable by the abuso process. A similar setup is needed for the RADIUS log analysis:

radius detail type: livingston
radius detail location: /var/log/radius/details/

This tells Mail::Abuse::Processor::Radius that we will be processing detail logs in the Livingston format, at any file under /var/log/radius/details/.

Conclusion

This is most of the tweaking required to tailor your Mail::Abuse installation to your network. Likely, after some testing, you will want to make some more adjusting. Read the included documentation and make changes. Just make sure that changes are small and incremental, and that you have plenty of backups.

Mail::Abuse ships along with some useful tools. Take the time to look at their documentation, as I'll be using some of them in part three of this series when I test our setup and verify that it is working.

Luis has been working in various areas of computer science since the late 1980s. Some people blame him for conspiring to bring the Internet into his home country, where currently he spends most of his time teaching others about Perl and fighting network abuse at the largest ISP there. He also believes that being a sys admin is supposed to be fun.