IP Policy Enforcement with Netinfo

Stefanos Harhalakis

A common problem faced by systems administrators who don't have complete management of their network is IP policy enforcement. There are a lot of tools for monitoring routers and switches (referred as network devices), but there are few for monitoring the leaf nodes and their status.

It is not uncommon to find that there is a duplicate IP in a network or that a user has changed his IP address or even the location of his computer. (Here, "location" describes the switch port of a user and not the actual physical location.) An administrator may need to spend a couple of hours to actually resolve a duplicate IP issue because he has to find both machines online.

Common policy enforcement techniques utilize MAC address databases where each machine must be registered to be allowed to access the network from specific switch ports. This solves the problem in a tightly managed network but cannot prevent intruders from abusing public access places. It is also almost impossible (and very time consuming) to create and maintain a valid up-to-date MAC address database.

Generally there are three entities comprising a network policy:

• IP addresses
• MAC addresses
• Switch ports

With the introduction of Layer 3 switches, a "switch port" is not different from a "router port". It is more convenient to refer to these as network device ports and distinguish them as:

• Switched ports
• Routed ports

Typically, an administrator will need to define restrictions on various combinations of these entities. For example, it may be desirable to restrict a particular MAC address to a specific switch port or range of ports or to bind a MAC address to a particular IP address. Similarly, the administrator may want to restrict a particular IP address to a specific set of ports.

This description can be represented with a triangle with IP, MAC, Port as corners and a two-way arrow on each edge:

Here an IP address restricts the MAC address, a MAC address restricts the IP address, a MAC address restricts the Port, a Port restricts the MAC address, and an IP address restricts the Port or a port restricts the IP address. It must be understood that IP->MAC is not the same as MAC->IP because an IP address may be used by one MAC address only, but a MAC address may be allowed to use more than one IP addresses.

Here are some examples that illustrate these restrictions:

• A user has her computer on a switched port (let's say FastEthernet 1). The administrator wants to allow this user to change her computer without being bothered, so he associates the port address to only allow access to this user's IP address. If this user ever changes her IP address, the administrator will be notified. Note that the user will be able to change her NIC. We may also want to restrict this IP address to only be allowed to be used at this switched port.
• A user owns a laptop computer and has a configured address for it. Now this laptop can be found in a lot of places; DHCP may help here, but sometimes we want to reserve special addresses for people so we must maintain another IP<->MAC database in the DHCP configuration. We simply allow this user to have a fixed IP address, and we associate his MAC address to his IP address and vice versa. This way no one else will be able to use this IP address, and this MAC address will use only this IP address.

Enforcing the policy consists of:

• Monitoring the network and collecting data.
• Processing data and detecting violations.
• Performing actions for detected violations.

There can be two approaches to policy enforcement. First, you can prevent violations from happening. This technique is quite hard to be implemented and administered. It also does not inform the administrator of violations (most of the time). Second, you can detect violations and perform some actions, either by hand or by means of scripts/programs.

Experience shows that policy "enforcement" is not always required, however, it is often more convenient simply to monitor policy violations instead of enforcing.

Users who violate network policy must not be considered as intruders. In fact there are cases where it is more important for a user to have network access even if he has violated the policy rules. After monitoring our network for about two years, we found that abuses made up about 1% of the total number of violations detected. Allowing a user who has performed a violation to have network access also helps in case of false alarms and administration mistakes.

The MAC<->IP restrictions are less frequently used when we have the MAC<->Port and IP<->Port options. This should be expected since a MAC address database for 1000+ machines is very hard to create and maintain. There are also cases where machines are discarded and stale entries remain in the database.

Collecting the Information

Collecting the information can be quite easy as long as the hardware supports it. Here I will focus on Cisco devices, but this technique can be applied to other hardware vendors, too. We're going to collect information using SNMP so it must be enabled to network devices (more on this later).

We'll use PostgreSQL to store the information. For our purposes, we will need to create tables to hold the data and will also need to timestamp them. Timestamping the collected data is essential because we want our system to remember what happened. Creating a simple program that only knows the current image is easier but it leads to great troubles. There are cases where devices are being shut off or are inaccessible that can mess up our image of the network.

We need to have a table for all the devices that we're going to monitor in our network (referred to as monitored network devices or monitored devices). This table holds the IP address of each device (routers and switches), the community string, a unique ID (we should not use the IP address for performance issues), and a convenient name.

Before beginning, we need to know which interfaces are available to each monitored network device so that we can map users to interfaces later. We do this by fetching IF-MIB::ifDescr and IF-MIB::ifAlias for each device, which will give us names and descriptions for each interface. We store this information in a table that contains a unique ID for each entry, a device ID, the ifindex value of each interface, the interface name, and the interface description.

We also need to collect the following:

• IF-MIB::ifPhysAddress, which holds the MAC addresses of each interface.
• IP-MIB::ipAdEntIfIndex, which holds the IP addresses that are configured on each interface. Such an entry indicates that a port is a routed port so we mark it appropriately. We combine this information with IP-MIB::ipAdEntNetMask and now we know which IP networks are available in our network. This is very useful because we will not have to manually update a list of our IP networks. This solves a problem when Proxy ARP is implemented and the ARP tables are filled with useless (for our purposes) information.
• mib-2.17.1.4.1.2, which has a mapping of ifIndex with an internal interface number. It is required for Cisco devices only.
• enterprises.9.9.46.1.3.1.1.3.1, which returns a list of all the configured VLANs in the switch. This is required for Cisco devices because they need to be queried for each VLAN as if they were different devices using a community name in the form of XXXX@vlanID where XXXX is the community name. Requesting information for public will return information about vlan1 only, so we need to perform another request using public@2 if there is a VLAN2 configured.
• IF-MIB::ifType, which holds the IANA interface type ID. This is required because we want to ignore some interfaces and mask out VLANs.

Next, we want to know which IP addresses are being used by each MAC address in our network. This can be done by collecting all the ARP tables (let's say every 15 minutes) using IP-MIB::ipNetToMediaPhysAddress. We want the ARP tables from routers because they hold associations for all the IPs that are known on each routed port. Since switches can be Layer 3 and can have routed ports, we are forced to collect the ARP tables from them, too. For each ARP entry we collect, we update our database by storing the new IP-MAC pairs and updating existing ones. ARP tables can have only one entry per IP address so we will use the IP address as the primary key of that table.

We also want to know where each MAC address is located. For Cisco switches this is known as "mac-address-table" (for switches running IOS) or "content addressable memory (CAM)" (for switches running CatOS). Both of these map a MAC address to an interface that is known by fetching mib-2.17.4.3.1.2 and returning the MAC address to port assignments. We also need to get mib-2.17.4.3.1.3, which tells us how each MAC address is known (learned, security, internal, etc.).

This information by itself is all we need to know about our network. The catch here is that when we have something like:

Switch1 (port Fa0/1) <---> (port Fa0/1) Switch2 <---> 10 machines

The MAC addresses of the 10 machines are known to Switch1 and Switch2 also. Switch1 will report them as being behind its Fa0/1, and Switch2 will report them in their correct location. In this case, we ignore Switch1 and use the information from Switch2 because it has more details. Since we're monitoring both switches, we know what the internal MAC addresses of Switch2 are (via IF-MIB::ifPhysAddress and mib-2.17.4.3.1.3).

By combining this information, we can find out that port Fa0/1 of Switch1 has another monitored device behind it. For the sake of simplicity, we will assume that all monitored devices are connected using point-to-point links (with no hubs or other non-monitored devices between them). Now we can filter out all information about Fa0/1 of Switch1. This applies only to non-routed interfaces.

We store this information in a table by itself that says that MAC address X is in interface Y, where Y is an ifid that can be used to look up in the interfaces table and return the device and interface name/description.

Figure 1 is the complete database schema that is used to store this information. A table named "netdevs" contains all the devices we are going to scan. This table is filled and maintained by us. Its fields are: a device ID, the IP address, a name, the community, and a timestamp that indicates when we got data from this device (this can help to detect devices that are not reachable).

There is also a table named "interfaces", which holds the discovered interfaces of the network. Here we have a unique ID, the device ID indicating which device this interface corresponds to, the ifDescr, the ifAlias, and the ifType. There are also two timestamps, as follows:

ts: This timestamp indicates when this interface was last detected. This is used to expire interfaces. There are times when a device is not reachable or a network error occurs during the SNMP communication, so we don't want to remove interface entries the first time they are not detected. Using a timestamp, we can expire interfaces after a number of days.

last_had_netdev: This timestamp indicates when a monitored network device was last detected on this port. Just as with "ts", we want to hold a timestamp and not a simple flag, because the connected device may become unavailable. This kind of instability will lead to an incorrect network view, cause false alarms, and mess up the recorded history of the network.

Next, the "macs" table lists all the discovered MAC addresses of our network. For each entry, there is the interface where this MAC entry was last seen and a timstamp.

The "ips" table stores all discovered ARP entries (ip, mac) with a timestamp (ts), and the "subnets" table stores all discovered subnets. The "history" table is explained later.

All tables except "netdevs" have a "flags" field, which is used to store various flags. There are also some fields named "shortdesc", "comments", or "description", which hold user notes and names.

Combining the Information

Now we can combine the collected data and see that IP address I is being used by MAC address M, which is being served by interface F. This information by itself is a great boost in our administration efforts. We can now view all our monitored devices and see which IP address and MAC addresses are on each interface (Figure 2). We can also find out who owns MAC address M and where that user is located (Figure 3).

Next, we need to store the history of our network. We create a table that holds IP:MAC:Interface pairs and a timestamp for each one. Every time we collect data, we update this table by inserting new entries and updating the timestamp of existing ones. This way we can easily find out which IP addresses were used by MAC address M in the past or which MAC addresses used IP address I (Figure 4). We can also locate duplicate IPs (Figure 5).

Creating Rules

We want to implement a schema that can hold the policy of our network. We do this by populating the collected data with more information (descriptions, notes, flags) and creating three more tables to hold the IP-MAC, IP-Interface, and MAC-Interface associations. Those are enough for all the six types of rules that were described.

We populate those tables with information regarding what is allowed for each entity and we set some flags. For example, we edit an interface and say that it should serve only MAC addresses M1, M2, and M3, and that only IP addresses I1 and I2 can appear there. This is done by inserting Interface-MAC and IP-Interface associations in the appropriate tables and by setting two flags that say that this interface should serve only the associated IPs and MACs.

Detecting Violations

Now we can validate our existing (and future) IP:MAC:Interface pairs against our policy and find out about intruders (Figure 6). Using PostgreSQL capabilities, we can create functions and views that return the violations with a single SELECT statement. Most of the job is done within the database schema using functions and custom views so the user interface only needs to be an interface and nothing more.

Policy Enforcement

All violations are associated with the MAC address that corresponds to the user who violated the policy. If we're going to enforce the policy, we will need to revoke network access from users that performed violations. To do this, we create another table named "blacklist", which will hold the MAC addresses of users in violation of policy and possibly a reason. This table is populated as the final phase of the collection. This table can also be populated by hand so that the administrator can manually lock out a user.

To perform actions on Cisco devices, we must telnet to them and perform an "enable". So, we need to know the telnet username/password and the enable password. The device table is extended to hold this information.

Now, we have come to a point where we want to lock out users from our network. We can do this by shutting down their switch ports but that is not acceptable when there is more than one user behind each port. We need to add a kind of access list on those devices to prevent them from accessing the network. Cisco switches can have a list of MAC addresses that are allowed behind each port (known as the "mac-address-table"), which can be configured by command prompt. For security reasons, we also don't consider adding MAC address tables to interfaces that are known to have other monitored network devices connected.

Locking out users can introduce problems so we want to restrict this as much as possible. We want to detect policy violations, but we want to perform policy enforcement only in some places. Thus, we need to flag interfaces that will be accessed during policy enforcement by introducing a new flag for each interface. We also add a flag for each monitored network device so that we can disable policy enforcement on a device as a whole.

There are two policy enforcement methods to be applied:

• On interfaces where we only allow a specific MAC address, we create a MAC address table and configure it without waiting for a violation to happen. If there is a user in this MAC address table who performed a violation then his or her MAC address will be removed during the next run.
• On all other interfaces, we wait for a policy violation to happen. Then we can get a list of all the known MAC addresses existing behind a given interface and add a MAC address table to it containing all those MACs except from the violators. This is a great relief for network administrators because they don't need to know which MAC addresses are behind each interface. After the MAC address table is added in the configuration, only those users will be able to access the network.

A policy enforcement tool can be created using simple shell script commands. I've created a simple shell+expect script that reads data from the blacklist table and tries to enforce the policy as follows:

• It determines which devices require locking.
• For each device, it telnets and removes the old configurations. Then it creates the new one.

In fact, the script tries to be a little more clever, so it creates a list of expect scripts on each run. Then it compares that list with the previous one and, if there is a change, it will run. It will also run if a certain amount of time has passed. This way there will be no extra load on switches.

Data Expiration

One tricky aspect is knowing when to remove entries from the database. We have the current image and a history table. We also have rules, notes, and descriptions that can become outdated in a couple of years. Discovered MAC addresses may not be needed after a couple of months.

To ease the administration of such a system, we expire data based on timestamps. If after X months MAC address M is no longer being used, we can remove it from the system. At this time, we can also remove any policy data associated with the address. After all, if the address is being purged, we no longer care that it's owned by user "Goofy" or bound to a particular IP address or port. This simple timeout algorithm goes a long way toward making the tool self-maintaining. With this approach, even for very large networks (say 100,000 MAC addresses) that are monitored for a couple of years, the database will not become very large and the performance will be pretty nice.

For a sample of 58 monitored network devices we have found 1400 interfaces. There are about 2100 unique IP addresses and 2600 known MAC addresses. The MAC addresses are more than the IP addresses because the switches have a lot of internal MACs (at least one for each interface), and there are other (not monitored) switches, too.

In case of malicious activity where someone tries to fill the mac-address-table by flooding the network with fake MAC addresses to make sniffing possible, a lot of new MAC addresses will be discovered. In one such case, there were more than 30,000 MAC addresses reported for a switch port and stored to the database. Because of the expiry, however, those entries were auto-removed a couple of months later.

The database access should be pretty fast even for very large networks. On a P3/866, it takes about 400 ms to combine all the data and produce a complete current view from within a quite complex PostgreSQL view (using "explain analyze select * from view_current" in the Netinfo database).

Netinfo

The algorithm and implementation described here is included in Netinfo v3.0, which is free software distributed under the General Public License. In fact, Netinfo does some more complex computations and provides other (not well-tested) functions like "Hunting", which can be used to insert rules for interfaces that have monitored network devices connected on them. These rules can then be auto-applied to all interfaces of all monitored network devices existing behind this interface.

The whole idea has gone through a lot of stages of patching and rewriting, and I consider it to be very stable. At first, this method was created as a monitoring tool to map IPs, MACs, and Interfaces for the Technological Education Institute of Thessaloniki in Greece. Later, it was extended to have a history and hold some policy rules and to perform user lock-out. Then, it was rewritten from scratch to resolve database management and performance issues (and more).

An administrator can add descriptions to subnets, IPs, MACs, or interfaces, which will be shown on each view. The program also knows about NIC vendors and determines them from the MAC address prefix.

Stefanos Harhalakis is a graduate of the Department of Informatics of Technological Educational Institute (TEI) of Thessaloniki, Greece. He works as a Systems and Network Administrator for the Dept. of Informatics and provides support to the Institute's NOC, and his activities include development of C/C++ applications and facilities for network security and monitoring. Stefanos can be reached at: v13@it.teithe.gr.