Article

This is the web application described in the December 2005 issue of Sys Admin’s article titled “Trap Customization in an Enterprise OpenView Operations/Network Node Manager Environment”. You may want to read that article for more information on the design behind this application.

 

The Problem

In the production environment, the OVO event system receives a large volume of SNMP events. Unabated, the events in the browser will grow and grow making it almost impossible for staff to make any sense of what the system is telling them. Consequently, mechanisms are required to make some of these alarms reported in the OVO event browser to be removed automatically (ie changed from active to history events) when the alarm condition no longer exists. Making this happen implies the need to define a clearing event to every alarm event where possible.

 

The Solution

The idea is to configure automatic actions in the event system on the NNM collection stations. The automatic actions convert the content of the trap to a corresponding opcmsg command and invoke it to send the information to OVO. The event forwarding process is as follows:

 

1.      NNM collection station receives a trap that needs forwarding to OVO

2.      NNM’s event system executes the automatic action, a custom perl script that packages the trap information into an opcmsg command and executes it to forward the event to OVO. The reason to execute a custom perl script as an automatic action and not the opcmsg command directly is that the perl script is able to do dynamic severity translation based on a selected trap varbind instead of hard-coding a fixed OpenView severity although the custom perl script can also be configured to send the opc message at a fixed OpenView severity. By supporting dynamic severity translation, the same trap can be used as both the alarm and clearing event depending on the severity value contained in the varbind. This feature is important because many MIBs do not have separate alarm and clearing events but rely on the severity contained in a particular varbind to indicate the kind of event.

3.      the opc message is sent to OVO using opcmsg

4.      the message text prefix string (described next) triggers the custom template

5.      the template causes the message correlation system to either add the event to the OVO active event browser for an alarm event or remove an existing alarm from the browser for a clearing event.

 

For our intended purpose, we use each field in the opcmsg command as follows.

 

 

The custom OVO template for handling these opcmsg messages must perform the following:

 

·        Identify the special short prefix text string in the Msg_text using Message Key and Message Relation defined from the Message Correlation Window

·        Use the object field as the correlator for message correlation

·        Remove the special short prefix from msg_text before display

·        A less severe event cancels a more severe event, For example, major cancels critical, minor cancels both major and critical, and so on.

 

GUI to Generate Custom Script and NNM Event Configuration

The web application is named trapapp. It contains 4 menu items:

 

 

Configuration Files Managed by trapapp

As you enter information to customize traps using the trapapp application, the information you entered is saved in different sections of a configuration file in Windows .ini file format using the Config::IniFiles module. A configuration file consists of 5 sections: trap, format, alarm, clearing and opcinfo. The format of each section is described below (please note that you do not have to know the format of the configuration file to use the application):

 

[trap]

trapnum=trapname~varbind1~varbind2~…~varbindN

 

The purpose of the trap section is to record the trap name and its varbinds and associate that information with a trap number.

 

[format]

trapnum=eventType~messageFormat

 

where eventType can be either alarm, clearing or ignore.

 

The purpose of the format section is to identify the traps used for generating alarm and clearing events and their message format. Traps with eventType ignore are ignored during script generation.

 

 [alarm]

trapnum=correlatorFormat~varbind/fixedSeverity~toNormal~toWarning~toMinor~toMajor~toCritical

 

where varbind/fixedSeverity contains either a fixed OpenView severity (one of normal, warning, minor, major or critical) or a varbind. In the latter case, toNormal, toWarning, etc. contains the translation mapping from the varbind’s severity to that of OpenView’s.

 

The purpose of the alarm section is to define the correlator for alarm and clearing events and translation mapping from MIB-specific severity to that of OpenView’s.

 

 [clearing]

trapnum=alarmTrapNum~fixedSeverity

 

where alarmTrapNum specifies the alarm trap this event is to clear and fixedSeverity specifies the fixed OpenView severity of the trap.

 

[opcinfo]

appname=appName

messagegroup=groupName

nodename=nodeName or $0

 

if a specific nodeName is given, it will be used as the name of the sender of trap. If $0 is specified, the  host name of the agent sending the trap will be specified as the source of the message. If you are using a gateway host to send information on devices not configured in the OVO Node Bank, use a fixed nodename instead of $0.

 

The purpose of this section is to supply the information to complete the opcmsg command.

trapapp Configuration File

The trapapp web application has a configuration file named trapapp.ini. Its content is shown below:

 

[app]

scriptdir=./gen/

opcmsg=/opt/OV/bin/OpC/opcmsg

trapdconf=C:/Program Files/HP OpenView/NNM/conf/C/trapd.conf

proddir=/opt/trapapp/

trustedcmddir=C:/Program Files/HP OpenView/NNM/conf/trustedCmds.conf/

 

It consists of an [app] section and the following entries:

 

 

When you click on the Generate button, the application performs the following tasks:

 

 

In summary, the trapapp web application is designed to reduce to a minium the amount of manual configuration required on the NNM collection stations.

Setup and Usage

It is assumed that the reader is familiar with the configuration of web server, creating OVO templates and configuring NNM automatic actions using the respective OVO and NNM tools. If you do, then setting up the trapapp web application is simple. All it does is generate a custom perl script and a partial trapd.conf  file for each MIB you uploaded. To set up trapapp, the following steps are required:

 

  1. Download and install the Config::IniFiles perl module either form CPAN or ActivePerl
  2. Unzip (on Windows) or jar xvf (on UNIX assuming you have Java 2 SDK installed) the trapapp.zip file into its own directory named trapapp (any name will do)
  3. Configure a virtual directory on the web server named ovtrap to point to that directory
  4. Configure default document on the web server as index.html

 

Invoke the trapapp application by using the url:

 

               http://localhost/trapapp/

 

You can now start experimenting on the trapapp application and examine the generated partial trapd.conf file. Once you are comfortable with the generated partial trapd.conf file. You can update NNM’s trapd.conf file with the information contained in the generated partial trapd.conf  file using the NNM command xnmevents as follows:

 

               Xnmevents –replace partialConfigFile

 

Where partialConfigFile is the full path name of the generated partial trapd.conf file.

 

It is trivial to include a menu item in the web application to do this. I do not implement it because by providing such a convenience feature makes it too easy to change the production trapd configuration unintentionally while experimenting with the web GUI. It is exactly for the same reason that the trusted command list contained in the $OV_CONF/trustedCmds.conf/trapapp is only updated if the generated perl script name is not is absent in the file.

 

In addition, there is a one-off setup involving the following tasks on the OVO Server:

 

  1. Define OVO template that implements the message correlation facility described in an earlier
  2. Distribute the template to the OVO agent on the OVO server

 

Problem CheckList

When things do not work as expected, the following checklist may help you resolve the problems:

 

  1. Does the information in the trapapp.ini file correctly reflect your environment?
  2. Has the message group specified in the opcmsg command been defined in OVO?
  3. Has the node name used in the opcmsg command been configured in the OVO Node Bank?
  4. Has the MIB you uploaded to trapapp been loaded using NNM’s xnmloadmib tool?
  5. Have you updated the trapd.conf file by using the xnmevents command as described earlier?
  6. Is the path information contained in OV_CONF/trustedCmds.conf/trapapp for the custom perl scripts correct?
  7. Has the generated perl script been moved to the production directory and made executable (on UNIX platforms only)?

 

An Example

An example configuration for the concord-diagmon.mib is set up when you installed trapapp. In this sample configuration, trap 21 (nhLiveAlarm) and 23 (nhLiveClearAlarm) have been set up as alarm and clearing events respectively. The correlator used is $A-$5-$16 (ie, agentHostName-nhElementIp-nhExceptionId). nhLiveAlarm uses content of the varbind nhSeverity as the severity to use in the opsmsg command. In this case, the possible values of nhSeverity are the same as those for OpenView.

Limitations

Please note that this web application is intended for OpenView administrator’s personal use only. It is not intended to be used by the casual OpenView user. No security or concurrent access control has been implemented. Hence, if more than one user is updating the configuration for the same MIB at the same time, the result is unpredictable.

Updates

You can find trapapp updates/fixes and other free tools under “Free Tools” on the web site of Kardinia Software,  http://www.kardinia.com.  If you have enhanced and/or fixed any bugs in trapapp, please send me a description and your changes so that I can post it on the Kardinia web site.

 

 

Andy Yuen

andyyuen@ozemail.com.au

 

October 8, 2005.