Monitoring logs with SEC

Splunk seems to become de-facto standard tool for log management. But free version lacks feature that lets you configure and send alerts whenever certain events occur. One need to pay for enterprise version which starts at $5000 in US and Canada.

So, I use Simple Event Correlator to notify me of interesting events in life of my router friends. Here, for example, sec template to send me email with syslog line in the body when somebody tries to go to configuration mode and execute certain commands:

type=Single
ptype=RegExp
pattern=.*cmd=(configure|clear|ip|no|interface|switchport|router|spanning-tree)
desc=$0
action=pipe '$0' /usr/bin/mail -s "router/switch config change is happening right now" noc@example.com

You need to put this template into SEC configuration file and tell it were to look for these messages:

sec -detach -conf=/etc/sec-tacacs.conf -input=/var/log/tac-plus/account

In this case it's TACACS+ log file, so you need to configure a router to report such activities:

aaa new-model
aaa authentication login default group tacacs+ none
aaa authentication enable default group tacacs+ none
aaa authorization exec default group tacacs+ none
aaa authorization commands 15 default group tacacs+ none
aaa accounting commands 15 default start-stop group tacacs+
tacacs-server host <server ip>
tacacs-server <key>

Here is another template to report all syslog messages coming from devices with loopback interface IP address in the range 10.9.20.0/24 or 10.9.25.0/24. Why loopback? See my previous post.

type=Single
ptype=RegExp
pattern=(.*)10\.9\.2[0|5]\.(.*)%[A-Z]*
desc=$0
action=pipe '$0' /usr/bin/mail -s " router syslog message" noc@example.com

Best practices. Sort of.

I tend to agree, that there is no "best practices", there are practices that fit best. Here is one of the things that I always configure on the router.

There are many advantages in configuring Loopback interface when you use dynamic routing, but I also find loopback helpful for syslog reporting and authentication and authorization queries. So, I always configure:

ip tacacs source-interface Loopback0
logging source-interface Loopback0

Next step is to either add loopback interfaces of your routers to DNS or /etc/hosts file on Tacacs and syslog servers.
The names are no good if you can not use them. I prefer syslog-ng for logging, so, in order to record names instead of IP addresses, you need to configure use_dns(yes) in "options" section of syslog-ng.conf. For TACACS+: run tac_plus with "-L" option.

Making same change on many routers

Suppose you need to make the same change on many routers, but do not have fancy software like Cisco Works to help you. No worries. Perl is the best friend of any network and system administrator. Here is the quick script that goes to a router and types command "logging source-interface loopback 0", saves configuration and exit. It can be used to run any command.
Place IP addresses of the routers, one per line, in file routers.txt. This file must be in the same directory as the script. Remember, you put your username, password and enable password in the script in clear text, so do not forget "chmod 700 "

#!/usr/bin/perl
use Net::Telnet::Cisco;
my $myfile="./routers.txt";
open (FH, $myfile) || die "Can not open $myfile\n";

while () {
chomp;
my $switchname=$_;
print "$switchname\n"; 

my $session = Net::Telnet::Cisco->new(Host => $switchname,Input_log => "$switchname.log"); 

# Replace username and password below with real username and password 

$session->login('username', 'password');

# Enable mode
if ($session->enable("enable password") ) { # insert your enable passowrd
    @output = $session->cmd('configure terminal');
    @output = $session->cmd('logging source-interface loopback 0');
    print @output;
    @output = $session->cmd('exit');
    @output = $session->cmd("copy run startup-config\n\n");
    print @output;
    } else {
         warn "Can't enable: " . $session->errmsg;
        }
$session->close;
}
 

Use at your own risk. 

Debian 5.0.4 on Dell 1950

Normally, installing Debian on Dell servers is piece of cake. This particular 1950 came with Broadcom NICs and PERC5 controller. Debian 5.0.4 does not include driver for Broadcom drivers due to some copyright restrictions. However, the driver is available as deb package. Download it and copy to FAT or FAT32 formatted USB drive. When prompted for NIC driver during the installation process, insert USB drive into USB port. As soon as server loads the driver and moves to the next screen in installation process, remove the drive. If you do not remove the USB drive before installation process gets to partitioning, your drive sequence will we out of whack. You'll have to boot from CD and edit /etc/fstab.
Since this server has hardware I wanted to use instead of configuring software RAID in Linux. The question is how to monitor RAID state from Debian. There is no deb package or source code, but LSI provides RPM. I downloaded "MegaCLI - Linux" from "Miscellaneous" section, unpacked it, installed "alien" on Debian (sudo apt-get install aliean) and then "sudo alien -i  MegaCli-1.01-0.i386.rpm". It install MegaCli under /opt/MegaRAID/MegaCli.  Moritz Mertinkat has great emergency cheat sheet for MegaCli usage.