Val(config)#

Monitoring trunk status via SNMP

2010-06-21T16:03:00.000-04:00

If you have not guessed yet, SNMP and monitoring are my favorites.

So, you have configured many trunks on your switch and now need to make sure all of them are actually in trunking mode. Here is 2 SNMP OID that can help you:

vlanTrunkPortDynamicState (1.3.6.1.4.1.9.9.46.1.6.1.1.13) - reports administrative state. From Cisco SNMP object navigator:
1 : on
2 : off
3 : desirable
4 : auto
5 : onNoNegotiate

vlanTrunkPortDynamicStatus (1.3.6.1.4.1.9.9.46.1.6.1.1.14) - reports operational state.
1 : trunking
2 : notTrunking

To get data for specific interface you need to add ifIndex to the end of the OID. For example, for interface ifIndex=10147
snmpwalk -v2c -Ov -Oq -c public myswitch 1.3.6.1.4.1.9.9.46.1.6.1.1.13.10147
To get ifIndex, you can either run "show snmp mib ifmib ifIndex" command in exec mode or query ifName OID with snmpwalk. Here is the quick script:

for int in ifIndex1 ifIndex2 ifIndexN
        do
        trunkoperstatus=`snmpwalk -v2c -Ov -Oq -c public myswitch \ 1.3.6.1.4.1.9.9.46.1.6.1.1.14.$int`
                if [ $trunkoperstatus -eq 2 ]
                then
                        trunkadminstatus=`snmpwalk -v2c -Ov -Oq -c public myswitch \ 1.3.6.1.4.1.9.9.46.1.6.1.1.13.$int`
                        if [ $trunkadminstatus -eq 1 ]
                        then
                                echo myswitch $int NotTrunking
                        fi
                fi
        done

Cisco 6500/7600 ACL side effect

2010-05-28T13:16:00.001-04:00

When you apply ACL to an interface on Cisco 6500 or 7600, it compiles it and puts into TCAM. The way Cisco 7600/6500 does it might have unintended consequences that can leave you open to DDoS attack. Let's consider following example:

We want to allow any server in 172.16.100.0/24 network to initiate any tcp connection and query any DNS server directly. Here is our ACL

ip access-list extended Test1
permit tcp any any established
permit udp any eq domain any
deny ip any any

We apply it to internet-facing interface of Cisco7600 router: "ip access-group Test1 in". Now let's look at what actually happened in TCAM:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments
    permit       tcp any any established match-any
    permit       udp any eq domain any

Our router automatically added "permit udp any any fragments", i.e. it allowed udp fragments. Now, let's see if it actually happens. First, take a look at the compiled ACL again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (41 matches)
    permit       tcp any any established match-any (220 matches)
    permit       udp any eq domain any

Not the counter - 41 matches. Next, on the "attacker" we'll generate fragmented UDP traffic targeting a server in 172.16.100.0/24 network:

hping2 -2 -d 1500 -c 1 -s 10000 -p 90 -m 500 -f 172.16.100.10

In the command above, we send 1 1500-byte UDP packet from port 10000 on local host to port 90 on 172.16.100.10 and we are telling the host that MTU is 500 bytes. On the target host we run tcpdump:

10:47:41.942010 IP (tos 0x0, ttl 63, id 130, offset 496, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942027 IP (tos 0x0, ttl 63, id 130, offset 1000, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942034 IP (tos 0x0, ttl 63, id 130, offset 1496, flags [none], length: 28) 172.16.0.101 > 172.16.100.10: udp

Now, the first fragment, containing IP and UDP header were dropped by our ACL, since we do not allow UDP packets coming from port 10000, but 3 other fragments got through. Let's check the counter again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (44 matches)
    permit       tcp any any established match-any (224 matches)
    permit       udp any eq domain any

The attacker can flood your web or email server with UDP fragments causing it to slow down while it is busy discarding incomplete packets. We can not block fragments completely since legitimate DNS replies can be quite big and require fragmentation. The solution would be to allow outbound UDP traffic and, hence, incoming replies only to specific hosts that need it. Like your caching DNS server and put good firewall in front of it.

Cisco 7600: Netflow and high CPU utilization

2010-05-24T14:59:00.000-04:00

Cisco documentation states, that:
If NetFlow is configured for version 7, the flow is performed by the Routing Processor, which could cause high CPU utilization.
For troubleshooting high CPU utilization due to Netflow version 7, configure mls nde sender version 5, as the Netflow export is performed by the SP, which is the default for version 5 or version 9.

It turns out, combination of NetFlow version 9 and NDE sender version 7 also creates high CPU load in certain situations. Here is the setup:

Both routers are Cisco 7604. Other than different IP addresses, the only difference between R1 and R2 was this:

on R1: mls nde sender
on R2: mls nde sender version 5
Default sender version is 7. Both routers configured with ip flow-export version 9.
When ever R2's eBGP session was interrupted, R1's CPU utilization skyrocketed to 100% and stayed there for 10-15 minutes rendering router unusable. "process cpu threshold" reported that "IP Input" was responsible for CPU load, not "BGP Router" as I expected, since these CPU
spikes only happened when eBGP session went down. After changing NDE sender version to 5 on R1, the problem went away.

Catching high CPU usage

2010-05-11T17:43:00.000-04:00

Suddenly your router stops responding and forwarding traffic, you can telnet into it, response on the console is very slow. Few minutes later everything is back to normal and only "show process cpu history" shows that CPU was at 100% for some time, but what caused it remains a mystery. To catch a process(es) that might have contributed to the problem, add following command in global configuration mode:

process cpu threshold type process rising 70 interval 5 falling 30 interval 5

It will generate syslog message every time CPU usage exceeds 70% for 5 or more seconds and falls below 30%. For example:

May 10 23:50:23.146 EDT: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilization(Total/Intr): 74%/26%, Top 3 processes(Pid/Util): 192/46%, 7/1%, 2/0%

Process id 192 contributed 46%. Let's see:

Router#sho proc cpu sor | i ^_192
192 904947881922327784 47 0.00% 0.18% 0.19% 0 IP Input

It was "IP Input" which is responsible for process-switching IP packets. Now we have something to work with and can start troubleshooting.

Monitoring logs with SEC

2010-03-26T15:07:00.002-04:00

Splunk seems to become de-facto standard tool for log management. But free version lacks feature that lets you configure and send alerts whenever certain events occur. One need to pay for enterprise version which starts at $5000 in US and Canada.

So, I use Simple Event Correlator to notify me of interesting events in life of my router friends. Here, for example, sec template to send me email with syslog line in the body when somebody tries to go to configuration mode and execute certain commands:

type=Single
ptype=RegExp
pattern=.*cmd=(configure|clear|ip|no|interface|switchport|router|spanning-tree)
desc=$0
action=pipe '$0' /usr/bin/mail -s "router/switch config change is happening right now" noc@example.com

You need to put this template into SEC configuration file and tell it were to look for these messages:

sec -detach -conf=/etc/sec-tacacs.conf -input=/var/log/tac-plus/account

In this case it's TACACS+ log file, so you need to configure a router to report such activities:

aaa new-model
aaa authentication login default group tacacs+ none
aaa authentication enable default group tacacs+ none
aaa authorization exec default group tacacs+ none
aaa authorization commands 15 default group tacacs+ none
aaa accounting commands 15 default start-stop group tacacs+
tacacs-server host <server ip>
tacacs-server <key>

Here is another template to report all syslog messages coming from devices with loopback interface IP address in the range 10.9.20.0/24 or 10.9.25.0/24. Why loopback? See my previous post.

type=Single
ptype=RegExp
pattern=(.*)10\.9\.2[0|5]\.(.*)%[A-Z]*
desc=$0
action=pipe '$0' /usr/bin/mail -s " router syslog message" noc@example.com

Best practices. Sort of.

2010-03-25T17:52:00.000-04:00

I tend to agree, that there is no "best practices", there are practices that fit best. Here is one of the thing that I always configure on the router.

There are many advantages in configuring Loopback interface when you use dynamic routing, but I also find loopback helpful for syslog reporting and authentication and authorization queries. So, I always configure:

ip tacacs source-interface Loopback0
logging source-interface Loopback0

Next step is to either add loopback interfaces of your routers to DNS or /etc/hosts file on Tacacs and syslog servers.
The names are no good if you can not use them. I prefer syslog-ng for logging, so, in order to record names instead of IP addresses, you need to configure use_dns(yes) in "options" section of syslog-ng.conf. For TACACS+: run tac_plus with "-L" option.

Making same change on many routers

2010-03-25T11:46:00.000-04:00

Suppose you need to make the same change on many routers, but do not have fancy software like Cisco Works to help you. No worries. Perl is the best friend of any network and system administrator. Here is the quick script that goes to a router and types command "logging source-interface loopback 0", saves configuration and exit. It can be used to run any command.
Place IP addresses of the routers, one per line, in file routers.txt. This file must be in the same directory as the script. Remember, you put your username, password and enable password in the script in clear text, so do not forget "chmod 700 "

#!/usr/bin/perl
use Net::Telnet::Cisco;
my $myfile="./routers.txt";
open (FH, $myfile) || die "Can not open $myfile\n";

while () {
chomp;
my $switchname=$_;
print "$switchname\n";

my $session = Net::Telnet::Cisco->new(Host => $switchname,Input_log => "$switchname.log");

# Replace username and password below with real username and password

$session->login('username', 'password');

# Enable mode
if ($session->enable("enable password") ) { # insert your enable passowrd
    @output = $session->cmd('configure terminal');
    @output = $session->cmd('logging source-interface loopback 0');
    print @output;
    @output = $session->cmd('exit');
    @output = $session->cmd("copy run startup-config\n\n");
    print @output;
    } else {
         warn "Can't enable: " . $session->errmsg;
        }
$session->close;
}

Use at your own risk.

Debian 5.0.4 on Dell 1950

2010-03-22T15:36:00.000-04:00

Normally, installing Debian on Dell servers is piece of cake. This particular 1950 came with Broadcom NICs and PERC5 controller. Debian 5.0.4 does not include driver for Broadcom drivers due to some copyright restrictions. However, the driver is available as deb package. Download it and copy to FAT or FAT32 formatted USB drive. When prompted for NIC driver during the installation process, insert USB drive into USB port. As soon as server loads the driver and moves to the next screen in installation process, remove the drive. If you do not remove the USB drive before installation process gets to partitioning, your drive sequence will we out of whack. You'll have to boot from CD and edit /etc/fstab.
Since this server has hardware I wanted to use instead of configuring software RAID in Linux. The question is how to monitor RAID state from Debian. There is no deb package or source code, but LSI provides RPM. I downloaded "MegaCLI - Linux" from "Miscellaneous" section, unpacked it, installed "alien" on Debian (sudo apt-get install aliean) and then "sudo alien -i MegaCli-1.01-0.i386.rpm". It install MegaCli under /opt/MegaRAID/MegaCli. Moritz Mertinkat has great emergency cheat sheet for MegaCli usage.

Cacti and 95th percentile

2010-01-06T15:41:00.006-05:00

I use Cacti to collect traffic data on my routers and I need to know what 95th percentile is. There are quite a ways to get 95th percentile line on Cacti graph. The problem with all those methods is that if time frame of the graph does not coincide with ISP billing period the 95th percentile value on the graph is useless. But all the necessary data is collected by Cacti into RRD file. All we have to do is to extract it.
First, I need to figure out where the RRD file is. In Cacti, go to Console -> Data Sources, select your edge router and click on IPS-facing interface. In "Data Source Path" field you'll see the name of the RRD file in Cacti's rra directory where data for this interface is stored

Second, we need to know what to extract from this file. I.e I need to know the names of RRD data sources:
rrdtool info border_router_1_traffic_in_14839.rrd
where border_router_1_traffic_in_14839.rrd is file name from previous step.

filename = "border_router_1_traffic_in_14839.rrd"
rrd_version = "0003"
step = 300
last_update = 1262806506
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = NaN
ds[traffic_in].last_ds = "437961211333"
ds[traffic_in].value = 4.9711447176e+05
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = NaN
ds[traffic_out].last_ds = "138465493978"
ds[traffic_out].value = 1.9428099668e+04
ds[traffic_out].unknown_sec = 0

truncated...

The data sources names are traffic_in and traffic_out and this is what we are going to extract. Before we proceed we need to remember, that RRD database size is fixed and determined at the time of creation. When limit is reached, oldest data is overwritten. To avoid losing any data, I am going to extract traffic numbers every hour for the last hour and put inbound and outbound data in separate files.
Incoming traffic:

rrdtool xport -s now-1h -e now DEF:xx=border_router_1_traffic_in_14839.rrd:traffic_in:AVERAGE CDEF:bb=xx,8,* XPORT:bb:"out bits" | grep \|grep -v Na | awk -F'' '{print $2}'| sed -e 's/<\/v><\/row>//'|sed -e 's/e+0/\t/' >> incoming.txt

Outgoing traffic:

rrdtool xport -s now-1h -e now \DEF:xx=border_router_1_traffic_in_14839.rrd:traffic_out:AVERAGE CDEF:bb=xx,8,* XPORT:bb:"out bits" | grep \|grep -v Na | awk -F'' '{print $2}'| sed -e 's/<\/v><\/row>//'|sed -e 's/e+0/\t/' >> outgoing.txt

Both commands should be in one line. Above I converted Bytes/sec into Bits/sec and removed XML formatting. You need these 2 lines into shell script and run it from cron every hour on 2 minutes after the hour so Cacti has time to finish collecting on top of the hour. You'll get 2 files - incoming.txt and outgoing.txt looking like this

6.9655133612 7
7.0568998690 7
6.9008144000 7
7.0245826541 7
7.2076520540 7
6.7448901179 7
6.7471832197 7
6.7365174531 7
6.9710477122 7
7.1586411237 7
7.0991637699 7
7.0189321194 7

This are measurements taken every 5 minutes by Cacti. "6.9655133612 7" means 6.9655133612 * 10^7 bits/sec or 69655133.612 bits/sec.

Now all you have to do on the 1st of the month right after midnight is to convert the data to get rid of second column, sort it and remove top 5%. For 30-day month:

cat incoming.txt |perl -e ' while(<>) {$input = $_; chomp($input);($traffic, $power)
=split(/\t/,$input); $traffic = $traffic*10**$power; print "$traffic\n";}'|egrep -v '^0$'|sort -n -r | head -433 |tail -1

How to generate lots of BGP routes

2009-10-14T14:37:00.004-04:00

I needed to test in the lab whether my Cisco router can handle more than 300K routes - size of current full BGP table. Now, Cisco router can only accept 200 network statements under router bgp configuration, so I would need 1500 routers. Even if I had that many routers to my disposal, it would have taken days to configure all of them. As always, open source software can help. Quagga lets you run OSPF, BGP, RIP, RIPng on Linux and Solaris. If you go with all default options, it is very easy to install. Download and unpack. Go to quagga directory, in my case it was quagga-0.98.6, type
./configure
make
sudo make install
That's it. By default, it went into /usr/local/. I have Debian 4 (Etch) with 2.6.8 kernel and a lot of development packages installed. The only thing I had to do was to add /usr/local/lib to /etc/ld.so.conf file and run /sbin/ldconfig.
Here is the setup

Now I need a valid configuration file for Quagga BGP. Adding 300000 network statements manually is not something system administrators do on Linux. Hence, here is the script

#!/usr/bin/perl

my $host="quagga-host";         #quagga router name
my $logpass="zebra";            #login password
my $enable="zebra";             #enable password
my $myasn="65099";              #local AS number
my $router_id="172.31.2.2";     #bgp router-id
my $remote_as="65001";          #remote-as number
my $remote_ip="172.31.2.1";     #BGP neighbor ip address
my $route_count=0;
my $max_routes=300000;              #max number of routers to generate

open (BGPCONF,'>bgpd.conf')|| die "Can not open bgpd.conf for writing";
print BGPCONF "hostname $host\npassword $logpass\nenable password $enable\nline vty \n";
print BGPCONF "router bgp $myasn\n  bgp router-id $router_id\n  neighbor $remote_ip remote-as $remote_as\n";
MAXR: while ($route_count <= $max_routes ) { 
$octet1=int(rand(223))+1; #generate 1st octet randomly in 1-223 range, 224 and up is multicust and class E  
if ($octet1 ==127) {next;} #need to make sure that 127.X.X.0/24 is excluded

$octet2=0;  
while ( $octet2 <= 255 ){
$octet3=0;
while ( $octet3 <= 255 ) {
print BGPCONF "  network $octet1\.$octet2\.$octet3\.0/24\n";
$octet3++;
$route_count++;
if ($route_count == $max_routes) {last MAXR;}
}
$octet2++;
}
}
close BGPCONF;

this script will generate bgpd.conf for Quagga. Since it is lab environment not connected to any real network, I do not really care about zebra configuration or restricting access to Quagga BGP console. Copy bgpd.conf file into /usr/local/etc and run /usr/local/sbin/bgpd -d -f /usr/local/etc/bgpd.conf -u root -g root Again, this is not production environment. Do not run Quagga as root in production. Here is relevant configuration from Cisco router:

interface GigabitEthernet0/0
ip address 172.31.2.1 255.255.255.0
network 172.31.2.0 mask 255.255.255.0
media-type rj45
negotiation auto
!
router bgp 65001
no synchronization
bgp log-neighbor-changes
neighbor 172.31.2.2 remote-as 65099
no auto-summary
!

Let's see if it works. On Linux host:

sh-2.05b$ telnet localhost 2605
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.

Hello, this is Quagga (version 0.98.6).
Copyright 1996-2005 Kunihiro Ishiguro, et al.


User Access Verification

Password:
quagga-host> sho ip bgp summ
BGP router identifier 172.31.2.2, local AS number 65099
2 BGP AS-PATH entries
0 BGP community entries

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
172.31.2.1      4 65001     283     606        0    0    0 03:41:55        1

Total number of neighbors 1
quagga-host>

on Cisco:

R1#sho ip bgp sum
BGP router identifier 192.0.2.2, local AS number 65001
BGP table version is 330002, main routing table version 330002
310001 network entries using 40920132 bytes of memory
310001 path entries using 16120052 bytes of memory
3/2 BGP path/bestpath attribute entries using 444 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
Bitfield cache entries: current 1 (at peak 2) using 32 bytes of memory
BGP using 57040684 total bytes of memory
BGP activity 310001/0 prefixes, 320001/10000 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
172.31.2.2      4 65099     602     322   330002    0    0 03:38:59   310000

Yep. It works.

BGP and BFD (Bidirectional Forwading Detection)

2009-10-09T16:08:00.006-04:00

If you have Ethernet uplink to your ISP, the chances are high that it looks like this:

The Layer2 device could be "on the wire" provider or Ethernet over MPLS service. The problem arises when, for example, connection between Layer2 device and ISP router goes down

The BGP session with R1 on ISP router will reset immediately, unless you configured "no bgp fast-external-fallover". But R1 will rely on BGP hello messages to detect if neighbor still alive. It might take R1 up to 3 minutes detect that ISP is not available and for these 3 minutes R1 will be sending traffic to black hole instead of re-converging and sending traffic to your backup link. You have backup link, don't you?
Here are the syslog messages from R1 and ISP routers. To imitate link failure I shutdown interface on ISP

router ISP:

Oct 9 17:55:18 UTC: %LINK-5-CHANGED: Interface GigabitEthernet2/5, changed state to administratively down
Oct 9 17:55:18 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.1 Down Interface flap

router R1:

Oct 9 17:57:24 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.2 Down BGP Notification sent
Oct 9 17:57:24 UTC: %BGP-3-NOTIFICATION: sent to neighbor 172.31.255.2 4/0 (hold time expired) 0 bytes

Note the timestamps of first and last messages. That's not good, especially if every minute of downtime costs you a bundle. You can adjust bgp timers, but lowest you can go is 1 second and it could be hard on CPU.
BFD protocol allows you to go to microseconds level. It is very lightweight and easy to configure.
On interfaces facing Layer2 device apply command:

bfd interval 100 min_rx 100 multiplier 3

To check if BFD is configured properly:

#sho bfd neighbor

OurAddr NeighAddr LD/RD RH/RS Holddown(mult) State Int
172.31.255.2 172.31.255.1 3/7 Up 0 (3 ) Up Gi2/5

under "router bgp" configuration:

neighbor [neighbor IP] fall-over bfd

Now let's imitate link failure again. Shutdown interface on router ISP:

Oct 9 18:14:27.408 UTC: %LINK-5-CHANGED: Interface GigabitEthernet2/5, changed state to administratively down
Oct 9 18:14:27.408 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.1 Down Interface flap

On R1

Oct 9 18:14:27.673 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.2 Down BFD adjacency down

The difference now in milliseconds. The hard part is to convince your ISP to configure BFD on their side.
At this moment Cisco supports BFD on Ethernet interfaces only and only for directly connected BGP peers, i.e. no multi-hop BGP.
BFD requires UDP ports 3784 and 3785 to be open in case you have ACL applied to your uplink interface.

CCIE R&S v4.0

2009-10-06T15:07:00.003-04:00

I just attended webcast about new CCIE R&S written and lab exam. It looks like MPLS portion of the exam is going to be much easier than 642-611 exam which is required for CCIP certification.

bpduguard vs. bpdufilter

2009-09-16T13:34:00.002-04:00

Suppose you have a switch with servers connected to it. It's called "access layer switch" in Cisco lingo. There is no need to run spannig-tree on these server-facing ports. So, you disable it with "switchport portfast" interface command. One of the benefits of disabling STP on the port is that if you server boots fast, it does not need to wait until port finishes going through all STP port states and can start transmitting data immediately.
Since servers normally should ignore BPDUs coming from switch, there is no need to send them to a server in first place. To filter out outgoing BPDUs apply interface command "spanning-tree bpdufilter enable".
But when BPDU is received on the port with bpdufilter enabled, the port is portfast status is disabled and port will participate in spanning-tree. At the time network needs to be protected from unauthorized device that might decide to participate in your spanning-tree topology and cause spanning-tree loop or try to hijack root. Interface command "spanning-tree bpduguard enable" puts interface in err-disable mode whenever BPDU is received from connected device.
What will happen if you have both bpdufilter and bpduguard enabled on the interface?

spanning-tree portfast
spanning-tree bpdufilter enable
spanning-tree bpduguard enable

bpdufilter takes precedence and bpdugard does not work. Although bpduguard needs more administrative overhead - port needs to be enabled manually - it makes your network more secure.

Cisco router as Frame-Relay switch

2009-08-17T16:15:00.002-04:00

There are a lot of examples on the 'Net how to configure Cisco router to act as Frame Relay switch. All of them use command "frame-relay route". Here is (one more) example

For the topology on the picture above, the configuration would look like this

interface Serial2/0
no ip address
encapsulation frame-relay
no fair-queue
clock rate 128000
frame-relay intf-type dce
frame-relay route 103 interface Serial2/2 301
!
interface Serial2/2
no ip address
encapsulation frame-relay
clock rate 128000
frame-relay intf-type dce
frame-relay route 301 interface Serial2/0 103
end

Here is another way to configure:

interface Serial2/0
no ip address
encapsulation frame-relay
no fair-queue
clock rate 128000
frame-relay intf-type dce
!
interface Serial2/2
no ip address
encapsulation frame-relay
clock rate 128000
frame-relay intf-type dce
!
connect mylink Serial2/0 103 Serial2/2 301

In latter case I used command "connect". Unlike "frame-relay route", "connect" command can be used only once for each pair of DLCIs that need to be able to talk to each other. If you need to have many Frame Relay links, using "connect" is less error-prone and configuration is easier to read.
To see connection status:

#show connection all

ID Name Segment 1 Segment 2 State
========================================
2 mylink Se2/0 103 Se2/2 301 UP

Need to find out Cisco switch serial number?

2009-07-20T12:02:00.003-04:00

Yes, you can do it by logging in and running "show idprom backplane" or "show inventory". What if you have hundreds of switches and need to update your inventory list. Going to every switch, typing commands and writing down serial numbers is very time consuming. There is also Expect script, but I am not a big fan of putting clear text passwords into code. So, my favorite SNMP to the rescue.
Serial number OID for modular switches (4500 and 6500) is .1.3.6.1.2.1.47.1.1.1.1.11.1. For Catalyst 2960 and 3500: 1.3.6.1.2.1.47.1.1.1.1.11.1001

Here is the quick shell script to get serial number and location of each switch.

for i in `cat switches.txt`
do
serial=`snmpwalk -Ov -Oq -v2c -c public $i 1.3.6.1.2.1.47.1.1.1.1.11.1001`
location=`snmpwalk -Ov -Oq -v2c -c public $i system.sysLocation`
echo -e "$i \t $serial \t $location"
done

File switches.txt contains ip addresses of my access switches, one per line. It goes without saying that each switch must have "snmp-server location" configured.

JUNOS as a second language

2009-06-22T16:20:00.004-04:00

Juniper was kind enough to post "JUNOS As A Second Language" e-learning course aimed at network administrators familiar with Cisco IOS. As somebody who works in production environment I could not help but notice how easy it is to revert changes in JUNOS with "commit confirmed" command.
Well, it turns out that IOS 12.4 has similar feature with "configure replace". From Cisco documentation:

Configuration Rollback Confirmed Change

The Configuration Rollback Confirmed Change feature enables an added criteria of a confirmation to configuration changes. This functionality enables a rollback to occur if a confirmation of the requested changes is not received in a configured time frame. Command failures can also be configured to trigger a configuration rollback.

The following steps outline how this process is achieved:

1. When entering configuration mode, this new option allows you to request confirmation (a confirmation time limit must be supplied) of the configuration changes.

2. After exiting configuration mode, you must enter the confirmation command. If no confirmation is entered within the requested time limit, the configuration will revert to its previous state.

configure replace target-url [nolock] [list] [force] [ignorecase] [revert trigger [error] [timer minutes] | time minutes]

It might come in handy next time when you change IP address on the wrong interface or make mistake in ACL.
Disclaimer: I strongly advise that you have console server with out-of-band access connected to all your critical production routers.

Last Error: PCALC:: No addresses to connect

2009-06-19T10:38:00.010-04:00

Ever since I passed "Implementing Cisco MPLS" (642-611) exam, I've been aching to try my hands at MPLS TE. So, here is the task:

Click on image to see bigger version.

Since R2-R4 link is T3, OSPF will pick R2-R3-R5 path from R1 to either R6 or R7 leaving R2-R4-R5 link underutilized. At the same time I know that traffic from R1 to 10.0.0.0/24 is going to low enough to fit comfortably into T3 link. I wanted to create explicit path tunnel R1-R2-R4-R5-R7 with backup dynamic tunnel. I am going to skip configuration part - there are more then enough examples on the 'Net.
Let's see if tunnel is up

R1#sho mpls traffic-eng tunnels

Name: 6503sw2_t0 (Tunnel0) Destination: 192.168.1.4
Status:
Admin: up Oper: down Path: not valid Signalling: Down path option 1, type explicit mypath

Config Parameters:
Bandwidth: 0 kbps (Global) Priority: 7 7 Affinity: 0x0/0xFFFF
Metric Type: TE (default)
AutoRoute: enabled LockDown: disabled Loadshare: 0 bw-based
auto-bw: disabled

History:
Tunnel:
Time since created: 20 hours, 1 minutes
Time since path change: 3 minutes, 24 seconds
Number of LSP IDs (Tun_Instances) used: 334
Prior LSP:
ID: path option 1 [309]
Removal Trigger: path option removed
Last Error: PCALC:: No addresses to connect 172.16.0.14 to 172.16.0.10

Not good. Searching for this type of error message gives one sage advise:

For cases involving an explicit path option, you can try to narrow down the problem by first checking every hop in the explicit hop list. You can also try to back down the tunnel: Move the tail one hop back each time to see if the tunnel comes up. If the tunnel comes up when you move the tail to a previous hop, you can conclude that there is a problem between that hop and the next hop, and you can start scrutinizing that hop carefully.

And so I did. Let's see how R2 and R5 see R4:

R2# show ip rsvp neighbor
Neighbor Encapsulation Time since msg rcvd/sent
172.16.0.18 Raw IP 00:02:12 00:02:10
172.16.0.21 Raw IP 00:02:10 00:02:23

R5#show ip rsvp neighbor
Neighbor Encapsulation Time since msg rcvd/sent
172.16.0.1 Raw IP 00:03:59 00:04:18
172.16.0.6 Raw IP 00:04:16 00:03:59

Hmm, they do not see R4 as RSVP neighbor at all. What's going on on R4?

R4# sho ip rsvp interface detail gi 0/0
Gi0/0:
Interface State: Up
Bandwidth:
Curr allocated: 0 bits/sec
Max. allowed (total): 30M bits/sec
Max. allowed (per flow): 30M bits/sec
Max. allowed for LSP tunnels using sub-pools: 0 bits/sec
Set aside by policy (total): 0 bits/sec
Admission Control:
Header Compression methods supported:
rtp (36 bytes-saved), udp (20 bytes-saved)
Traffic Control:
RSVP Data Packet Classification is ON
Signalling:
DSCP value used in RSVP msgs: 0x3F
Number of refresh intervals to enforce blockade state: 4
Number of missed refresh messages: 4
Refresh interval: 30
Authentication: disabled

How does it compare with R4-facing interface on R5?

R5#sho ip rsvp interface detail gi 0/2

Gi0/2:
RSVP: Enabled
Interface State: Up
Bandwidth:
Curr allocated: 0 bits/sec
Max. allowed (total): 40M bits/sec
Max. allowed (per flow): 40M bits/sec
Max. allowed for LSP tunnels using sub-pools: 0 bits/sec
Set aside by policy (total): 0 bits/sec
Admission Control:
Header Compression methods supported:
rtp (36 bytes-saved), udp (20 bytes-saved)
Traffic Control:
RSVP Data Packet Classification is ON via CEF callbacks
Signalling:
DSCP value used in RSVP msgs: 0x3F
Number of refresh intervals to enforce blockade state: 4
Authentication: disabled
Key chain:
Type: md5
Window size: 1
Challenge: disabled
Hello Extension:
State: Disabled

"RSVP: Enabled" string is missing in R4 command output. After some research I found that I need "T" type IOS to run RSVP-TE on R4 which is Cisco 3825.
Let's see what we have after the upgrade:

R4#sho ip rsvp interface detail gi 0/0

Gi0/0:
RSVP: Enabled
Interface State: Up
Bandwidth:
Curr allocated: 5M bits/sec
Max. allowed (total): 30M bits/sec
Max. allowed (per flow): 30M bits/sec
Max. allowed for LSP tunnels using sub-pools: 0 bits/sec
Set aside by policy (total): 0 bits/sec
Admission Control:
Header Compression methods supported:
rtp (36 bytes-saved), udp (20 bytes-saved)
Traffic Control:
RSVP Data Packet Classification is ON
Signalling:
DSCP value used in RSVP msgs: 0x3F
Number of refresh intervals to enforce blockade state: 4
Authentication: disabled
Key chain:
Type: md5
Window size: 1
Challenge: disabled
Hello Extension:
State: Disabled

and tunnel:

R1#sho mpls traffic-eng tunnels bri
Signalling Summary:
LSP Tunnels Process: running
Passive LSP Listener: running
RSVP Process: running
Forwarding: enabled
Periodic reoptimization: every 3600 seconds, next in 3042 seconds
Periodic FRR Promotion: Not Running
Periodic auto-bw collection: every 300 seconds, next in 42 seconds
TUNNEL NAME DESTINATION UP IF DOWN IF STATE/PROT
6503sw2_t0 192.168.1.4 - Gi2/1 up/up

Much better. Confirm routing

R1#sho ip route 10.0.0.0 255.255.255.0
Routing entry for 10.0.0.0/24
Known via "ospf 100", distance 110, metric 14, type intra area
Last update from 192.168.1.4 on Tunnel0, 19:41:13 ago
Routing Descriptor Blocks:
* 192.168.1.4, from 192.168.1.4, 19:41:13 ago, via Tunnel0
Route metric is 14, traffic share count is 1

Now, since MPLS TE tunnel is unidirectional, I need to create R7-R5-R4-R2-R1 tunnel as well.

CCIP accomplished

2009-05-21T14:35:00.004-04:00

Yesterday I passed "Implementing Cisco MPLS" (642-611) exam. That was last one for me to complete CCIP certification process. I want to share my "CCIP reading list":

Routing TCP/IP, Volume 1 (2nd Edition)(CCIE Professional Development)

Routing TCP/IP, Volume II (CCIE Professional Development)

BGP Design and Implementation

Cisco QOS Exam Certification Guide (IP Telephony Self-Study) (2nd Edition)

MPLS Fundamentals

The list is not exclusive. In addition to that I used Cisco's command reference guide and IOS documentation on Cisco's website and built small lab to practice. I also tried using exam simulators like RealExams, a.k.a. Pass4Sure, a.k.a. TestKing. Actually, I am not sure if it is the same company or they steal from each other, but simulators' content is almost identical and I found all of them to be waste of money - ridden with errors and outdated. For 642-611 exam, I tried Bosons Software simulator. This one was of much better quality, but turned out to be too easy to be a good help for MPLS exam preparation. Even if you manage to achieve 80-90% accuracy in this simulator, your real exam score won't be high enough to pass.

2 more Netscreen's OIDs

2009-05-05T11:52:00.003-04:00

1.3.6.1.4.1.3224.4.1.1.1.27.X shows what Netscreen thinks latency (in ms) is through IPSec tunnel number X. See my old post on how to find tunnel number.

Here is another one. How much data was transferred via policy number Y.

1.3.6.1.4.1.3224.10.2.1.6.Y.0
Beware, it reports in Bytes/sec. In order to make it work, you'll have to configure policy with "count" option.

3 down, 1 more to go

2009-03-19T14:16:00.002-04:00

Yesterday I passed "Implementing Cisco QoS" exam 642-642. I need to pass 642-611 "Implementing Cisco MPLS" to complete CCIP certification.

Tracking routing entries on Cisco7600 via SNMP

2009-03-10T13:59:00.004-04:00

I have Cisco7604 with SUP720 running c7600rsp72043-ipservicesk9-mz.122-33.SRD.bin. Cisco's SNMP object navigator claims that this image supports RFC1213-MIB which has ipRouteTable. In reality here is what I got

~$ snmpwalk -v2c -c public [myrouter] ipRouteDest
RFC1213-MIB::ipRouteDest = No Such Object available on this agent at this OID

After "Asking" around, I found RFC2096, that states:

The ipForwardTable updates the RFC 1213 ipRouteTable to display multipath IP Routes. This is in turn obsoleted by the ipCidrRouteTable.

The ipCidrRouteTable updates the RFC 1213 ipRouteTable to display multipath IP Routes having the same network number but differing network masks.

ipCidrRouteTable is actually supported by my IOS image, even though Cisco's SNMP object navigator declares it "depricated" and recommends using inetCidrRouteTable (1.3.6.1.2.1.4.24.7)

~$ snmpwalk -v2c -c public [myrouter] inetCidrRouteTable
IP-FORWARD-MIB::inetCidrRouteTable = No Such Object available on this agent at this OID

No luck again, but ipCidrRouteTable (numeric 1.3.6.1.2.1.4.24.4) works just fine.

Here is the list of the objects in ipCidrRouteTable
ipCidrRouteTable 1.3.6.1.2.1.4.24.4
ipCidrRouteEntry 1.3.6.1.2.1.4.24.4.1
ipCidrRouteDest 1.3.6.1.2.1.4.24.4.1.1
ipCidrRouteNextHopAS 1.3.6.1.2.1.4.24.4.1.10
ipCidrRouteMetric1 1.3.6.1.2.1.4.24.4.1.11
ipCidrRouteMetric2 1.3.6.1.2.1.4.24.4.1.12
ipCidrRouteMetric3 1.3.6.1.2.1.4.24.4.1.13
ipCidrRouteMetric4 1.3.6.1.2.1.4.24.4.1.14
ipCidrRouteMetric5 1.3.6.1.2.1.4.24.4.1.15
ipCidrRouteStatus 1.3.6.1.2.1.4.24.4.1.16
ipCidrRouteMask 1.3.6.1.2.1.4.24.4.1.2
ipCidrRouteTos 1.3.6.1.2.1.4.24.4.1.3
ipCidrRouteNextHop 1.3.6.1.2.1.4.24.4.1.4
ipCidrRouteIfIndex 1.3.6.1.2.1.4.24.4.1.5
ipCidrRouteType 1.3.6.1.2.1.4.24.4.1.6
ipCidrRouteProto 1.3.6.1.2.1.4.24.4.1.7
ipCidrRouteAge 1.3.6.1.2.1.4.24.4.1.8
ipCidrRouteInfo 1.3.6.1.2.1.4.24.4.1.9

If you run BGP and accept all routes from the internet, it might be useful to watch the size of your routing table. ipCidrRouteNumber (1.3.6.1.2.1.4.24.3)does just that. If you want to graph it in Cacti use 1.3.6.1.2.1.4.24.3.0 For some reason Cacti needs .0 in the end

I worked in environment like this once.

2009-02-25T13:18:00.004-05:00

This is what I call really bad cabling.

Connect F5 LTM/GTM to Perle IOLAN console server

2009-02-13T14:01:00.003-05:00

I personally prefer Cyclades (now part of Avocent) console servers since they run Linux and I can use vi to edit configuration files. The only problem is that Cyclade's console servers come with only one Ethernet port. What if a switch to which Cyclade console server is connected via Ethernet dies and you need to access its console to figure out what is going on? You can always hook up modem to you console server and dial in, but in this case only one person can connect and with proliferation of VoIP analog phone lines in offices are getting hard to come by.
So, after some search I found Perle's IOLAN console server which have 2 Ethernet ports. These ports can be used in Active/Standby configuration, meaning you can assign 1 IP address and when 1st Ethernet port goes down, IOLAN moves this IP address automatically to second port. For this feature alone I am willing to tolerate their Windows configuration client software.

Now I need to connect IOLAN to F5 LTM/GTM console. Since LTM's console is DB9 male connector and IOLAN is RJ45, I need RJ45-DB9 female converter. I buy these converters at pccables.com. The wires in that converter on RJ45 side are connected like this:
1 Blue
2 Orange
3 Black
4 Red
5 Green
6 Yellow
7 Brown
8 White

I need to plug those wires into DB9 side of the converter like this
RJ45 DB9
Blue 6
Orange 8
Black 4
Red 2
Green 3
Yellow 5
Brown 7
White 1

Now, take regular straight network cable, plug one end into RJ45-DB9 converter, another into IOLAN port. Done. Do not forget that by default IOLAN's port speed is 9600 and F5's is 19200.

Packetization Layer Path MTU Discovery

2009-02-04T12:01:00.006-05:00

A lot of network administrators block all ICMP traffic on the network's edge. Although some ICMP packets can be used in DDoS, ICMP type 3 plays important role in Paht MTU discovery. If ICMP is completely blocked, sending side can not reliably determine MTU and that can lead to re-transmissions and slower data transfer. Sometimes, ICMP blocking network could be outside your control, so you can not change ACL or firewall rules. In this case Packetization Layer Path MTU Discovery can be used. It utilizes TCP for maximum packet size discovery (RFC4821). To enable PLPMTUD on Linux:
echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
or
in /etc/sysctl.conf net.ipv4.tcp_mtu_probing = 2

Exam 642-901

2008-11-21T13:19:00.002-05:00

I just passed Cisco's 642-901 exam - Building Scalable Cisco Internetworks. What interesting, though, is that I scored higher for OSPF than for BGP and EIGRP sections of the exam, even though I work with BGP and EIGRP every day. I guess I got overconfident in what I thought I knew. On the other hand, I knew my weak OSPF spots and payed more attention.