CCIE R&S written

After getting CCIP and CCNP certifications I did not have a reason to postpone taking CCIE written exam any longer. So, I went for it last Wednesday and passed it. CCIE R&S written covers broad range of topics. All these topics are covered in depth by exams required to obtain CCIP and CCNP. If you took and passed ROUTE, SWITCH, TSHOOT, QOS, BGP and MPLS exams, CCIE R&S written looks fairly easy and does not require any additional preparations.

Cisco TSHOOT exam great success

My second shot at TSHOOT exam was much better. I went to another testing center and did not experience a single glitch. Out of 35-40 questions I got only one wrong.
I used Boson's practice exam during the preparation to one of the previous Cisco exams before and was really disappointed. The practice exam was not hard enough and there were few errors, but I was very pleased with their TSHOOT simulator. It is probably even more difficult than real Cisco exam, but no simulator can replace on-the-job experience.

Cisco TSHOOT exam epic fail

I have one last exam left to add CCNP trophy to my CCNA and CCIP. The exam is new Cisco TSHOOT. I've read a lot of good reviews about the exam and demo looks great.

The exam has 2 parts: multiple choice questions and tickets. After I finished multiple choice part, exam software crashed throwing some Flash error. I was given another computer and it crashed again in the same place. Kind folks at the testing center spent about an hour on the phone with tech support only to see test exam software crashing again and again with the same Flash error.
It looks like Steve Jobs was onto something when he banned Flash from iPad.
So, I have to re-schedule the exam and hope that next time it works.

OSPFv3-EIGRPv6 redistribution. Strange "include-connected" behavior.

I've been practicing IPv6 and notice strange thing with "include-connected" option when redistributing between EIGRPv6 and OSPFv3. First, short preamble. In IPv4 redistribution, when source protocol runs on connected network, this network is automatically included in redistribution in destination protocol. Apparently, this is not the case in IPv6. You have to include connected network explicitly. Here is my network:

All routers run IOS 12.4(22)T
R2 and R3 are doing redistribution:

R2#sho run | sec ipv6 router
ipv6 router ospf 11
log-adjacency-changes
redistribute rip RIPng include-connected
ipv6 router rip RIPng
redistribute ospf 11 metric 5 include-connected

R3#sho run | sec ipv6 router
ipv6 router eigrp 1
no shutdown
default-metric 1000000 1 255 1 1500
redistribute ospf 11 include-connected
ipv6 router ospf 11
log-adjacency-changes
default-metric 30
redistribute eigrp 1 include-connected

Notice "include-connected" in redistribute command. Now, let's see what routing table looks like on R1 and R4:

R4>sho ipv6 route
EX 2000:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2000:2::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2000:4::/64 [0/0]
via Loopback10, directly connected
L 2000:4::1/128 [0/0]
via Loopback10, receive
EX 2001:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2001:3::/64 [0/0]
via GigabitEthernet0/3, directly connected
L 2001:3::2/128 [0/0]
via GigabitEthernet0/3, receive
L FF00::/8 [0/0]
via Null0, receive


R1>1>sho ipv6 route
C 2000:1::/64 [0/0]
via Loopback10, directly connected
L 2000:1::1/128 [0/0]
via Loopback10, receive
R 2000:2::/64 [120/2]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:3::1/128 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:4::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
C 2001:1::/64 [0/0]
via GigabitEthernet0/2, directly connected
L 2001:1::1/128 [0/0]
via GigabitEthernet0/2, receive
R 2001:2::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
L FF00::/8 [0/0]
via Null0, receive

R1 does not have 2001:3::/64 network and R4 is missing 2001:2::/64.
2001:3::/64 is EIGRPv6 network and is supposed to be redistributed into OSPFv3 and then into RIPng.
2001:2::/64 is OSPFv3 network and should be redistributed into EIGRPv6.
So, even though "include-connected" is added to "redistribute ospf" and "redistribute eigrp" it does not seem to work. At the same time, on R2, RIPng redistribute connected OSPF networks.
Let's add explicit "redistribute connected" statements on R3.

R3(config)#ipv6 router eigrp 1
R3(config-rtr)#redistribute connected
R3(config-rtr)#ipv6 router ospf 11
R3(config-rtr)#redistribute connected
R3(config-rtr)#end

Let's check R1 and R4 again:

R1>sho ipv6 route
C 2000:1::/64 [0/0]
via Loopback10, directly connected
L 2000:1::1/128 [0/0]
via Loopback10, receive
R 2000:2::/64 [120/2]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:3::1/128 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:4::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
C 2001:1::/64 [0/0]
via GigabitEthernet0/2, directly connected
L 2001:1::1/128 [0/0]
via GigabitEthernet0/2, receive
R 2001:2::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2001:3::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
L FF00::/8 [0/0]
via Null0, receive

R4>show ipv6 route
EX 2000:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2000:2::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2000:3::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2000:4::/64 [0/0]
via Loopback10, directly connected
L 2000:4::1/128 [0/0]
via Loopback10, receive
EX 2001:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2001:2::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2001:3::/64 [0/0]
via GigabitEthernet0/3, directly connected
L 2001:3::2/128 [0/0]
via GigabitEthernet0/3, receive
L FF00::/8 [0/0]
via Null0, receive

It works now. To make sure:

R4>traceroute 2000:1::1

Type escape sequence to abort.
Tracing the route to 2000:1::1

1 2001:3::1 0 msec 0 msec 0 msec
2 2001:2::1 0 msec 4 msec 0 msec
3 2001:1::1 0 msec 0 msec 0 msec

So far I was not able to find if this is intended behavior.

Cisco 642-813 SWITCH exam

I recently took and passed Cisco 642-813 SWITCH exam. All those things you might have read about crashing simulators, poorly worded or bad grammar questions are true. In my case sim crashed 3 times when I entered same command, so I simply skipped the command just to continue with the exam. It made task incomplete and cost me few precious points. And do not even try Cisco's practice tests - there are too many incorrect answers to take these tests seriously.
All this is not to say that test is impossible to pass, but if you are not in a rush, I'd recommend to wait a few months before taking it. Hopefully, Cisco will clean up its act.

Monitoring trunk status via SNMP

If you have not guessed yet, SNMP and monitoring are my favorites.

So, you have configured many trunks on your switch and now need to make sure all of them are actually in trunking mode. Here is 2 SNMP OID that can help you:

vlanTrunkPortDynamicState (1.3.6.1.4.1.9.9.46.1.6.1.1.13) - reports administrative state. From Cisco SNMP object navigator:
1 : on
2 : off
3 : desirable
4 : auto
5 : onNoNegotiate

vlanTrunkPortDynamicStatus (1.3.6.1.4.1.9.9.46.1.6.1.1.14) - reports operational state.
1 : trunking
2 : notTrunking

To get data for specific interface you need to add ifIndex to the end of the OID. For example, for interface ifIndex=10147
snmpwalk -v2c -Ov -Oq -c public myswitch  1.3.6.1.4.1.9.9.46.1.6.1.1.13.10147
To get ifIndex, you can either run  "show snmp mib ifmib ifIndex" command in exec mode or query ifName OID with snmpwalk. Here is the quick script:

  for int in ifIndex1 ifIndex2 ifIndexN
        do
        trunkoperstatus=`snmpwalk -v2c -Ov -Oq -c public myswitch \ 1.3.6.1.4.1.9.9.46.1.6.1.1.14.$int`
                if [ $trunkoperstatus -eq 2 ]
                then
                        trunkadminstatus=`snmpwalk -v2c -Ov -Oq -c public myswitch \ 1.3.6.1.4.1.9.9.46.1.6.1.1.13.$int`
                        if [ $trunkadminstatus -eq 1 ]
                        then
                                echo myswitch $int NotTrunking
                        fi
                fi
        done

Cisco 6500/7600 ACL side effect

When you apply ACL to an interface on Cisco 6500 or 7600, it compiles it and puts into TCAM. The way Cisco 7600/6500 does it might have unintended consequences that can leave you open to DDoS attack. Let's consider following example:

We want to allow any server in 172.16.100.0/24 network to initiate any tcp connection and query any DNS server directly. Here is our ACL

ip access-list extended Test1
 permit tcp any any established
 permit udp any eq domain any
 deny  ip any any

 We apply it to internet-facing interface of Cisco7600 router: "ip access-group Test1 in". Now let's look at what actually happened in TCAM:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared


Entries from Bank 0


Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments
    permit       tcp any any established match-any
    permit       udp any eq domain any

Our router automatically added "permit  udp any any fragments", i.e. it allowed udp fragments. Now, let's see if it actually happens. First, take a look at the compiled ACL again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared


Entries from Bank 0


Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (41 matches)
    permit       tcp any any established match-any (220 matches)
    permit       udp any eq domain any

Not the counter - 41 matches. Next, on the "attacker" we'll generate fragmented UDP traffic targeting a server in 172.16.100.0/24 network:

hping2 -2 -d 1500 -c 1 -s 10000 -p 90 -m 500 -f 172.16.100.10

In the command above, we send 1 1500-byte UDP packet from port 10000 on local host to port 90 on 172.16.100.10 and we are telling the host that MTU is 500 bytes. On the target host we run tcpdump:

10:47:41.942010 IP (tos 0x0, ttl  63, id 130, offset 496, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942027 IP (tos 0x0, ttl  63, id 130, offset 1000, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942034 IP (tos 0x0, ttl  63, id 130, offset 1496, flags [none], length: 28) 172.16.0.101 > 172.16.100.10: udp

 Now, the first fragment, containing IP and UDP header were dropped by our ACL, since we do not allow UDP packets coming from port 10000, but 3 other fragments got through. Let's check the counter again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared


Entries from Bank 0


Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (44 matches)
    permit       tcp any any established match-any (224 matches)
    permit       udp any eq domain any

The attacker can flood your web or email server with UDP fragments causing it to slow down while it is busy discarding incomplete packets. We can not block fragments completely since legitimate DNS replies can be quite big and require fragmentation. The solution would be to allow outbound UDP traffic and, hence, incoming replies only to specific hosts that need it. Like your caching DNS server and put good firewall in front of it.

Cisco 7600: Netflow and high CPU utilization

Cisco documentation states, that:
 If NetFlow is configured for version 7, the flow is performed by the Routing Processor, which could cause high CPU utilization.
For troubleshooting high CPU utilization due to Netflow version 7, configure mls nde sender version 5, as the Netflow export is performed by the SP, which is the default for version 5 or version 9.

It turns out, combination of NetFlow version 9 and NDE sender version 7 also creates high CPU load in certain situations. Here is the setup:

Both routers are Cisco 7604. Other than different IP addresses, the only difference between R1 and R2 was this:

on R1:  mls nde sender
on R2:  mls nde sender version 5
Default sender version is 7. Both routers configured with ip flow-export version 9.
When ever R2's eBGP session was interrupted, R1's CPU utilization skyrocketed to 100% and stayed there for 10-15 minutes rendering router unusable. "process cpu threshold" reported that "IP Input" was responsible for CPU load, not "BGP Router" as I expected, since these CPU 
spikes only happened when eBGP session went down. After changing NDE sender version to 5 on R1, the problem went away.

Catching high CPU usage

Suddenly your router stops responding and forwarding traffic, you can telnet into it, response on the console is very slow. Few minutes later everything is back to normal and only "show process cpu history" shows that CPU was at 100% for some time, but what caused it remains a mystery. To catch a process(es) that might have contributed to the problem, add following command in global configuration mode:

process cpu threshold type process rising 70 interval 5 falling 30 interval 5

It will generate syslog message every time CPU usage exceeds 70% for 5 or more seconds and falls below 30%. For example:

May 10 23:50:23.146 EDT: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilization(Total/Intr): 74%/26%, Top 3 processes(Pid/Util): 192/46%, 7/1%, 2/0%

Process id 192 contributed 46%. Let's see:

Router#sho proc cpu sor | i ^_192
192   904947881922327784 47 0.00% 0.18% 0.19% 0 IP Input

 It was "IP Input" which is responsible for process-switching IP packets. Now we have something to work with and can start troubleshooting.

Monitoring logs with SEC

Splunk seems to become de-facto standard tool for log management. But free version lacks feature that lets you configure and send alerts whenever certain events occur. One need to pay for enterprise version which starts at $5000 in US and Canada.

So, I use Simple Event Correlator to notify me of interesting events in life of my router friends. Here, for example, sec template to send me email with syslog line in the body when somebody tries to go to configuration mode and execute certain commands:

type=Single
ptype=RegExp
pattern=.*cmd=(configure|clear|ip|no|interface|switchport|router|spanning-tree)
desc=$0
action=pipe '$0' /usr/bin/mail -s "router/switch config change is happening right now" noc@example.com

You need to put this template into SEC configuration file and tell it were to look for these messages:

sec -detach -conf=/etc/sec-tacacs.conf -input=/var/log/tac-plus/account

In this case it's TACACS+ log file, so you need to configure a router to report such activities:

aaa new-model
aaa authentication login default group tacacs+ none
aaa authentication enable default group tacacs+ none
aaa authorization exec default group tacacs+ none
aaa authorization commands 15 default group tacacs+ none
aaa accounting commands 15 default start-stop group tacacs+
tacacs-server host <server ip>
tacacs-server <key>

Here is another template to report all syslog messages coming from devices with loopback interface IP address in the range 10.9.20.0/24 or 10.9.25.0/24. Why loopback? See my previous post.

type=Single
ptype=RegExp
pattern=(.*)10\.9\.2[0|5]\.(.*)%[A-Z]*
desc=$0
action=pipe '$0' /usr/bin/mail -s " router syslog message" noc@example.com

Best practices. Sort of.

I tend to agree, that there is no "best practices", there are practices that fit best. Here is one of the things that I always configure on the router.

There are many advantages in configuring Loopback interface when you use dynamic routing, but I also find loopback helpful for syslog reporting and authentication and authorization queries. So, I always configure:

ip tacacs source-interface Loopback0
logging source-interface Loopback0

Next step is to either add loopback interfaces of your routers to DNS or /etc/hosts file on Tacacs and syslog servers.
The names are no good if you can not use them. I prefer syslog-ng for logging, so, in order to record names instead of IP addresses, you need to configure use_dns(yes) in "options" section of syslog-ng.conf. For TACACS+: run tac_plus with "-L" option.

Making same change on many routers

Suppose you need to make the same change on many routers, but do not have fancy software like Cisco Works to help you. No worries. Perl is the best friend of any network and system administrator. Here is the quick script that goes to a router and types command "logging source-interface loopback 0", saves configuration and exit. It can be used to run any command.
Place IP addresses of the routers, one per line, in file routers.txt. This file must be in the same directory as the script. Remember, you put your username, password and enable password in the script in clear text, so do not forget "chmod 700 "

#!/usr/bin/perl
use Net::Telnet::Cisco;
my $myfile="./routers.txt";
open (FH, $myfile) || die "Can not open $myfile\n";

while () {
chomp;
my $switchname=$_;
print "$switchname\n"; 

my $session = Net::Telnet::Cisco->new(Host => $switchname,Input_log => "$switchname.log"); 

# Replace username and password below with real username and password 

$session->login('username', 'password');

# Enable mode
if ($session->enable("enable password") ) { # insert your enable passowrd
    @output = $session->cmd('configure terminal');
    @output = $session->cmd('logging source-interface loopback 0');
    print @output;
    @output = $session->cmd('exit');
    @output = $session->cmd("copy run startup-config\n\n");
    print @output;
    } else {
         warn "Can't enable: " . $session->errmsg;
        }
$session->close;
}
 

Use at your own risk. 

Debian 5.0.4 on Dell 1950

Normally, installing Debian on Dell servers is piece of cake. This particular 1950 came with Broadcom NICs and PERC5 controller. Debian 5.0.4 does not include driver for Broadcom drivers due to some copyright restrictions. However, the driver is available as deb package. Download it and copy to FAT or FAT32 formatted USB drive. When prompted for NIC driver during the installation process, insert USB drive into USB port. As soon as server loads the driver and moves to the next screen in installation process, remove the drive. If you do not remove the USB drive before installation process gets to partitioning, your drive sequence will we out of whack. You'll have to boot from CD and edit /etc/fstab.
Since this server has hardware I wanted to use instead of configuring software RAID in Linux. The question is how to monitor RAID state from Debian. There is no deb package or source code, but LSI provides RPM. I downloaded "MegaCLI - Linux" from "Miscellaneous" section, unpacked it, installed "alien" on Debian (sudo apt-get install aliean) and then "sudo alien -i  MegaCli-1.01-0.i386.rpm". It install MegaCli under /opt/MegaRAID/MegaCli.  Moritz Mertinkat has great emergency cheat sheet for MegaCli usage.

Cacti and 95th percentile

I use Cacti to collect traffic data on my routers and I need to know what 95th percentile is. There are quite a ways to get 95th percentile line on Cacti graph. The problem with all those methods is that if time frame of the graph does not coincide with ISP billing period the 95th percentile value on the graph is useless. But all the necessary data is collected by Cacti into RRD file. All we have to do is to extract it.
First, I need to figure out where the RRD file is. In Cacti, go to Console -> Data Sources, select your edge router and click on IPS-facing interface. In "Data Source Path" field you'll see the name of the RRD file in Cacti's rra directory where data for this interface is stored

Second, we need to know what to extract from this file. I.e I need to know the names of RRD data sources:
rrdtool info border_router_1_traffic_in_14839.rrd
where border_router_1_traffic_in_14839.rrd is file name from previous step.

filename = "border_router_1_traffic_in_14839.rrd"
rrd_version = "0003"
step = 300
last_update = 1262806506
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = NaN
ds[traffic_in].last_ds = "437961211333"
ds[traffic_in].value = 4.9711447176e+05
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = NaN
ds[traffic_out].last_ds = "138465493978"
ds[traffic_out].value = 1.9428099668e+04
ds[traffic_out].unknown_sec = 0

truncated...

The data sources names are traffic_in and traffic_out and this is what we are going to extract. Before we proceed we need to remember, that RRD database size is fixed and determined at the time of creation. When limit is reached, oldest data is overwritten. To avoid losing any data, I am going to extract traffic numbers every hour for the last hour and put inbound and outbound data in separate files.
Incoming traffic:

rrdtool xport -s now-1h -e now DEF:xx=border_router_1_traffic_in_14839.rrd:traffic_in:AVERAGE CDEF:bb=xx,8,* XPORT:bb:"out bits" | grep \|grep -v Na | awk -F'' '{print $2}'| sed -e 's/<\/v><\/row>//'|sed -e 's/e+0/\t/' >> incoming.txt

Outgoing traffic:

rrdtool xport -s now-1h -e now \DEF:xx=border_router_1_traffic_in_14839.rrd:traffic_out:AVERAGE CDEF:bb=xx,8,* XPORT:bb:"out bits" | grep \|grep -v Na | awk -F'' '{print $2}'| sed -e 's/<\/v><\/row>//'|sed -e 's/e+0/\t/' >> outgoing.txt

Both commands should be in one line. Above I converted Bytes/sec into Bits/sec and removed XML formatting. You need these 2 lines into shell script and run it from cron every hour on 2 minutes after the hour so Cacti has time to finish collecting on top of the hour. You'll get 2 files - incoming.txt and outgoing.txt looking like this

6.9655133612 7
7.0568998690 7
6.9008144000 7
7.0245826541 7
7.2076520540 7
6.7448901179 7
6.7471832197 7
6.7365174531 7
6.9710477122 7
7.1586411237 7
7.0991637699 7
7.0189321194 7

This are measurements taken every 5 minutes by Cacti. "6.9655133612 7" means 6.9655133612 * 10^7 bits/sec or 69655133.612 bits/sec.

Now all you have to do on the 1st of the month right after midnight is to convert the data to get rid of second column, sort it and remove top 5%. For 30-day month:

cat incoming.txt |perl -e ' while(<>) {$input = $_; chomp($input);($traffic, $power)
=split(/\t/,$input); $traffic = $traffic*10**$power; print "$traffic\n";}'|egrep -v '^0$'|sort -n -r | head -433 |tail -1