The wait is over

Finally, in NX-OS 6.0(2) for Nexus 5000 platform Cisco implemented "default interface" command which lets you return interface to its factory default configuration. It is very-very-very useful feature in the lab environment when one has to do a lot of re-configuration and something does not work as expected simply because of left-over configuration from the previous test. 

This command has been available in IOS since 11.1 and in NX-OS for Nexus 7K since 5.1(1)

PACL and MAC address learning

I have unenviable task to drag legacy application to 21st century. I am talking about 80-the legacy and some of its functions do not even use IP protocols. One of the proposed solution included 2 servers with the same

IP and MAC addresses (I know, but splitting network in 2 separate VLANs was not an option) connected to different switches, but in the same VLAN. ClientA and ClientB should be able to talk to each other and server connected to the same switch as client. ServerA and ServerB should not even know about each other's existence, so they won't complain about duplicate IP address. The switches are Cisco 6500s. One of the obvious  solutions is to put Port Access Control List on either side of the inter-switch link. PACL successfully blocked the traffic between servers, but switches still learned MAC address of the blocked traffic source and placed it MAC address table. This would cause MAC address flapping on the switch every time clients send ARP query for server's MAC or when both servers need to send traffic to their clients. Why would switch need to keep MAC address of the discarded traffic? Oh, well. Another network mystery.

ip accounting-list

Cisco has interface level command "ip accounting" which records number of bytes and packets passed through the router. If you want to count traffic only for specific IP address, you need to use "ip accounting-list" command. There seems to be a tiny bug in context help in 12.4(15)T14:

R2(config)#ip accounting-list 1.1.1.1 ?
  A.B.C.D  IP address mask

Note that context help says you need to enter address mask. 

R2(config)#ip accounting-list 1.1.1.1 255.255.255.255

After checking

R2(config)#do sho run | i accounting

ip accounting-list 0.0.0.0 255.255.255.255

That's not what I entered. In reality, you are supposed to enter wildcard mask, which makes more sense. Specifying ACL number or name would have made even more sense. So, let's fix it:

R2(config)#no ip accounting-list 0.0.0.0 255.255.255.255

R2(config)#ip accounting-list 1.1.1.1 0.0.0.0

R2(config)#interface fa 0/0

R2(config-if)#ip accounting

and test it. Traffic from R3 to 192.168.12.1 and 1.1.1.1 must pass through R2's Fa0/0 interface: 

R3#ping 192.168.12.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.12.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 28/58/120 ms

R3#ping 1.1.1.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 16/47/88 ms

R2#sho ip account

   Source           Destination              Packets               Bytes

 192.168.23.3     1.1.1.1                          5                 500

As you can see from the output above, only pings to 1.1.1.1 got counted.

Cisco 6500/7600 ACL side effect

When you apply ACL to an interface on Cisco 6500 or 7600, it compiles it and puts into TCAM. The way Cisco 7600/6500 does it might have unintended consequences that can leave you open to DDoS attack. Let's consider following example:

We want to allow any server in 172.16.100.0/24 network to initiate any tcp connection and query any DNS server directly. Here is our ACL

ip access-list extended Test1
 permit tcp any any established
 permit udp any eq domain any
 deny  ip any any

 We apply it to internet-facing interface of Cisco7600 router: "ip access-group Test1 in". Now let's look at what actually happened in TCAM:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared


Entries from Bank 0


Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments
    permit       tcp any any established match-any
    permit       udp any eq domain any

Our router automatically added "permit  udp any any fragments", i.e. it allowed udp fragments. Now, let's see if it actually happens. First, take a look at the compiled ACL again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared


Entries from Bank 0


Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (41 matches)
    permit       tcp any any established match-any (220 matches)
    permit       udp any eq domain any

Not the counter - 41 matches. Next, on the "attacker" we'll generate fragmented UDP traffic targeting a server in 172.16.100.0/24 network:

hping2 -2 -d 1500 -c 1 -s 10000 -p 90 -m 500 -f 172.16.100.10

In the command above, we send 1 1500-byte UDP packet from port 10000 on local host to port 90 on 172.16.100.10 and we are telling the host that MTU is 500 bytes. On the target host we run tcpdump:

10:47:41.942010 IP (tos 0x0, ttl  63, id 130, offset 496, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942027 IP (tos 0x0, ttl  63, id 130, offset 1000, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942034 IP (tos 0x0, ttl  63, id 130, offset 1496, flags [none], length: 28) 172.16.0.101 > 172.16.100.10: udp

 Now, the first fragment, containing IP and UDP header were dropped by our ACL, since we do not allow UDP packets coming from port 10000, but 3 other fragments got through. Let's check the counter again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared


Entries from Bank 0


Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (44 matches)
    permit       tcp any any established match-any (224 matches)
    permit       udp any eq domain any

The attacker can flood your web or email server with UDP fragments causing it to slow down while it is busy discarding incomplete packets. We can not block fragments completely since legitimate DNS replies can be quite big and require fragmentation. The solution would be to allow outbound UDP traffic and, hence, incoming replies only to specific hosts that need it. Like your caching DNS server and put good firewall in front of it.

Cisco 7600: Netflow and high CPU utilization

Cisco documentation states, that:
 If NetFlow is configured for version 7, the flow is performed by the Routing Processor, which could cause high CPU utilization.
For troubleshooting high CPU utilization due to Netflow version 7, configure mls nde sender version 5, as the Netflow export is performed by the SP, which is the default for version 5 or version 9.

It turns out, combination of NetFlow version 9 and NDE sender version 7 also creates high CPU load in certain situations. Here is the setup:

Both routers are Cisco 7604. Other than different IP addresses, the only difference between R1 and R2 was this:

on R1:  mls nde sender
on R2:  mls nde sender version 5
Default sender version is 7. Both routers configured with ip flow-export version 9.
When ever R2's eBGP session was interrupted, R1's CPU utilization skyrocketed to 100% and stayed there for 10-15 minutes rendering router unusable. "process cpu threshold" reported that "IP Input" was responsible for CPU load, not "BGP Router" as I expected, since these CPU 
spikes only happened when eBGP session went down. After changing NDE sender version to 5 on R1, the problem went away.

Catching high CPU usage

Suddenly your router stops responding and forwarding traffic, you can telnet into it, response on the console is very slow. Few minutes later everything is back to normal and only "show process cpu history" shows that CPU was at 100% for some time, but what caused it remains a mystery. To catch a process(es) that might have contributed to the problem, add following command in global configuration mode:

process cpu threshold type process rising 70 interval 5 falling 30 interval 5

It will generate syslog message every time CPU usage exceeds 70% for 5 or more seconds and falls below 30%. For example:

May 10 23:50:23.146 EDT: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilization(Total/Intr): 74%/26%, Top 3 processes(Pid/Util): 192/46%, 7/1%, 2/0%

Process id 192 contributed 46%. Let's see:

Router#sho proc cpu sor | i ^_192
192   904947881922327784 47 0.00% 0.18% 0.19% 0 IP Input

 It was "IP Input" which is responsible for process-switching IP packets. Now we have something to work with and can start troubleshooting.

Making same change on many routers

Suppose you need to make the same change on many routers, but do not have fancy software like Cisco Works to help you. No worries. Perl is the best friend of any network and system administrator. Here is the quick script that goes to a router and types command "logging source-interface loopback 0", saves configuration and exit. It can be used to run any command.
Place IP addresses of the routers, one per line, in file routers.txt. This file must be in the same directory as the script. Remember, you put your username, password and enable password in the script in clear text, so do not forget "chmod 700 "

#!/usr/bin/perl
use Net::Telnet::Cisco;
my $myfile="./routers.txt";
open (FH, $myfile) || die "Can not open $myfile\n";

while () {
chomp;
my $switchname=$_;
print "$switchname\n"; 

my $session = Net::Telnet::Cisco->new(Host => $switchname,Input_log => "$switchname.log"); 

# Replace username and password below with real username and password 

$session->login('username', 'password');

# Enable mode
if ($session->enable("enable password") ) { # insert your enable passowrd
    @output = $session->cmd('configure terminal');
    @output = $session->cmd('logging source-interface loopback 0');
    print @output;
    @output = $session->cmd('exit');
    @output = $session->cmd("copy run startup-config\n\n");
    print @output;
    } else {
         warn "Can't enable: " . $session->errmsg;
        }
$session->close;
}
 

Use at your own risk. 

bpduguard vs. bpdufilter

Suppose you have a switch with servers connected to it. It's called "access layer switch" in Cisco lingo. There is no need to go through all spannig-tree states on these host-facing ports. So, you make port transition faster to forwarding mode with "switchport portfast" interface command. One of the benefits is that if your server boots fast, it does not need to wait until port finishes going through all STP port states and can start transmitting data immediately.
Since servers normally should ignore BPDUs coming from a switch, there is no need to send them to a server in the first place. To filter out outgoing BPDUs apply interface command "spanning-tree bpdufilter enable".
But when BPDU is received on the port with bpdufilter enabled, the port's portfast status is disabled and port will participate in spanning-tree.
 At all times network needs to be protected from unauthorized device that might decide to participate in your spanning-tree topology and cause spanning-tree loop or try to hijack STP root. Interface command "spanning-tree bpduguard enable" puts interface in err-disable mode whenever BPDU is received from connected device.
What will happen if you have both bpdufilter and bpduguard enabled on the interface?

spanning-tree portfast
spanning-tree bpdufilter enable
spanning-tree bpduguard enable

bpdufilter takes precedence and bpdugard does not work. Although bpduguard needs more administrative overhead - port needs to be enabled manually - it makes your network more secure.

Last Error: PCALC:: No addresses to connect

Ever since I passed "Implementing Cisco MPLS" (642-611) exam, I've been aching to try my hands at MPLS TE. So, here is the task:

Click on image to see bigger version.

Since R2-R4 link is T3, OSPF will pick R2-R3-R5 path from R1 to either R6 or R7 leaving R2-R4-R5 link underutilized. At the same time I know that traffic from R1 to 10.0.0.0/24 is going to low enough to fit comfortably into T3 link. I wanted to create explicit path tunnel R1-R2-R4-R5-R7 with backup dynamic tunnel. I am going to skip configuration part - there are more then enough examples on the 'Net.
Let's see if tunnel is up

R1#sho mpls traffic-eng tunnels

Name: 6503sw2_t0 (Tunnel0) Destination: 192.168.1.4
Status:
Admin: up Oper: down Path: not valid Signalling: Down path option 1, type explicit mypath

Config Parameters:
Bandwidth: 0 kbps (Global) Priority: 7 7 Affinity: 0x0/0xFFFF
Metric Type: TE (default)
AutoRoute: enabled LockDown: disabled Loadshare: 0 bw-based
auto-bw: disabled

History:
Tunnel:
Time since created: 20 hours, 1 minutes
Time since path change: 3 minutes, 24 seconds
Number of LSP IDs (Tun_Instances) used: 334
Prior LSP:
ID: path option 1 [309]
Removal Trigger: path option removed
Last Error: PCALC:: No addresses to connect 172.16.0.14 to 172.16.0.10

Not good. Searching for this type of error message gives one sage advise:

For cases involving an explicit path option, you can try to narrow down the problem by first checking every hop in the explicit hop list. You can also try to back down the tunnel: Move the tail one hop back each time to see if the tunnel comes up. If the tunnel comes up when you move the tail to a previous hop, you can conclude that there is a problem between that hop and the next hop, and you can start scrutinizing that hop carefully.

And so I did. Let's see how R2 and R5 see R4:

R2# show ip rsvp neighbor
Neighbor Encapsulation Time since msg rcvd/sent
172.16.0.18 Raw IP 00:02:12 00:02:10
172.16.0.21 Raw IP 00:02:10 00:02:23




R5#show ip rsvp neighbor
Neighbor Encapsulation Time since msg rcvd/sent
172.16.0.1 Raw IP 00:03:59 00:04:18
172.16.0.6 Raw IP 00:04:16 00:03:59

Hmm, they do not see R4 as RSVP neighbor at all. What's going on on R4?

R4# sho ip rsvp interface detail gi 0/0
Gi0/0:
Interface State: Up
Bandwidth:
Curr allocated: 0 bits/sec
Max. allowed (total): 30M bits/sec
Max. allowed (per flow): 30M bits/sec
Max. allowed for LSP tunnels using sub-pools: 0 bits/sec
Set aside by policy (total): 0 bits/sec
Admission Control:
Header Compression methods supported:
rtp (36 bytes-saved), udp (20 bytes-saved)
Traffic Control:
RSVP Data Packet Classification is ON
Signalling:
DSCP value used in RSVP msgs: 0x3F
Number of refresh intervals to enforce blockade state: 4
Number of missed refresh messages: 4
Refresh interval: 30
Authentication: disabled

How does it compare with R4-facing interface on R5?

R5#sho ip rsvp interface detail gi 0/2

Gi0/2:
RSVP: Enabled
Interface State: Up
Bandwidth:
Curr allocated: 0 bits/sec
Max. allowed (total): 40M bits/sec
Max. allowed (per flow): 40M bits/sec
Max. allowed for LSP tunnels using sub-pools: 0 bits/sec
Set aside by policy (total): 0 bits/sec
Admission Control:
Header Compression methods supported:
rtp (36 bytes-saved), udp (20 bytes-saved)
Traffic Control:
RSVP Data Packet Classification is ON via CEF callbacks
Signalling:
DSCP value used in RSVP msgs: 0x3F
Number of refresh intervals to enforce blockade state: 4
Authentication: disabled
Key chain:
Type: md5
Window size: 1
Challenge: disabled
Hello Extension:
State: Disabled

"RSVP: Enabled" string is missing in R4 command output. After some research I found that I need "T" type IOS to run RSVP-TE on R4 which is Cisco 3825.
Let's see what we have after the upgrade:

R4#sho ip rsvp interface detail gi 0/0

Gi0/0:
RSVP: Enabled
Interface State: Up
Bandwidth:
Curr allocated: 5M bits/sec
Max. allowed (total): 30M bits/sec
Max. allowed (per flow): 30M bits/sec
Max. allowed for LSP tunnels using sub-pools: 0 bits/sec
Set aside by policy (total): 0 bits/sec
Admission Control:
Header Compression methods supported:
rtp (36 bytes-saved), udp (20 bytes-saved)
Traffic Control:
RSVP Data Packet Classification is ON
Signalling:
DSCP value used in RSVP msgs: 0x3F
Number of refresh intervals to enforce blockade state: 4
Authentication: disabled
Key chain:
Type: md5
Window size: 1
Challenge: disabled
Hello Extension:
State: Disabled

and tunnel:

R1#sho mpls traffic-eng tunnels bri
Signalling Summary:
LSP Tunnels Process: running
Passive LSP Listener: running
RSVP Process: running
Forwarding: enabled
Periodic reoptimization: every 3600 seconds, next in 3042 seconds
Periodic FRR Promotion: Not Running
Periodic auto-bw collection: every 300 seconds, next in 42 seconds
TUNNEL NAME DESTINATION UP IF DOWN IF STATE/PROT
6503sw2_t0 192.168.1.4 - Gi2/1 up/up

Much better. Confirm routing

R1#sho ip route 10.0.0.0 255.255.255.0
Routing entry for 10.0.0.0/24
Known via "ospf 100", distance 110, metric 14, type intra area
Last update from 192.168.1.4 on Tunnel0, 19:41:13 ago
Routing Descriptor Blocks:
* 192.168.1.4, from 192.168.1.4, 19:41:13 ago, via Tunnel0
Route metric is 14, traffic share count is 1

Now, since MPLS TE tunnel is unidirectional, I need to create R7-R5-R4-R2-R1 tunnel as well.

SUP720-3C TCAM exeption

If you are running BGP on Catalyst 6500 or 7600 with SUP 72-3C and receiving full routing table from a peer, sooner rather than later you'll get this message
%CFIB-SP-7-CFIB_EXCEPTION: FIB TCAM exception, Some entries will be software switched
Meaning that your router's performance will be degraded, since some of the traffic processed by CPU as opposed to hardware. The only way to get out of this state is to reboot the router. Here is how to check whether your router is in exception state:

cisco-7600#show mls cef exception status
Current IPv4 FIB exception state = TRUE
Current IPv6 FIB exception state = FALSE
Current MPLS FIB exception state = FALSE

If you are not running IPv6 or MPLS you can postpone inevitable upgrade to 720-3CXL by allocating entire TCAM memory to IPv4 routes
cisco-7600#mls cef maximum-routes ip 239
reboot

To see TCAM utilization
cisco-7600##show platform hardware capacity | begin L3 Forwarding Resources
L3 Forwarding Resources
FIB TCAM usage: Total Used %Used
72 bits (IPv4, MPLS, EoM) 196608 184013 94%
144 bits (IP mcast, IPv6) 32768 3 1%


.....skipping the rest