Val(config)#

ip accounting-list

2012-02-04T14:25:00.002-05:00

Cisco has interface level command "ip accounting" which records number of bytes and packets passed through the router. If you want to count traffic only for specific IP address, you need to use "ip accounting-list" command. There seems to be a tiny bug in context help in 12.4(15)T14:

R2(config)#ip accounting-list 1.1.1.1 ?
A.B.C.D IP address mask

Note that context help says you need to enter address mask.

R2(config)#ip accounting-list 1.1.1.1 255.255.255.255

After checking

R2(config)#do sho run | i accounting

ip accounting-list 0.0.0.0 255.255.255.255

That's not what I entered. In reality, you are supposed to enter wildcard mask, which makes more sense. Specifying ACL number or name would have made even more sense. So, let's fix it:

R2(config)#no ip accounting-list 0.0.0.0 255.255.255.255

R2(config)#ip accounting-list 1.1.1.1 0.0.0.0

R2(config)#interface fa 0/0

R2(config-if)#ip accounting

and test it. Traffic from R3 to 192.168.12.1 and 1.1.1.1 must pass through R2's Fa0/0 interface:

R3#ping 192.168.12.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.12.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 28/58/120 ms

R3#ping 1.1.1.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 16/47/88 ms

R2#sho ip account

Source Destination Packets Bytes

192.168.23.3 1.1.1.1 5 500

As you can see from the output above, only pings to 1.1.1.1 got counted.

IGP: administrative distance per prefix

2012-01-14T15:18:00.000-05:00

Routing protocol administrative distance defines route from which protocol will be placed in RIB - lower is better. However, AD can be changed via "distance" command on Cisco routers. The full syntax is:

distance ip-address wildcard-mask [ip-standard-acl | ip-extended-acl | access-list-name]

access-list option assumes that AD can be changed per IP subnet. Let's see how it works in RIPv2, EIGRP and OSPF.

I have very simple topology here

R1-------------------R2

Router R1 advertises 2 networks into RIP which we can see on R2:

R2#show ip route rip
R 192.168.200.0/24 [120/1] via 192.168.12.1, 00:00:11, FastEthernet0/0
R 192.168.100.0/24 [120/1] via 192.168.12.1, 00:00:11, FastEthernet0/0

Both routes have administrative distance 120 as it is default for RIP. Let's change AD for 192.168.100.0/24R2#conf t
R2(config)#access-list 10 permit 192.168.100.0 0.0.0.255
R2(config)#router rip
R2(config-router)#distance 150 192.168.12.1 0.0.0.0 10
R2(config-router)#end

Now, we'll give it some time since RIP is notoriously slow to converge protocol and check

R2#show ip route rip
R 192.168.200.0/24 [120/1] via 192.168.12.1, 00:00:02, FastEthernet0/0
R 192.168.100.0/24 [150/1] via 192.168.12.1, 00:00:02, FastEthernet0/0

As you can see, 192.168.100.0/24 now has administrative distance 150

2. EIGRP
Now I configure EIGRP between my two routers

R2#show ip route eigrp
D 192.168.200.0/24 [90/156160] via 192.168.12.1, 00:00:13, FastEthernet0/0
D 192.168.100.0/24 [90/156160] via 192.168.12.1, 00:00:13, FastEthernet0/0

And repeat:

R2(config)#router eigrp 1
R2(config-router)#distance 150 192.168.12.1 0.0.0.0 10
R2(config-router)#end

Unlike RIP, EIGRP converges almost instantly:

R2#show ip route eigrp

D 192.168.200.0/24 [90/156160] via 192.168.12.1, 00:00:02, FastEthernet0/0

D 192.168.100.0/24 [150/156160] via 192.168.12.1, 00:00:02, FastEthernet0/0

3. OSPF

R2#show ip route ospf

O 192.168.200.0/24 [110/2] via 192.168.12.1, 00:00:17, FastEthernet0/0

O 192.168.100.0/24 [110/2] via 192.168.12.1, 00:00:17, FastEthernet0/0

In case of OSPF IP address in distance command should be router-id of OSPF neighbor from which route is learned.

R2#conf t

R2(config)#router ospf 1

R2(config-router)#distance 150 1.1.1.1 0.0.0.0 10

R2(config-router)#end

R2#sho ip route ospf

O 192.168.200.0/24 [110/2] via 192.168.12.1, 00:02:55, FastEthernet0/0

O 192.168.100.0/24 [150/2] via 192.168.12.1, 00:02:55, FastEthernet0/0

Once again, AD has changed to 150 for 192.168.100.0/24

Let's consider more complex OSPF scenario:

R2 and R3 advertise 192.168.100.0/24 and 192.168.200.0/24 to R4.

R4#sho ip route ospf | begin 192.168.200.0

O 192.168.200.0/24 [110/2] via 192.168.34.3, 00:00:10, FastEthernet0/0

[110/2] via 192.168.24.2, 00:00:10, FastEthernet0/1

O 192.168.100.0/24 [110/2] via 192.168.34.3, 00:00:10, FastEthernet0/0

[110/2] via 192.168.24.2, 00:00:10, FastEthernet0/1

Both paths are equal and R4 will use both of them by default. Now, for some hard to explain reason we want to use R3 as our primary path to 192.168.100.0/24. It should be easy, all we need to do is to apply our access-list 10 from above to routes we receive from R2 (OSPF router-id 2.2.2.2):

R4#conf t

R4(config)#router ospf 1

R4(config-router)#distance 150 2.2.2.2 0.0.0.0 10

R4(config-router)#end

We can not use "ip ospf cost" command since it affects all routes coming via that interface. Routing check:

R4#sho ip route ospf | begin 192.168.100.0

O 192.168.100.0/24 [150/2] via 192.168.34.3, 00:15:07, FastEthernet0/0

[150/2] via 192.168.24.2, 00:15:07, FastEthernet0/1

Hmm, 192.168.100.0/24 still has AD of 150 for both next hops. What happened? After doing a lot of digging I found this post from Mike Timm. Cisco bug CSCeh44993 prevents modifying administrative distance per route per neighbor in OSPF. Alas, Cisco decided not to fix it and make it a feature.

IPexpert puzzle

2012-01-04T22:07:00.000-05:00

IPexpert posted interesting puzzle today. Here is my solution:

R2:
router ospf 1
router-id 192.168.0.2
log-adjacency-changes
network 192.168.0.0 0.0.255.255 area 0
default-information originate

R5:
router bgp 5
no synchronization
bgp router-id 192.168.0.5
bgp log-neighbor-changes
redistribute ospf 1
neighbor 172.16.45.4 remote-as 4
neighbor 172.16.45.4 default-originate route-map DEFAULT
no auto-summary
!
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0
!
route-map DEFAULT permit 10
match ip address prefix-list DEFAULT

Now let's head to R4 and check BGP routes:

R4#sho ip route bgp
B 192.168.25.0/24 [20/0] via 172.16.45.5, 00:55:28
192.168.0.0/32 is subnetted, 2 subnets
B 192.168.0.2 [20/2] via 172.16.45.5, 00:55:28
B 192.168.0.5 [20/0] via 172.16.45.5, 00:55:28
B* 0.0.0.0/0 [20/0] via 172.16.45.5, 00:37:36

I am still trying to find out why OSPF would not redistribute static default route. BGP will not redistribute default route even it's in source protocol routing table. It must be loop prevention mechanism, but I can not come up with a scenario when redistributing default route as oppose to originating it can cause routing loop. Especially in OSPF, where "default-information originate" creates Type5 LSA - same type as "redistribute" command would have created:

R2#sho ip ospf database | begin Type-5
Type-5 AS External Link States
Link ID ADV Router Age Seq# Checksum Tag
0.0.0.0 192.168.0.2 391 0x80000003 0x001F26 1

BGP network route-map command

2011-12-22T16:22:00.001-05:00

Let's consider following simple network:

Here is related configuration
R1
interface Loopback0
ip address 1.1.1.1 255.255.255.255

interface FastEthernet0/0
ip address 192.168.12.1 255.255.255.0
!

router bgp 1

no synchronization

bgp log-neighbor-changes

network 1.1.1.1 mask 255.255.255.255

neighbor 192.168.12.2 remote-as 2

neighbor 192.168.12.2 send-community both

no auto-summary

ip bgp-community new-format

interface FastEthernet0/0

ip address 192.168.12.2 255.255.255.0

end

interface FastEthernet0/1

ip address 192.168.23.2 255.255.255.0

end

router bgp 2

no synchronization

bgp log-neighbor-changes

neighbor 192.168.12.1 remote-as 1

neighbor 192.168.23.3 remote-as 3

no auto-summary

ip bgp-community new-format

interface FastEthernet0/0

ip address 192.168.23.3 255.255.255.0

router bgp 3

no synchronization

bgp log-neighbor-changes

neighbor 192.168.23.2 remote-as 2

no auto-summary

The prefix 1.1.1.1/32 is advertised to R2 and R3:

R2#sho ip bgp 1.1.1.1

BGP routing table entry for 1.1.1.1/32, version 14

Paths: (1 available, best #1, table Default-IP-Routing-Table)

Flag: 0x820

Advertised to update-groups:

192.168.12.1 from 192.168.12.1 (1.1.1.1)

Origin IGP, metric 0, localpref 100, valid, external, best

Now, let's add community attribute to 1.1.1.1/32 prefix by using network route-map command. First, we create route-map

R1#conf t
R1(config)#route-map LOOP1
R1(config-route-map)#set community 1:100
R1(config-route-map)#exit

Second, apply this route-map to the prefix
R1(config)#router bgp 1
R1(config-router)#network 1.1.1.1 mask 255.255.255.255 route-map LOOP1

We can see on R2 that community 1:100 was indeed added to 1.1.1.1/32

R2#show ip bgp 1.1.1.1

BGP routing table entry for 1.1.1.1/32, version 15

Paths: (1 available, best #1, table Default-IP-Routing-Table)

Flag: 0x8A0

Advertised to update-groups:

192.168.12.1 from 192.168.12.1 (1.1.1.1)

Origin IGP, metric 0, localpref 100, valid, external, best

Community: 1:100

So far, so good. Now I am going to try well-known community no-export, so R2 does not advertise 1.1.1.1/32 to R3

R1# conf t

R1(config)#route-map LOOP1

R1(config-route-map)#no set community 1:100

R1(config-route-map)#set community no-export

Debug output on R1 indicates that 1.1.1.1/32 is not advertised to R2

*Mar 1 02:37:19.771: BGP(0): sourced route for 1.1.1.1/32 path #0 changed (weight 32768)

*Mar 1 02:37:20.231: BGP(0): nettable_walker 1.1.1.1/32 route sourced locally

*Mar 1 02:37:20.231: BGP(0): 192.168.12.2 send unreachable 1.1.1.1/32

*Mar 1 02:37:20.231: BGP(0): 192.168.12.2 send UPDATE 1.1.1.1/32 -- unreachable

*Mar 1 02:37:20.319: BGP(0): 192.168.12.2 rcv UPDATE about 1.1.1.1/32 -- withdrawn

Sure enough:

R2#show ip bgp 1.1.1.1/32

% Network not in table

That's not what I intended. What happened? The thing is route-map key in BGP network command changes attributes BEFORE affected prefix is inserted into BGP routing table. In this case no-export community is added first, then R1 puts 1.1.1.1/32 in BGP routing table on R1, where R1 sees that this prefix can not be advertised outside its AS. To verify, let's change R1-R2 from eBGP into iBGP:

R1#sho run | sec r b

router bgp 1

no synchronization

bgp log-neighbor-changes

network 1.1.1.1 mask 255.255.255.255 route-map LOOP1

neighbor 192.168.12.2 remote-as 1

neighbor 192.168.12.2 send-community both

no auto-summary

R2#sho run | sec r b

router bgp 1

no synchronization

bgp log-neighbor-changes

neighbor 192.168.12.1 remote-as 1

neighbor 192.168.23.3 remote-as 3

no auto-summary

R3#sho run | sec r b

router bgp 3

no synchronization

bgp log-neighbor-changes

neighbor 192.168.23.2 remote-as 1

no auto-summary

1.1.1.1/32 should appear in R2's BGP routing table:

R2#sho ip bgp 1.1.1.1

BGP routing table entry for 1.1.1.1/32, version 3

Paths: (1 available, best #1, table Default-IP-Routing-Table, not advertised to EBGP peer)

Not advertised to any peer

Local

192.168.12.1 from 192.168.12.1 (1.1.1.1)

Origin IGP, metric 0, localpref 100, valid, internal, best

Community: no-export

Since R1 and R2 are in the same ASN, no-export community does not affect route advertisement. However,

R3#show ip bgp 1.1.1.1/32

% Network not in table

Important thing to remember is that in this case route-map changes attributes BEFORE prefix added into BGP routing table and advertised to other BGP peers.

Hidden OSPF command

2011-11-19T19:34:00.000-05:00

I was testing some OSPF features and wanted to check routes learned via OSPF. Before going into LSA database, I checked RIB, but the output format was not what I was expecting:

OSPF Router with ID (1.1.1.1) (Process ID 1)

Area BACKBONE(0)

Intra-area Route List

* 1.1.1.1/32, Intra, cost 1, area 0, Connected

via 1.1.1.1, Loopback0

* 11.11.11.11/32, Intra, cost 1, area 0, Connected

via 11.11.11.11, Loopback11

Intra-area Router Path List

i 4.4.4.4 [564] via 192.168.12.2, Serial1/0.12, ABR, Area 0, SPF 11

Inter-area Route List

*> 4.4.4.4/32, Inter, cost 565, area 0

via 192.168.12.2, Serial1/0.12

Area 1

Intra-area Route List

*> 192.168.24.0/24, Intra, cost 564, area 1

via 192.168.12.2, Serial1/0.12

So, I checked what I typed in the command line. It was "R1#sho ip ospf route" instead of "show ip route ospf". However, "route" does not show up as an option for "show ip ospf" command:

R1#sho ip ospf ?
<1-65535> Process ID number
border-routers Border and Boundary Router Information
database Database summary
flood-list Link state flood list
interface Interface information
max-metric Max-metric origination information
mpls MPLS related information
neighbor Neighbor list
request-list Link state request list
retransmission-list Link state retransmission list
rib Routing Information Base (RIB)
sham-links Sham link information
statistics Various OSPF Statistics
summary-address Summary-address redistribution Information
timers OSPF timers information
traffic Traffic related statistics
virtual-links Virtual link information
| Output modifiers
R1#sho ip ospf route ?
% Unrecognized command

I am running 12.4(15)T14 on the router. This is very useful command and output is easier to interpret than "show ip ospf database". Why does Cisco keep it hidden?

CCIE R&S written

2010-12-14T10:10:00.001-05:00

After getting CCIP and CCNP certifications I did not have a reason to postpone taking CCIE written exam any longer. So, I went for it last Wednesday and passed it. CCIE R&S written covers broad range of topics. All these topics are covered in depth by exams required to obtain CCIP and CCNP. If you took and passed ROUTE, SWITCH, TSHOOT, QOS, BGP and MPLS exams, CCIE R&S written looks fairly easy and does not require any additional preparations.

Cisco TSHOOT exam great success

2010-11-19T12:48:00.000-05:00

My second shot at TSHOOT exam was much better. I went to another testing center and did not experience a single glitch. Out of 35-40 questions I got only one wrong.
I used Boson's practice exam during the preparation to one of the previous Cisco exams before and was really disappointed. The practice exam was not hard enough and there were few errors, but I was very pleased with their TSHOOT simulator. It is probably even more difficult than real Cisco exam, but no simulator can replace on-the-job experience.

Cisco TSHOOT exam epic fail

2010-11-10T16:03:00.000-05:00

I have one last exam left to add CCNP trophy to my CCNA and CCIP. The exam is new Cisco TSHOOT. I've read a lot of good reviews about the exam and demo looks great.

The exam has 2 parts: multiple choice questions and tickets. After I finished multiple choice part, exam software crashed throwing some Flash error. I was given another computer and it crashed again in the same place. Kind folks at the testing center spent about an hour on the phone with tech support only to see test exam software crashing again and again with the same Flash error.
It looks like Steve Jobs was onto something when he banned Flash from iPad.
So, I have to re-schedule the exam and hope that next time it works.

OSPFv3-EIGRPv6 redistribution. Strange "include-connected" behavior.

2010-11-08T11:47:00.000-05:00

I've been practicing IPv6 and notice strange thing with "include-connected" option when redistributing between EIGRPv6 and OSPFv3. First, short preamble. In IPv4 redistribution, when source protocol runs on connected network, this network is automatically included in redistribution in destination protocol. Apparently, this is not the case in IPv6. You have to include connected network explicitly. Here is my network:

All routers run IOS 12.4(22)T
R2 and R3 are doing redistribution:

R2#sho run | sec ipv6 router
ipv6 router ospf 11
log-adjacency-changes
redistribute rip RIPng include-connected
ipv6 router rip RIPng
redistribute ospf 11 metric 5 include-connected

R3#sho run | sec ipv6 router
ipv6 router eigrp 1
no shutdown
default-metric 1000000 1 255 1 1500
redistribute ospf 11 include-connected
ipv6 router ospf 11
log-adjacency-changes
default-metric 30
redistribute eigrp 1 include-connected

Notice "include-connected" in redistribute command. Now, let's see what routing table looks like on R1 and R4:

R4>sho ipv6 route
EX 2000:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2000:2::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2000:4::/64 [0/0]
via Loopback10, directly connected
L 2000:4::1/128 [0/0]
via Loopback10, receive
EX 2001:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2001:3::/64 [0/0]
via GigabitEthernet0/3, directly connected
L 2001:3::2/128 [0/0]
via GigabitEthernet0/3, receive
L FF00::/8 [0/0]
via Null0, receive

R1>1>sho ipv6 route
C 2000:1::/64 [0/0]
via Loopback10, directly connected
L 2000:1::1/128 [0/0]
via Loopback10, receive
R 2000:2::/64 [120/2]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:3::1/128 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:4::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
C 2001:1::/64 [0/0]
via GigabitEthernet0/2, directly connected
L 2001:1::1/128 [0/0]
via GigabitEthernet0/2, receive
R 2001:2::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
L FF00::/8 [0/0]
via Null0, receive

R1 does not have 2001:3::/64 network and R4 is missing 2001:2::/64.
2001:3::/64 is EIGRPv6 network and is supposed to be redistributed into OSPFv3 and then into RIPng.
2001:2::/64 is OSPFv3 network and should be redistributed into EIGRPv6.
So, even though "include-connected" is added to "redistribute ospf" and "redistribute eigrp" it does not seem to work. At the same time, on R2, RIPng redistribute connected OSPF networks.
Let's add explicit "redistribute connected" statements on R3.

R3(config)#ipv6 router eigrp 1
R3(config-rtr)#redistribute connected
R3(config-rtr)#ipv6 router ospf 11
R3(config-rtr)#redistribute connected
R3(config-rtr)#end

Let's check R1 and R4 again:

R1>sho ipv6 route
C 2000:1::/64 [0/0]
via Loopback10, directly connected
L 2000:1::1/128 [0/0]
via Loopback10, receive
R 2000:2::/64 [120/2]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:3::1/128 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2000:4::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
C 2001:1::/64 [0/0]
via GigabitEthernet0/2, directly connected
L 2001:1::1/128 [0/0]
via GigabitEthernet0/2, receive
R 2001:2::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
R 2001:3::/64 [120/6]
via FE80::21E:7AFF:FE75:521A, GigabitEthernet0/2
L FF00::/8 [0/0]
via Null0, receive

R4>show ipv6 route
EX 2000:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2000:2::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2000:3::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2000:4::/64 [0/0]
via Loopback10, directly connected
L 2000:4::1/128 [0/0]
via Loopback10, receive
EX 2001:1::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
EX 2001:2::/64 [170/3072]
via FE80::21E:7AFF:FE94:117, GigabitEthernet0/3
C 2001:3::/64 [0/0]
via GigabitEthernet0/3, directly connected
L 2001:3::2/128 [0/0]
via GigabitEthernet0/3, receive
L FF00::/8 [0/0]
via Null0, receive

It works now. To make sure:

R4>traceroute 2000:1::1

Type escape sequence to abort.
Tracing the route to 2000:1::1

1 2001:3::1 0 msec 0 msec 0 msec
2 2001:2::1 0 msec 4 msec 0 msec
3 2001:1::1 0 msec 0 msec 0 msec

So far I was not able to find if this is intended behavior.

Cisco 642-813 SWITCH exam

2010-08-11T15:35:00.000-04:00

I recently took and passed Cisco 642-813 SWITCH exam. All those things you might have read about crashing simulators, poorly worded or bad grammar questions are true. In my case sim crashed 3 times when I entered same command, so I simply skipped the command just to continue with the exam. It made task incomplete and cost me few precious points. And do not even try Cisco's practice tests - there are too many incorrect answers to take these tests seriously.
All this is not to say that test is impossible to pass, but if you are not in a rush, I'd recommend to wait a few months before taking it. Hopefully, Cisco will clean up its act.

Monitoring trunk status via SNMP

2010-06-21T16:03:00.000-04:00

If you have not guessed yet, SNMP and monitoring are my favorites.

So, you have configured many trunks on your switch and now need to make sure all of them are actually in trunking mode. Here is 2 SNMP OID that can help you:

vlanTrunkPortDynamicState (1.3.6.1.4.1.9.9.46.1.6.1.1.13) - reports administrative state. From Cisco SNMP object navigator:
1 : on
2 : off
3 : desirable
4 : auto
5 : onNoNegotiate

vlanTrunkPortDynamicStatus (1.3.6.1.4.1.9.9.46.1.6.1.1.14) - reports operational state.
1 : trunking
2 : notTrunking

To get data for specific interface you need to add ifIndex to the end of the OID. For example, for interface ifIndex=10147
snmpwalk -v2c -Ov -Oq -c public myswitch 1.3.6.1.4.1.9.9.46.1.6.1.1.13.10147
To get ifIndex, you can either run "show snmp mib ifmib ifIndex" command in exec mode or query ifName OID with snmpwalk. Here is the quick script:

for int in ifIndex1 ifIndex2 ifIndexN
        do
        trunkoperstatus=`snmpwalk -v2c -Ov -Oq -c public myswitch \ 1.3.6.1.4.1.9.9.46.1.6.1.1.14.$int`
                if [ $trunkoperstatus -eq 2 ]
                then
                        trunkadminstatus=`snmpwalk -v2c -Ov -Oq -c public myswitch \ 1.3.6.1.4.1.9.9.46.1.6.1.1.13.$int`
                        if [ $trunkadminstatus -eq 1 ]
                        then
                                echo myswitch $int NotTrunking
                        fi
                fi
        done

Cisco 6500/7600 ACL side effect

2010-05-28T13:16:00.001-04:00

When you apply ACL to an interface on Cisco 6500 or 7600, it compiles it and puts into TCAM. The way Cisco 7600/6500 does it might have unintended consequences that can leave you open to DDoS attack. Let's consider following example:

We want to allow any server in 172.16.100.0/24 network to initiate any tcp connection and query any DNS server directly. Here is our ACL

ip access-list extended Test1
permit tcp any any established
permit udp any eq domain any
deny ip any any

We apply it to internet-facing interface of Cisco7600 router: "ip access-group Test1 in". Now let's look at what actually happened in TCAM:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments
    permit       tcp any any established match-any
    permit       udp any eq domain any

Our router automatically added "permit udp any any fragments", i.e. it allowed udp fragments. Now, let's see if it actually happens. First, take a look at the compiled ACL again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (41 matches)
    permit       tcp any any established match-any (220 matches)
    permit       udp any eq domain any

Not the counter - 41 matches. Next, on the "attacker" we'll generate fragmented UDP traffic targeting a server in 172.16.100.0/24 network:

hping2 -2 -d 1500 -c 1 -s 10000 -p 90 -m 500 -f 172.16.100.10

In the command above, we send 1 1500-byte UDP packet from port 10000 on local host to port 90 on 172.16.100.10 and we are telling the host that MTU is 500 bytes. On the target host we run tcpdump:

10:47:41.942010 IP (tos 0x0, ttl 63, id 130, offset 496, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942027 IP (tos 0x0, ttl 63, id 130, offset 1000, flags [+], length: 520) 172.16.0.101 > 172.16.100.10: udp

10:47:41.942034 IP (tos 0x0, ttl 63, id 130, offset 1496, flags [none], length: 28) 172.16.0.101 > 172.16.100.10: udp

Now, the first fragment, containing IP and UDP header were dropped by our ACL, since we do not allow UDP packets coming from port 10000, but 3 other fragments got through. Let's check the counter again:

Cisco7600#show tcam int gi 1/1 acl in ip

* Global Defaults shared

Entries from Bank 0

Entries from Bank 1

    permit       tcp any any fragments
    permit       udp any any fragments (44 matches)
    permit       tcp any any established match-any (224 matches)
    permit       udp any eq domain any

The attacker can flood your web or email server with UDP fragments causing it to slow down while it is busy discarding incomplete packets. We can not block fragments completely since legitimate DNS replies can be quite big and require fragmentation. The solution would be to allow outbound UDP traffic and, hence, incoming replies only to specific hosts that need it. Like your caching DNS server and put good firewall in front of it.

Cisco 7600: Netflow and high CPU utilization

2010-05-24T14:59:00.000-04:00

Cisco documentation states, that:
If NetFlow is configured for version 7, the flow is performed by the Routing Processor, which could cause high CPU utilization.
For troubleshooting high CPU utilization due to Netflow version 7, configure mls nde sender version 5, as the Netflow export is performed by the SP, which is the default for version 5 or version 9.

It turns out, combination of NetFlow version 9 and NDE sender version 7 also creates high CPU load in certain situations. Here is the setup:

Both routers are Cisco 7604. Other than different IP addresses, the only difference between R1 and R2 was this:

on R1: mls nde sender
on R2: mls nde sender version 5
Default sender version is 7. Both routers configured with ip flow-export version 9.
When ever R2's eBGP session was interrupted, R1's CPU utilization skyrocketed to 100% and stayed there for 10-15 minutes rendering router unusable. "process cpu threshold" reported that "IP Input" was responsible for CPU load, not "BGP Router" as I expected, since these CPU
spikes only happened when eBGP session went down. After changing NDE sender version to 5 on R1, the problem went away.

Catching high CPU usage

2010-05-11T17:43:00.000-04:00

Suddenly your router stops responding and forwarding traffic, you can telnet into it, response on the console is very slow. Few minutes later everything is back to normal and only "show process cpu history" shows that CPU was at 100% for some time, but what caused it remains a mystery. To catch a process(es) that might have contributed to the problem, add following command in global configuration mode:

process cpu threshold type process rising 70 interval 5 falling 30 interval 5

It will generate syslog message every time CPU usage exceeds 70% for 5 or more seconds and falls below 30%. For example:

May 10 23:50:23.146 EDT: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilization(Total/Intr): 74%/26%, Top 3 processes(Pid/Util): 192/46%, 7/1%, 2/0%

Process id 192 contributed 46%. Let's see:

Router#sho proc cpu sor | i ^_192
192 904947881922327784 47 0.00% 0.18% 0.19% 0 IP Input

It was "IP Input" which is responsible for process-switching IP packets. Now we have something to work with and can start troubleshooting.

Monitoring logs with SEC

2010-03-26T15:07:00.002-04:00

Splunk seems to become de-facto standard tool for log management. But free version lacks feature that lets you configure and send alerts whenever certain events occur. One need to pay for enterprise version which starts at $5000 in US and Canada.

So, I use Simple Event Correlator to notify me of interesting events in life of my router friends. Here, for example, sec template to send me email with syslog line in the body when somebody tries to go to configuration mode and execute certain commands:

type=Single
ptype=RegExp
pattern=.*cmd=(configure|clear|ip|no|interface|switchport|router|spanning-tree)
desc=$0
action=pipe '$0' /usr/bin/mail -s "router/switch config change is happening right now" noc@example.com

You need to put this template into SEC configuration file and tell it were to look for these messages:

sec -detach -conf=/etc/sec-tacacs.conf -input=/var/log/tac-plus/account

In this case it's TACACS+ log file, so you need to configure a router to report such activities:

aaa new-model
aaa authentication login default group tacacs+ none
aaa authentication enable default group tacacs+ none
aaa authorization exec default group tacacs+ none
aaa authorization commands 15 default group tacacs+ none
aaa accounting commands 15 default start-stop group tacacs+
tacacs-server host <server ip>
tacacs-server <key>

Here is another template to report all syslog messages coming from devices with loopback interface IP address in the range 10.9.20.0/24 or 10.9.25.0/24. Why loopback? See my previous post.

type=Single
ptype=RegExp
pattern=(.*)10\.9\.2[0|5]\.(.*)%[A-Z]*
desc=$0
action=pipe '$0' /usr/bin/mail -s " router syslog message" noc@example.com

Best practices. Sort of.

2010-03-25T17:52:00.001-04:00

I tend to agree, that there is no "best practices", there are practices that fit best. Here is one of the things that I always configure on the router.

There are many advantages in configuring Loopback interface when you use dynamic routing, but I also find loopback helpful for syslog reporting and authentication and authorization queries. So, I always configure:

ip tacacs source-interface Loopback0
logging source-interface Loopback0

Next step is to either add loopback interfaces of your routers to DNS or /etc/hosts file on Tacacs and syslog servers.
The names are no good if you can not use them. I prefer syslog-ng for logging, so, in order to record names instead of IP addresses, you need to configure use_dns(yes) in "options" section of syslog-ng.conf. For TACACS+: run tac_plus with "-L" option.

Making same change on many routers

2010-03-25T11:46:00.000-04:00

Suppose you need to make the same change on many routers, but do not have fancy software like Cisco Works to help you. No worries. Perl is the best friend of any network and system administrator. Here is the quick script that goes to a router and types command "logging source-interface loopback 0", saves configuration and exit. It can be used to run any command.
Place IP addresses of the routers, one per line, in file routers.txt. This file must be in the same directory as the script. Remember, you put your username, password and enable password in the script in clear text, so do not forget "chmod 700 "

#!/usr/bin/perl
use Net::Telnet::Cisco;
my $myfile="./routers.txt";
open (FH, $myfile) || die "Can not open $myfile\n";

while () {
chomp;
my $switchname=$_;
print "$switchname\n";

my $session = Net::Telnet::Cisco->new(Host => $switchname,Input_log => "$switchname.log");

# Replace username and password below with real username and password

$session->login('username', 'password');

# Enable mode
if ($session->enable("enable password") ) { # insert your enable passowrd
    @output = $session->cmd('configure terminal');
    @output = $session->cmd('logging source-interface loopback 0');
    print @output;
    @output = $session->cmd('exit');
    @output = $session->cmd("copy run startup-config\n\n");
    print @output;
    } else {
         warn "Can't enable: " . $session->errmsg;
        }
$session->close;
}

Use at your own risk.

Debian 5.0.4 on Dell 1950

2010-03-22T15:36:00.000-04:00

Normally, installing Debian on Dell servers is piece of cake. This particular 1950 came with Broadcom NICs and PERC5 controller. Debian 5.0.4 does not include driver for Broadcom drivers due to some copyright restrictions. However, the driver is available as deb package. Download it and copy to FAT or FAT32 formatted USB drive. When prompted for NIC driver during the installation process, insert USB drive into USB port. As soon as server loads the driver and moves to the next screen in installation process, remove the drive. If you do not remove the USB drive before installation process gets to partitioning, your drive sequence will we out of whack. You'll have to boot from CD and edit /etc/fstab.
Since this server has hardware I wanted to use instead of configuring software RAID in Linux. The question is how to monitor RAID state from Debian. There is no deb package or source code, but LSI provides RPM. I downloaded "MegaCLI - Linux" from "Miscellaneous" section, unpacked it, installed "alien" on Debian (sudo apt-get install aliean) and then "sudo alien -i MegaCli-1.01-0.i386.rpm". It install MegaCli under /opt/MegaRAID/MegaCli. Moritz Mertinkat has great emergency cheat sheet for MegaCli usage.

Cacti and 95th percentile

2010-01-06T15:41:00.006-05:00

I use Cacti to collect traffic data on my routers and I need to know what 95th percentile is. There are quite a ways to get 95th percentile line on Cacti graph. The problem with all those methods is that if time frame of the graph does not coincide with ISP billing period the 95th percentile value on the graph is useless. But all the necessary data is collected by Cacti into RRD file. All we have to do is to extract it.
First, I need to figure out where the RRD file is. In Cacti, go to Console -> Data Sources, select your edge router and click on IPS-facing interface. In "Data Source Path" field you'll see the name of the RRD file in Cacti's rra directory where data for this interface is stored

Second, we need to know what to extract from this file. I.e I need to know the names of RRD data sources:
rrdtool info border_router_1_traffic_in_14839.rrd
where border_router_1_traffic_in_14839.rrd is file name from previous step.

filename = "border_router_1_traffic_in_14839.rrd"
rrd_version = "0003"
step = 300
last_update = 1262806506
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = NaN
ds[traffic_in].last_ds = "437961211333"
ds[traffic_in].value = 4.9711447176e+05
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = NaN
ds[traffic_out].last_ds = "138465493978"
ds[traffic_out].value = 1.9428099668e+04
ds[traffic_out].unknown_sec = 0

truncated...

The data sources names are traffic_in and traffic_out and this is what we are going to extract. Before we proceed we need to remember, that RRD database size is fixed and determined at the time of creation. When limit is reached, oldest data is overwritten. To avoid losing any data, I am going to extract traffic numbers every hour for the last hour and put inbound and outbound data in separate files.
Incoming traffic:

rrdtool xport -s now-1h -e now DEF:xx=border_router_1_traffic_in_14839.rrd:traffic_in:AVERAGE CDEF:bb=xx,8,* XPORT:bb:"out bits" | grep \|grep -v Na | awk -F'' '{print $2}'| sed -e 's/<\/v><\/row>//'|sed -e 's/e+0/\t/' >> incoming.txt

Outgoing traffic:

rrdtool xport -s now-1h -e now \DEF:xx=border_router_1_traffic_in_14839.rrd:traffic_out:AVERAGE CDEF:bb=xx,8,* XPORT:bb:"out bits" | grep \|grep -v Na | awk -F'' '{print $2}'| sed -e 's/<\/v><\/row>//'|sed -e 's/e+0/\t/' >> outgoing.txt

Both commands should be in one line. Above I converted Bytes/sec into Bits/sec and removed XML formatting. You need these 2 lines into shell script and run it from cron every hour on 2 minutes after the hour so Cacti has time to finish collecting on top of the hour. You'll get 2 files - incoming.txt and outgoing.txt looking like this

6.9655133612 7
7.0568998690 7
6.9008144000 7
7.0245826541 7
7.2076520540 7
6.7448901179 7
6.7471832197 7
6.7365174531 7
6.9710477122 7
7.1586411237 7
7.0991637699 7
7.0189321194 7

This are measurements taken every 5 minutes by Cacti. "6.9655133612 7" means 6.9655133612 * 10^7 bits/sec or 69655133.612 bits/sec.

Now all you have to do on the 1st of the month right after midnight is to convert the data to get rid of second column, sort it and remove top 5%. For 30-day month:

cat incoming.txt |perl -e ' while(<>) {$input = $_; chomp($input);($traffic, $power)
=split(/\t/,$input); $traffic = $traffic*10**$power; print "$traffic\n";}'|egrep -v '^0$'|sort -n -r | head -433 |tail -1

How to generate lots of BGP routes

2009-10-14T14:37:00.004-04:00

I needed to test in the lab whether my Cisco router can handle more than 300K routes - size of current full BGP table. Now, Cisco router can only accept 200 network statements under router bgp configuration, so I would need 1500 routers. Even if I had that many routers to my disposal, it would have taken days to configure all of them. As always, open source software can help. Quagga lets you run OSPF, BGP, RIP, RIPng on Linux and Solaris. If you go with all default options, it is very easy to install. Download and unpack. Go to quagga directory, in my case it was quagga-0.98.6, type
./configure
make
sudo make install
That's it. By default, it went into /usr/local/. I have Debian 4 (Etch) with 2.6.8 kernel and a lot of development packages installed. The only thing I had to do was to add /usr/local/lib to /etc/ld.so.conf file and run /sbin/ldconfig.
Here is the setup

Now I need a valid configuration file for Quagga BGP. Adding 300000 network statements manually is not something system administrators do on Linux. Hence, here is the script

#!/usr/bin/perl

my $host="quagga-host";         #quagga router name
my $logpass="zebra";            #login password
my $enable="zebra";             #enable password
my $myasn="65099";              #local AS number
my $router_id="172.31.2.2";     #bgp router-id
my $remote_as="65001";          #remote-as number
my $remote_ip="172.31.2.1";     #BGP neighbor ip address
my $route_count=0;
my $max_routes=300000;              #max number of routers to generate

open (BGPCONF,'>bgpd.conf')|| die "Can not open bgpd.conf for writing";
print BGPCONF "hostname $host\npassword $logpass\nenable password $enable\nline vty \n";
print BGPCONF "router bgp $myasn\n  bgp router-id $router_id\n  neighbor $remote_ip remote-as $remote_as\n";
MAXR: while ($route_count <= $max_routes ) { 
$octet1=int(rand(223))+1; #generate 1st octet randomly in 1-223 range, 224 and up is multicust and class E  
if ($octet1 ==127) {next;} #need to make sure that 127.X.X.0/24 is excluded

$octet2=0;  
while ( $octet2 <= 255 ){
$octet3=0;
while ( $octet3 <= 255 ) {
print BGPCONF "  network $octet1\.$octet2\.$octet3\.0/24\n";
$octet3++;
$route_count++;
if ($route_count == $max_routes) {last MAXR;}
}
$octet2++;
}
}
close BGPCONF;

this script will generate bgpd.conf for Quagga. Since it is lab environment not connected to any real network, I do not really care about zebra configuration or restricting access to Quagga BGP console. Copy bgpd.conf file into /usr/local/etc and run /usr/local/sbin/bgpd -d -f /usr/local/etc/bgpd.conf -u root -g root Again, this is not production environment. Do not run Quagga as root in production. Here is relevant configuration from Cisco router:

interface GigabitEthernet0/0
ip address 172.31.2.1 255.255.255.0
network 172.31.2.0 mask 255.255.255.0
media-type rj45
negotiation auto
!
router bgp 65001
no synchronization
bgp log-neighbor-changes
neighbor 172.31.2.2 remote-as 65099
no auto-summary
!

Let's see if it works. On Linux host:

sh-2.05b$ telnet localhost 2605
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.

Hello, this is Quagga (version 0.98.6).
Copyright 1996-2005 Kunihiro Ishiguro, et al.


User Access Verification

Password:
quagga-host> sho ip bgp summ
BGP router identifier 172.31.2.2, local AS number 65099
2 BGP AS-PATH entries
0 BGP community entries

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
172.31.2.1      4 65001     283     606        0    0    0 03:41:55        1

Total number of neighbors 1
quagga-host>

on Cisco:

R1#sho ip bgp sum
BGP router identifier 192.0.2.2, local AS number 65001
BGP table version is 330002, main routing table version 330002
310001 network entries using 40920132 bytes of memory
310001 path entries using 16120052 bytes of memory
3/2 BGP path/bestpath attribute entries using 444 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
Bitfield cache entries: current 1 (at peak 2) using 32 bytes of memory
BGP using 57040684 total bytes of memory
BGP activity 310001/0 prefixes, 320001/10000 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
172.31.2.2      4 65099     602     322   330002    0    0 03:38:59   310000

Yep. It works.

BGP and BFD (Bidirectional Forwading Detection)

2009-10-09T16:08:00.006-04:00

If you have Ethernet uplink to your ISP, the chances are high that it looks like this:

The Layer2 device could be "on the wire" provider or Ethernet over MPLS service. The problem arises when, for example, connection between Layer2 device and ISP router goes down

The BGP session with R1 on ISP router will reset immediately, unless you configured "no bgp fast-external-fallover". But R1 will rely on BGP hello messages to detect if neighbor still alive. It might take R1 up to 3 minutes detect that ISP is not available and for these 3 minutes R1 will be sending traffic to black hole instead of re-converging and sending traffic to your backup link. You have backup link, don't you?
Here are the syslog messages from R1 and ISP routers. To imitate link failure I shutdown interface on ISP

router ISP:

Oct 9 17:55:18 UTC: %LINK-5-CHANGED: Interface GigabitEthernet2/5, changed state to administratively down
Oct 9 17:55:18 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.1 Down Interface flap

router R1:

Oct 9 17:57:24 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.2 Down BGP Notification sent
Oct 9 17:57:24 UTC: %BGP-3-NOTIFICATION: sent to neighbor 172.31.255.2 4/0 (hold time expired) 0 bytes

Note the timestamps of first and last messages. That's not good, especially if every minute of downtime costs you a bundle. You can adjust bgp timers, but lowest you can go is 1 second and it could be hard on CPU.
BFD protocol allows you to go to microseconds level. It is very lightweight and easy to configure.
On interfaces facing Layer2 device apply command:

bfd interval 100 min_rx 100 multiplier 3

To check if BFD is configured properly:

#sho bfd neighbor

OurAddr NeighAddr LD/RD RH/RS Holddown(mult) State Int
172.31.255.2 172.31.255.1 3/7 Up 0 (3 ) Up Gi2/5

under "router bgp" configuration:

neighbor [neighbor IP] fall-over bfd

Now let's imitate link failure again. Shutdown interface on router ISP:

Oct 9 18:14:27.408 UTC: %LINK-5-CHANGED: Interface GigabitEthernet2/5, changed state to administratively down
Oct 9 18:14:27.408 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.1 Down Interface flap

On R1

Oct 9 18:14:27.673 UTC: %BGP-5-ADJCHANGE: neighbor 172.31.255.2 Down BFD adjacency down

The difference now in milliseconds. The hard part is to convince your ISP to configure BFD on their side.
At this moment Cisco supports BFD on Ethernet interfaces only and only for directly connected BGP peers, i.e. no multi-hop BGP.
BFD requires UDP ports 3784 and 3785 to be open in case you have ACL applied to your uplink interface.

CCIE R&S v4.0

2009-10-06T15:07:00.003-04:00

I just attended webcast about new CCIE R&S written and lab exam. It looks like MPLS portion of the exam is going to be much easier than 642-611 exam which is required for CCIP certification.

bpduguard vs. bpdufilter

2009-09-16T13:34:00.002-04:00

Suppose you have a switch with servers connected to it. It's called "access layer switch" in Cisco lingo. There is no need to go through all spannig-tree states on these host-facing ports. So, you make port transition faster to forwarding mode with "switchport portfast" interface command. One of the benefits is that if your server boots fast, it does not need to wait until port finishes going through all STP port states and can start transmitting data immediately.
Since servers normally should ignore BPDUs coming from a switch, there is no need to send them to a server in the first place. To filter out outgoing BPDUs apply interface command "spanning-tree bpdufilter enable".
But when BPDU is received on the port with bpdufilter enabled, the port's portfast status is disabled and port will participate in spanning-tree.
At all times network needs to be protected from unauthorized device that might decide to participate in your spanning-tree topology and cause spanning-tree loop or try to hijack STP root. Interface command "spanning-tree bpduguard enable" puts interface in err-disable mode whenever BPDU is received from connected device.
What will happen if you have both bpdufilter and bpduguard enabled on the interface?

spanning-tree portfast
spanning-tree bpdufilter enable
spanning-tree bpduguard enable

bpdufilter takes precedence and bpdugard does not work. Although bpduguard needs more administrative overhead - port needs to be enabled manually - it makes your network more secure.

Cisco router as Frame-Relay switch

2009-08-17T16:15:00.002-04:00

There are a lot of examples on the 'Net how to configure Cisco router to act as Frame Relay switch. All of them use command "frame-relay route". Here is (one more) example

For the topology on the picture above, the configuration would look like this

interface Serial2/0
no ip address
encapsulation frame-relay
no fair-queue
clock rate 128000
frame-relay intf-type dce
frame-relay route 103 interface Serial2/2 301
!
interface Serial2/2
no ip address
encapsulation frame-relay
clock rate 128000
frame-relay intf-type dce
frame-relay route 301 interface Serial2/0 103
end

Here is another way to configure:

interface Serial2/0
no ip address
encapsulation frame-relay
no fair-queue
clock rate 128000
frame-relay intf-type dce
!
interface Serial2/2
no ip address
encapsulation frame-relay
clock rate 128000
frame-relay intf-type dce
!
connect mylink Serial2/0 103 Serial2/2 301

In latter case I used command "connect". Unlike "frame-relay route", "connect" command can be used only once for each pair of DLCIs that need to be able to talk to each other. If you need to have many Frame Relay links, using "connect" is less error-prone and configuration is easier to read.
To see connection status:

#show connection all

ID Name Segment 1 Segment 2 State
========================================
2 mylink Se2/0 103 Se2/2 301 UP

Need to find out Cisco switch serial number?

2009-07-20T12:02:00.003-04:00

Yes, you can do it by logging in and running "show idprom backplane" or "show inventory". What if you have hundreds of switches and need to update your inventory list. Going to every switch, typing commands and writing down serial numbers is very time consuming. There is also Expect script, but I am not a big fan of putting clear text passwords into code. So, my favorite SNMP to the rescue.
Serial number OID for modular switches (4500 and 6500) is .1.3.6.1.2.1.47.1.1.1.1.11.1. For Catalyst 2960 and 3500: 1.3.6.1.2.1.47.1.1.1.1.11.1001

Here is the quick shell script to get serial number and location of each switch.

for i in `cat switches.txt`
do
serial=`snmpwalk -Ov -Oq -v2c -c public $i 1.3.6.1.2.1.47.1.1.1.1.11.1001`
location=`snmpwalk -Ov -Oq -v2c -c public $i system.sysLocation`
echo -e "$i \t $serial \t $location"
done

File switches.txt contains ip addresses of my access switches, one per line. It goes without saying that each switch must have "snmp-server location" configured.