tag:blogger.com,1999:blog-159457262024-03-01T00:39:54.853-05:00Val(config)#Musings about various system administration and network projects I am working on. Lab use only.Unknownnoreply@blogger.comBlogger75125tag:blogger.com,1999:blog-15945726.post-14410743817073291602018-05-03T10:11:00.001-04:002018-05-03T10:11:16.852-04:00NXOS sandbox In my previous post I promised to tell how I setup my own NXOS sandbox to experiment with OpenConfig on NXOS. Cisco's DevNet lab is very useful, but they cut you off after an hour or so and you have to start over. One hour is not nearly enough, so having your own sandbox is big help. You'll need Vagrant and VirtualBox. I used Vagrant 2.0.1 and VirtualBox 5.2.10 on Ubuntu 16.04LTS, but should work on OSX too.<br />
First step - download nxosv-final.7.0.3.I7.3.box from <a href="https://software.cisco.com/download/home/286312239/type/282088129/release/7.0%25283%2529I7%25283%2529" rel="nofollow" target="_blank">Cisco's website</a>. Add it to the list of your Vagrant boxes:<br />
<blockquote class="tr_bq">
vagrant box add --name nxos-sdbx nxosv-final.7.0.3.I7.1.box</blockquote>
<br />
Clone or download Vagrant file and startup-config from <a href="https://github.com/vglinskiy/nxos-sdbx" rel="nofollow" target="_blank">my repo</a> at GitHub.<br />
Create iso file with startup-config (you can find more details <a href="https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/nx-osv/configuration/guide/b_NX-OSv_9000/b_NX-OSv_chapter_01.html" target="_blank">here</a>), in the directory where you put Vagrantfile and nxos_config.txt run command:<br />
<blockquote class="tr_bq">
mkisofs -o nxos-sdbx.iso --iso-level 2 nxos_config.txt</blockquote>
Vagrant will configure VirtualBox to forward sandbox's ssh port to port 5522, http (NX-API) port to 8980 and NETCONF port to 8830.<br />
Now you are ready to power up the switch:<br />
<blockquote class="tr_bq">
vagrant up</blockquote>
It will take some time to boot, you can monitor boot progress by looking at output of<br />
<blockquote class="tr_bq">
ncat -U /tmp/nxos-sdbx</blockquote>
Once switch is up (you'll see login prompt) you can login by either running:<br />
<blockquote class="tr_bq">
vagrant ssh</blockquote>
or<br />
<blockquote class="tr_bq">
ssh -p 5522 admin@localhost</blockquote>
<br />
In first case you'll be dropped into bash shell, type "su - admin" to get familiar CLI prompt. The password is "admin" (no quotes). In the 2nd case, just type "admin" when prompted.<br />
If you want to test OpenConfig models on NXOS, you also need to download RPM files from <a href="https://devhub.cisco.com/artifactory/open-nxos-agents/7.0-3-I7-1/x86_64/" target="_blank">Cisco's devhub</a>. The file names are almost self-explanatory and include model's name. For example, mtx-openconfig-if-ip-1.0.0-7.0.3.I7.1.lib32_n9000.rpm contains OpenConfig model for IP interface configuration. The space in bootflash is limited, so if you want to try all available models, install one RPM at a time and delete it.<br />
Copy file to the sandbox: <br />
<blockquote class="tr_bq">
scp -P 5522 mtx-openconfig-if-ip-1.0.0-7.0.3.I7.1.lib32_n9000.rpm admin@localhost:</blockquote>
Now, log into switch by using one of 2 ways above and run following commands:<br />
<blockquote class="tr_bq">
run bash</blockquote>
<blockquote class="tr_bq">
sudo su -</blockquote>
<blockquote class="tr_bq">
cd /bootflash</blockquote>
<blockquote class="tr_bq">
yum install -y mtx-openconfig-if-ip-1.0.0-7.0.3.I7.1.lib32_n9000.rpm </blockquote>
now start netconf:<br />
<blockquote class="tr_bq">
netconfctl start </blockquote>
Now you can use any NETCONF client and push OpenConfig-compliant XML to the sandbox. I wrote my own simple script which you can find <a href="https://github.com/vglinskiy/openconfig-test/blob/master/send_config.py" target="_blank">here</a>. Standard disclaimer: use at your own risk, no warranty, yadda, yadda, yadda. <br />
<div>
<br /></div>
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
<blockquote class="tr_bq">
</blockquote>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-15945726.post-38347480590596576102018-04-08T17:39:00.000-04:002018-05-01T11:49:01.918-04:00YAML, YANG, ETC I started reading on YANG models and <a href="http://openconfig.net/">OpenConfig</a> about 2 years ago. Around that time I also wrote a script to provision Clos IP fabric. The data structure that script uses to populate Jinja2 templates was totally made up. So, I decided to convert that data structure into OpenConfig-compliant one with a distant goal of feeding it to the switches directly via NETCONF, bypassing the templates. Since I have little desire to create XML files manually, the plan was to write data in YAML, convert into JSON with a <a href="https://github.com/vglinskiy/openconfig-test/blob/master/y2j.py">simple python script</a> and then use <a href="https://github.com/mbj4668/pyang">pyang</a> to validate data structure and generate model-compliant XML.<br />
Before doing all that I completed Cisco's DevNet "NETCONF/YANG on Nexus" lab parts <a href="https://learninglabs.cisco.com/lab/yang_devnet-format_part1/step/1">1</a>, <a href="https://learninglabs.cisco.com/lab/yang_devnet-format_part2/step/1">2</a> and <a href="https://learninglabs.cisco.com/lab/yang_devnet-format_part3/step/1">3</a>. Part 3 of the lab uses OpenConfig YANG models, so you can skip part 2 entirely. <a href="https://github.com/DevNetSandbox/sbx_nxos/blob/master/learning_labs/yang/03-yang/add_oc_loopback.py">XML in Cisco's script</a> looked simple enough, but it took me few short days of reading RFCs and data models, trials, errors and reading data models again to come up with YAML file that pyang finally validated and converted into XML. Here is what I got:<br />
<blockquote class="tr_bq">
<pre style="white-space: pre-wrap; word-wrap: break-word;">---
"openconfig-interfaces:interfaces":
interface:
- name: eth1/1
config:
name: eth1/1
type: ethernetCsmacd
subinterfaces:
subinterface:
- index: 0
openconfig-if-ip:ipv4:
addresses:
address:
- ip: 172.16.1.0
config:
ip: 172.16.1.0
prefix-length: 31</pre>
</blockquote>
You can see XML file that pyang generated <a href="https://github.com/vglinskiy/openconfig-test/blob/master/1_int.xml">here</a>. Pyang writes everything in one line, so I edited it for readability. I used <a href="https://github.com/vglinskiy/openconfig-test/blob/master/send_config.py">another script</a> to push resulted XML to virtual Nexus switch. And it did not work. The error message said that namespace was empty. (Since Cisco gives access to NX-OS sandbox only for one hour, I had to setup my own sandbox. I'll write another post about it). Another few short hours and manual XML editing - nobody should be editing XML manually - I came up with something that my virtual Nexus accepted (<a href="https://github.com/vglinskiy/openconfig-test/blob/master/1_int_formatted.xml">github</a>).<br />
<blockquote class="tr_bq">
<pre style="word-wrap: break-word;"><span style="white-space: pre-wrap;"><config>
<interfaces xmlns="http://openconfig.net/yang/interfaces">
<interface>
<name>eth1/1</name>
<config>
<name>eth1/1</name>
<description>OpenConfig</description>
<type>ianaift:ethernetCsmacd</type>
</config>
<subinterfaces>
<subinterface>
<index>0</index>
<ipv4>
<addresses>
<address>
<ip>172.16.1.0</ip>
<config>
<ip>172.16.1.0</ip>
<prefix-length>31</prefix-length>
</config>
</address>
</addresses>
</ipv4>
</subinterface>
</subinterfaces>
</interface>
</interfaces>
</config>
</span></pre>
</blockquote>
But I had to add "no switchport" command to Eth1/1 manually first. I could not find anything in OpenConfig or IETF models to make switch do it.<br />
Now is time to save my hard-won changes to startup config - see commented line 29 in the <a href="https://github.com/vglinskiy/openconfig-test/blob/master/send_config.py">script</a>. It's commented, because virtual switch complained that "startup" is incorrect datastore, while RFC6241 says otherwise. It became clear why switch and RFC disagreed after I looked at switch's NETCONF capabilities:<br />
<blockquote class="tr_bq">
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:capability:writable-running:1.0</span><br />
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:capability:rollback-on-error:1.0</span><br />
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:capability:confirmed-commit:1.1</span><br />
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:capability:validate:1.1</span><br />
<a href="http://cisco.com/ns/yang/cisco-nx-os-device?revision=2017-08-31&module=Cisco-NX-OS-device&deviations=Cisco-NX-OS-device-deviations" style="background-color: white; font-family: Calibri, Helvetica, sans-serif; font-size: 16px; text-size-adjust: auto;">http://cisco.com/ns/yang/cisco-nx-os-device?revision=2017-08-31&module=Cisco-NX-OS-device&deviations=Cisco-NX-OS-device-deviations</a><br />
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:base:1.0</span><br />
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:base:1.1</span><br />
<span style="background-color: white; font-family: "calibri" , "helvetica" , sans-serif; font-size: 16px;">urn:ietf:params:netconf:capability:candidate:1.0</span><br />
<a href="http://openconfig.net/yang/bgp?revision=2016-06-06&module=openconfig-bgp&deviations=openconfig-bgp-deviations" style="background-color: white; font-family: Calibri, Helvetica, sans-serif; font-size: 16px; text-size-adjust: auto;">http://openconfig.net/yang/bgp?revision=2016-06-06&module=openconfig-bgp&deviations=openconfig-bgp-deviations</a><br />
<a href="http://openconfig.net/yang/interfaces?revision=2016-05-26&module=openconfig-interfaces&deviations=openconfig-interfaces-deviations" style="background-color: white; font-family: Calibri, Helvetica, sans-serif; font-size: 16px; text-size-adjust: auto;">http://openconfig.net/yang/interfaces?revision=2016-05-26&module=openconfig-interfaces&deviations=openconfig-interfaces-deviations</a><br />
<a href="http://openconfig.net/yang/interfaces/ip?revision=2016-05-26&module=openconfig-if-ip&deviations=openconfig-if-ip-deviations" style="background-color: white; font-family: Calibri, Helvetica, sans-serif; font-size: 16px; text-size-adjust: auto;">http://openconfig.net/yang/interfaces/ip?revision=2016-05-26&module=openconfig-if-ip&deviations=openconfig-if-ip-deviations</a></blockquote>
<br />
Capability ":startup" is missing. So, no way to save configuration via NETCONF, I guess.<br />
I am going to check the extent of OpenConfig support on Juniper and Arista devices and for now implement 1st step of my plan - convert totally bogus data structure into OpenConfig-compliant oneUnknownnoreply@blogger.com1tag:blogger.com,1999:blog-15945726.post-76386604255824127802018-03-13T12:51:00.000-04:002018-03-13T22:41:23.341-04:00DIY routing to the host Cumulus Networks promotes routing to the host via <a href="https://cumulusnetworks.com/products/host-pack/" rel="nofollow">Host Pack</a> software package as a way to provide host network redundancy without using proprietary MLAG or mostly incompatible EVPN ESI multihoming solutions from switch vendors. While Host Pack seems to be geared towards hosts running Linux containers, it got me thinking how can I do routing to bare metal host. The routing protocol of choice is BGP. Now I need an IP address on the interface that never goes down and make sure that my server and client applications use that IP. That same IP will be advertised via BGP from the host. Loopback interface is obvious choice for this kind of interface. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBXa7erhBwVWengw9m1GfKEtcqRLgGLmv2stI_Ri_JBlX3EBqjwunKq2-n5ZIH1JrdNgN1esKw8u2cqYqpXjpaHbOBJxSYbzhaOf-Mp1_SS94e9Jy9SMCCgB0mMNjVhX190Hek/s1600/L3toHOST.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="537" data-original-width="794" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBXa7erhBwVWengw9m1GfKEtcqRLgGLmv2stI_Ri_JBlX3EBqjwunKq2-n5ZIH1JrdNgN1esKw8u2cqYqpXjpaHbOBJxSYbzhaOf-Mp1_SS94e9Jy9SMCCgB0mMNjVhX190Hek/s400/L3toHOST.png" width="400" /></a></div>
srv1 and srv2 are Vagrant minimal/xenial64 boxes. srv1, tor1, tor2 and tor3 run BGP, srv2 is connected to 172.16.99.0/24 network hosted on tor3. Let's configure "always-up" IP address on srv1:<br />
<blockquote class="tr_bq">
sudo ip addr add 100.100.100.100/32 dev lo:100</blockquote>
While binding server application like Apache to specific IP address or interface is pretty straightforward task, selecting source address for outgoing connection is a bit more complicated. Here is how <a href="http://linux-ip.net/html/routing-saddr-selection.html#ftn.idm140337857476896">Linux selects source IP address</a>:<br />
<blockquote class="tr_bq">
The application can request a
particular IP
<a class="footnote" href="http://linux-ip.net/html/routing-saddr-selection.html#ftn.idm140337857476896" name="idm140337857476896"><sup class="footnote">[20]</sup></a>,
the kernel will use the <code class="constant">src</code> hint from the chosen
route path
<a class="footnote" href="http://linux-ip.net/html/routing-saddr-selection.html#ftn.idm140337857472176" name="idm140337857472176"><sup class="footnote">[21]</sup></a>,
or, lacking this hint, the kernel will choose the first address
configured on the interface which falls in the same network as the
destination address or the nexthop router.</blockquote>
I want it to be transparent for the applications and left on its own, Linux most likely will select IP address of one of the physical interfaces. The only option left is to make sure that route to 172.16.99.0/24 on srv1 is programmed with src 100.100.100.100.<br />
In this lab I am using <a href="http://bird.network.cz/">BIRD</a> 1.6 to run BGP on srv1, but Free Rang Routing will work too.<br />
<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<blockquote class="tr_bq">
router id 100.100.100.100;<br />
<br />
filter my_vip<br />
{<br />
if net = 100.100.100.100/32 then accept;<br />
reject;<br />
}<br />
filter remote_site<br />
{<br />
if net ~ [ 172.16.99.0/24 ] then<br />
{<br />
krt_prefsrc = 100.100.100.100; #set src <br />
accept;<br />
}<br />
reject;<br />
}<br />
<br />
protocol kernel {<br />
scan time 60;<br />
import none;<br />
export filter remote_site;<br />
persist; # routes stay even if bird is down<br />
merge paths on; # ECMP<br />
}<br />
<br />
protocol device {<br />
scan time 60;<br />
}<br />
<br />
protocol direct {<br />
interface "enp0s[8|9]", "lo*";<br />
}<br />
protocol bgp host_2rtr1 {<br />
local as 65499;<br />
neighbor 192.168.11.3 as 64900;<br />
export filter my_vip;<br />
import filter remote_site;<br />
}<br />
protocol bgp host_2rtr2 {<br />
local as 65499;<br />
neighbor 192.168.22.3 as 64920;<br />
export filter my_vip;<br />
import filter remote_site;<br />
}</blockquote>
</blockquote>
</blockquote>
<br />
Let's see BGP routes we get from tor1 and tor2:<br />
bird> show route 172.16.99.0/24<br />
172.16.99.0/24 via 192.168.11.3 on enp0s8 [host_2rtr1 16:15:49] * (100) [AS65000i]<br />
via 192.168.22.3 on enp0s9 [host_2rtr2 16:15:49] (100) [AS65000i]<br />
<br />
Only one route is marked as primary, I could not find "bestpath as-path multipath-relax" equivalent in BIRD. It's required because tor1 and tor2 have different AS numbers. But no worries, "merge path on" under "protocol kernel" will take care of this. Indeed:<br />
vagrant@srv1:~$ ip route show 172.16.99.0/24 <br />
172.16.99.0/24 proto bird src 100.100.100.100 <br />
nexthop via 192.168.11.3 dev enp0s8 weight 1 <br />
nexthop via 192.168.22.3 dev enp0s9 weight 1<br />
both routes are installed and claim to use 100.100.100.100 as a source IP address to reach srv2 network.<br />
Let's verify. I start pinging srv2 from srv1 and run tcpdump on srv2 side.<br />
<br />
<br />
<br />
vagrant@srv1:~$ ping 172.16.99.200<br />
Here is tcpdump output on srv2: <br />
02:22:09.753864 IP <span style="color: red;">100.100.100.100</span> > 172.16.99.200: ICMP echo request, id 5024, seq 1, length 64<br />
02:22:09.753906 IP 172.16.99.200 > 100.100.100.100: ICMP echo reply, id 5024, seq 1, length 64<br />
02:22:10.750884 IP <span style="color: red;">100.100.100.100</span> > 172.16.99.200: ICMP echo request, id 5024, seq 2, length 64<br />
02:22:10.750920 IP 172.16.99.200 > 100.100.100.100: ICMP echo reply, id 5024, seq 2, length 64<br />
<br />
<br />
As you can see, packets are coming from 100.100.100.100, even though I did not specify source IP address for the ping.<br />
Similar test with ssh<br />
vagrant@srv1:~$ ssh 172.16.99.200<br />
<br />
02:31:37.652419 IP <span style="color: red;">100.100.100.100</span>.54634 > 172.16.99.200.ssh: Flags [S], seq 3479345726, win 29200, options [mss 1460,sackOK,TS val 2387063 ecr 0,nop,wscale 7], length 0<br />
02:31:37.652473 IP 172.16.99.200.ssh > 100.100.100.100.54634: Flags [S.], seq 2929355359, ack 3479345727, win 28960, options [mss 1460,sackOK,TS val 2404414 ecr 2387063,nop,wscale 7], length 0<br />
02:31:37.665081 IP <span style="color: red;">100.100.100.100</span>.54634 > 172.16.99.200.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 2387066 ecr 2404414], length 0<br />
02:31:37.666605 IP <span style="color: red;">100.100.100.100</span>.54634 > 172.16.99.200.ssh: Flags [P.], seq 1:42, ack 1, win 229, options [nop,nop,TS val 2387066 ecr 2404414], length 41<br />
02:31:37.666621 IP 172.16.99.200.ssh > 100.100.100.100.54634: Flags [.], ack 42, win 227, options [nop,nop,TS val 2404418 ecr 2387066], length 0<br />
<br />
<br />
<br />
<br />
And last test - failover. Since my lab setup is entirely virtual, the goal was to test if failover works at all and not how fast it does. You need real hardware to check the speed of failover.<br />
<br />
vagrant@srv1:~$ iperf -s -B 100.100.100.100<br />
<br />
vagrant@srv2:~$ iperf -M 1000 -b 80K -i 1 -c 100.100.100.100 -t 120<br />
<br />
In my case traffic took srv2 -> tor3 -> tor1 -> srv1 path. While iperf was running, I shutdown BGP session between tor1 and srv1. Here are the results from iperf:<br />
[ 3] 46.0-47.0 sec 384 KBytes 3.15 Mbits/sec<br />
[ 3] 47.0-48.0 sec 512 KBytes 4.19 Mbits/sec<br />
[ 3] 48.0-49.0 sec 384 KBytes 3.15 Mbits/sec<br />
[ 3] 49.0-50.0 sec 0.00 Bytes 0.00 bits/sec<br />
[ 3] 50.0-51.0 sec 0.00 Bytes 0.00 bits/sec<br />
[ 3] 51.0-52.0 sec 0.00 Bytes 0.00 bits/sec<br />
[ 3] 52.0-53.0 sec 256 KBytes 2.10 Mbits/sec<br />
[ 3] 53.0-54.0 sec 512 KBytes 4.19 Mbits/sec<br />
<br />
<br />
That 3-second interval of 0.00bits/sec is failover time. Again, since it's virtual environment, your mileage may vary.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-22158372631950788092018-03-05T10:47:00.000-05:002018-03-05T10:47:12.277-05:00Interface uptime time formatI was looking into why my <a href="https://napalm-automation.net/">NAPALM-based</a> script could not validate state of SVI interface on Cisco Nexus and decided to dig into NAPALM source code. I found something amusing in <a href="https://github.com/napalm-automation/napalm/blob/develop/napalm/nxos/nxos.py">nxos.py</a> module line 230:<div>
<blockquote class="tr_bq">
<span class="pl-k" style="background-color: white; box-sizing: border-box; color: #d73a49; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">def</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;"> </span><span class="pl-en" style="background-color: white; box-sizing: border-box; color: #6f42c1; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">_compute_timestamp</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">(</span><span class="pl-smi" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">stupid_cisco_output</span><span style="background-color: white; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; white-space: pre;">):</span></blockquote>
The code that follows after that tries to convert Cisco's way of reporting uptime into epoch. I totally understand the frustration. Let's say you want to find out when interface flapped last time. Here are the few examples:<br />
<blockquote class="tr_bq">
Last link flapped 5d02h <br />Last link flapped 16week(s) 5day(s)<br />Last link flapped never<br />Last link flapped 23:39:41</blockquote>
I understand, that "show" command output is intended for human consumption and it's easy to read. Unfortunately, Cisco provides same kind of time format in XML output, which is supposed to be consumed by some kind of automation. Good luck parsing it. While Arista and Juniper also display interface uptime in similar fashion in plain text output, they do much better job in structured output. Here is JUNOS output in JSON:<br />
<br />
<blockquote>
"interface-flapped" : [ <br />{ <br /> "data" : "2017-09-13 14:39:29 EDT (24w0d 20:45 ago)",<br /> "attributes" : {"junos:seconds" : "14589956"} <br />} <br />], </blockquote>
or XML:<br />
<br />
<blockquote class="tr_bq">
<interface-flapped junos:seconds="14590210" > 2017-09-13 14:39:29 EDT (24w0d 20:50 ago)</interface-flapped></blockquote>
<div>
Arista's JSON:</div>
</div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
"Ethernet5/1": { <br /> "lastStatusChangeTimestamp": 1519771449.121221,</blockquote>
<br />
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-2696431308255089922018-02-24T14:08:00.000-05:002018-02-24T14:08:58.658-05:00IP fabric over unnumbered interfaces So, you read the industry websites, know that IP fabric is the next best thing in data center networking and decided to take a plunge and build your own. Nothing big to start with: 2 spine and 8 leaf switches. Now you realize that your IPAM system does not have API and you have to assign 16 IP addresses for transit links and 10 IPs for loopback interfaces manually. While not insurmountable task, it's tedious. Fortunately, Cisco's NXOS and Juniper's JUNOS let you configure ip unnumbered ethernet interface and now you need only 10 IPs for loopbacks.<br />
I created a small virtual lab of 3 NXOSv switches - 2 spines and 2 leafs - to test the concept. I could not make BGP work directly over unnumbered interfaces, so I configured OSPF to advertise loopbacks and BGP peering between loopbacks. To simplify configuration even more, I configured dynamic BGP peering on the spines.<br />
Why would you need BGP if you already have OSPF? You might want to run another next best thing - VXLAN with EVPN control plane.<br />
To run the lab you'll need Vagrant, VirtualBox, Linux machine with 32GB of memory and Vagrant <a href="https://software.cisco.com/download/release.html?mdfid=286312239&softwareid=282088129&release=7.0(3)I5(1)" rel="nofollow">package of NXOS from Cisco</a>. You need to have CCO account and may be maintenance contract to download the image. Do not ask me to provide the image. I used NXOSv image <span style="background-color: white; color: #24292e; font-size: 16px;"><span style="font-family: Times, Times New Roman, serif;">nxosv-final.7.0.3.I7.1.box</span></span><span style="background-color: white; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">, </span>Vagrant 2.0.1, VirtualBox 5.2.6 and Ubuntu Linux 16.04LTS. Although the lab worked with earlier versions of Vagrant and VirtualBox and should run on any Linux distro.<br />
<br />
<ul>
<li><a href="https://github.com/vglinskiy/BGPoverUnnumbered" rel="nofollow">Clone or download from git</a> configuration files</li>
<li>Run "create_iso.sh" script to build ISO files with configuration for each NXOSv switch</li>
<li>run "vagrant up". Depending on the resources, it might take up 10 minutes for all 4 switches to come up fully.</li>
</ul>
<div>
You can see boot progress by connecting to consoles: "ncat -U /tmp/<switch name="">", where <switch name=""> are leaf1, leaf2, spine1 or spine2. After switches are up, you can log in by running "vagrant ssh <switch name="">". You'll be presented with bash shell, to get to NXOS prompt, type "su - admin", password is admin.</switch></switch></switch></div>
<br />
Disclaimer: this is in no way shape or form production-ready configuration and was not tested for any side effects. Use it at your own risk.<br />
<br />
Happy Labbing!Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-9566571826313919532018-01-27T18:56:00.000-05:002018-01-27T18:56:26.282-05:00Cattle vs Pets rantOk, by now pretty much everybody in IT knows about <a href="http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/">cattle and pets</a> meme. Your IT infrastructure should be disposable cattle and not precious pets. While I agree that IT infrastructure should be disposable, I have a problem with this specific choice of words. Only people who never set foot in a cattle ranch can say that. Any farm roaming bovine actually brings money, so treating one dead cow like it's nothing to worry about is going to cost you. One sick cow must get as much, if not more, attention than pet cat or your entire operations can be in jeopardy.<br />
Let's come up with different analogy.Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-15945726.post-62279804388574426812015-07-13T18:56:00.001-04:002015-07-13T18:56:21.333-04:00It does not take that longWhen some overly enthusiastic SDN neophyte or sysadmin-turned-devops-engineer tells me about how long it used to take to create a VLAN and how quick it can be done with new magic SDN controller or automation tool of the day, I have to assume that this person does not really know what she or he is talking about.<br />
What takes long time is change control procedure and no amount of automation or SDN is going to change it.<br />
Since we are talking about VLANs here, in modern data center network you do not need to create a VLAN on more than one switch.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-16555248931161226292014-12-15T22:00:00.003-05:002018-03-24T14:08:03.481-04:00Manage network devices with Ansible<div class="p1">
Ansible is one of the best technologies we took from buggers after the war. I loved "Ender's game" book, not the movie.</div>
<div class="p1">
Inspired by <a href="https://pynet.twb-tech.com/blog/ansible/ansible-cfg-template.html" target="_blank">excellent posts</a> by Kirk Byers I decided to try <a href="http://www.ansible.com/home" target="_blank">Ansible</a> not only to generate configuration for network switches, but to make configuration changes. I have virtual Arista switch running in VirtualBox, so this is where I ran my tests, but it's easy to replicate with Juniper or Cisco Nexus switches. I used user "root", although any user with priviledge level 15 will do.</div>
<div class="p1">
First, enable root user on Arista switch:</div>
<div class="p1">
Arista-5#(conf) aaa root secret SecretPassword</div>
<div class="p2">
<br /></div>
<div class="p1">
Next step is to go to managemnent server and generate ssh key without password. Resulted public key should be added to /root/authorized_keys file on Arista switch.</div>
<div class="p2">
<br /></div>
<div class="p1">
Now, to Ansible.</div>
<div class="p1">
<br /></div>
<div class="p1">
My ansible.cfg:</div>
<div>
<span style="background-color: white;">[defaults]</span></div>
<div>
<span style="background-color: white;">host_key_checking=False</span></div>
<div>
<span style="background-color: white;">hostfile=/home/user1/ansible/hosts</span></div>
<div>
<span style="background-color: white;">log_path=~/ansible.log</span></div>
<div>
<br /></div>
<div>
Let's do very simple task: copy new OS image file and update boot variable. Here is my very simple playbook upgrade.yml:</div>
<span style="background-color: white;">---<br />- hosts: arista<br /> remote_user: root <br /> tasks:<br /> - name: Push image<br /> copy: src=/home/user1/Documents/ansible/vEOS-1.swi dest=/mnt/flash/vEOS-1.swi<br /> - name: Change boot variable<br /> command: FastCli -p15 -c "install source vEOS-1.swi now"</span><br />
<div class="p2">
<br /></div>
<div class="p1">
Really simple inventory file:</div>
<div class="p2">
[arista]</div>
<div class="p2">
arista-5</div>
<div>
<br /></div>
<div class="p1">
Let's run it:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCPreoAFITPtOeRJGbqZJHkBX0JH99eblRDxl_Jz1zTejloBtIDSRs-t5sE1q3mHZRF89lGeTdaroNlT7WWcWqvay0CIRuQtPHikthydWrhX8vNFDPhNIOHWae_jkmUNVruWsx/s1600/playbook.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="203" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCPreoAFITPtOeRJGbqZJHkBX0JH99eblRDxl_Jz1zTejloBtIDSRs-t5sE1q3mHZRF89lGeTdaroNlT7WWcWqvay0CIRuQtPHikthydWrhX8vNFDPhNIOHWae_jkmUNVruWsx/s1600/playbook.png" width="640" /></a></div>
<div class="p1">
<br /></div>
<div class="p2">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
It worked, boot variable now points to vEOS-1.swi file.</div>
<div class="p1">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi55iBkpH9tW2BBgewjURwiEszZc-gAOgsiLcTq3VPEPVbjvKEd4XXYCf8J_ah7rjmXFVpTmGG1SqMOV1V39kc49DJhXn2aGuuY1VcSQrjapEoOPAo2nEZQI8e9p6fAgVjG0jQY/s1600/arista5.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="354" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi55iBkpH9tW2BBgewjURwiEszZc-gAOgsiLcTq3VPEPVbjvKEd4XXYCf8J_ah7rjmXFVpTmGG1SqMOV1V39kc49DJhXn2aGuuY1VcSQrjapEoOPAo2nEZQI8e9p6fAgVjG0jQY/s1600/arista5.png" width="640" /></a></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<div class="p1">
What happens if you you use RADIUS for authentication and have to enter </div>
<div class="p1">
password to log into your switch? In this case Ansible uses <a href="http://sourceforge.net/projects/sshpass/" target="_blank">sshpass</a> which stores your password in the memory. From sshpass man page:</div>
<div class="p2">
<br /></div>
<blockquote class="tr_bq">
It is close to impossible to securely store the password, and users of sshpass should consider whether ssh's public key authentication provides the same end-user experience, while involving less hassle and being more secure.</blockquote>
<div class="p2">
<br /></div>
<br />
<div class="p1">
If you are willing to take this risk, insert "ask_pass=True" line into your ansible.cfg to be prompted for password or run ansible_playbook command with -k option.</div>
Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-15945726.post-65815555744639191202014-11-22T23:03:00.000-05:002014-11-22T23:03:55.754-05:00Before you finish your next automation scriptAn excellent <a href="http://online.wsj.com/articles/automation-makes-us-dumb-1416589342" target="_blank">article from Wall Street Journal </a>about pitfalls of automation:<br />
<br />
<br />
<blockquote class="tr_bq">
<span class="s1">This philosophy traps people in a vicious cycle of de-skilling. By isolating them from hard work, it dulls their skills and increases the odds that they will make mistakes. When those mistakes happen, designers respond by seeking to further restrict people’s responsibilities—spurring a new round of de-skilling.</span></blockquote>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Something to consider when you hand over that new automation script to your network operations center.</span></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-80328661469724696922014-11-19T21:35:00.000-05:002014-11-19T21:35:01.586-05:00Al-too-na! Al-too-na!There is a lot of press these days about new <a href="https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/" target="_blank">Facebook datacenter </a>and how it (almost) means <a href="http://www.wired.com/2014/11/facebooks-new-data-center-bad-news-cisco/" target="_blank">death of Cisco-Juniper-Arista</a>. Here are the questions:<br />
Did somebody try to run multicast on <a href="http://cumulusnetworks.com/media/cumulus/pdf/misc/Cumulus-Linux-Datasheet.pdf" target="_blank">white-box switch</a>?<br />Where do you place Rendezvous Point in <a href="http://www.jedelman.com/home/facebook-altoona-network-diagram-in-2-d" target="_blank">this network</a>?<br />There is only handful of companies with Facebook- or Google-size network, but hundreds of companies requiring multicast support in their data center. Until white box switches learn to run IGMP/PIM/MSDP Cisco and Juniper will have plenty of customers with deep pockets.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-87345577251399624152014-06-18T15:54:00.001-04:002014-06-18T15:54:43.225-04:00Auto ProvisioningNetwork automation is hot topic lately and has been my favorite past time for quite a while. While vendors provided tools to make changes on the switches and routers - Cisco Works come to mind - getting initial configuration on the switch was still manual process. Network operators would come up with some kind of template in Excel or text file and then replace host name and management IP address. To get initial configuration on the switch, one would use copy/paste, a process prone to errors, or upload text file to the switch and copy it to startup-config or candidate configuration. Latter process required configured management IP address and default gateway. In addition to that operators had to upgrade/downgrade operating system.<br />
Fast forward to today's era of Chef/Puppet/Ansible/Other appropriated household word. Cisco came up with (and Arista duly replicated) Power-on Auto Provisioning for its Nexus switches, POAP for short. In DHCP option 67 you need to provide Python script name that switch will download and execute. Arista also supports Bash scripts. Cisco even provides a script that can upgrade NX-OS and download corresponding configuration file based on one of the following parameters: switch name, management interface MAC address, serial number or CDP neighbors. You can write your own script to generate configuration on the fly.<br />
Juniper came up with somewhat catchier name - Zero Touch Provisioning. It does not allow you run scripts. Juniper touted Junos as first network operating system that allowed scripts long before Cisco came up with EEM. Also, you have to use DHCP vendor options to encode configuration file name and Junos image file name. So, in your DHCP server you have to put something like this: 0x00306a696e7374616c6c2d7166782d332d31332e325835302d4431352e332d646f6d65737469<br />
632d7369676e65642e74677a0111646e6a722d6c61622d716678312e636667. Not very informative. Juniper promised to implement SLAX script support in near future and abandon DHCP vendor option in favor of option 67. Python support might come later.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-13339178213694614992014-04-23T12:02:00.002-04:002014-04-23T12:02:47.986-04:00Python on NexusIf you are relying on handy "cisco" python module on your Nexus 5500 or 6K switches - stop. Cisco made considerable changes in version 7.0(1)N1(1).<br />
Here what it looked like in 6.0(2)N2(3):<br />
switch# python<br />
Python 2.7.2 (default, Nov 27 2012, 17:50:33)<br />
[GCC 4.3.2] on linux2<br />
Type "help", "copyright", "credits" or "license" for more information.<br />
Loaded cisco NxOS lib!<br />
>>> import cisco<br />
>>> dir(cisco)<br />
['BGPSession', 'BufferDepthMonitor', 'CLI', 'CheckPortDiscards', 'CiscoSecret', 'CiscoSocket', 'Feature', 'History', 'IPv4ACL', 'IPv6ACL', 'Interface', 'Key', 'LineParser', 'MacAddressTable', 'OSPFSession', 'Routes', 'SectionParser', 'VRF', 'Vlan', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'acl', 'bfd', 'bgp', 'buffer_depth_monitor', 'check_port_discards', 'cisco_secret', 'cisco_socket', 'cli', 'dhcp', 'eigrp', 'feature', 'get_global_vrf', 'get_valid_port', 'history', 'hsrp', 'interface', 'interface-vlan', 'key', 'lacp', 'line_parser', 'mac_address_table', 'md5sum', 'msdp', 'ospf', 'ospfv3', 'pim', 'private-vlan', 'ptp', 'rip', 'routes', 'scheduler', 'section_parser', 'set_global_vrf', 'show_queues', 'show_run', 'ssh', 'tacacs', 'telnet', 'transfer', 'udld', 'vlan', 'vpc', 'vrf', 'vrrp', 'vtp']<br />
>>><br />
<br />
In 7.0(1)N1(1)<br />
>>> import cisco<br />
>>> dir(cisco)<br />
['__doc__', '__name__', '__package__', 'cli', 'cli_execution_error', 'cli_syntax_error', 'clid', 'clip']<br />
<br />
According to Cisco representative "Starting 7.0(1)N1(1), python interpreter on N6K has been modified to look like python interpreter on N7K."<br />
WHY DID NOT THEY DO OTHER WAY AROUND: MODIFY PYTHON ON N7K TO MATCH N6K?Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-41430837245047133702013-09-10T15:19:00.001-04:002014-11-19T21:36:25.683-05:00Stop using "ships in the night" It seems that every month brings news of yet another overlay network. STT, OTV, VXLAN, NVGRE and may be many more I've not heard of. People often use "ships in the night" words in the same sentence referring to the fact that these overlay network know nothing about underlying physical network infrastructure. However, nowadays even the smallest sea or ocean going vessel not only knows about any ship nearby, but acutely aware of a few satellites thanks to radar and GPS systems.<br />
So, to keep up with pace of time and advances in technologies (after all, we are in tech business) I propose new analogy for overlay and physical networks: polar bear and penguin. Those 2 definitely do not meet in real life. I am not considering corner case like zoo.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-24320042269459724452013-09-04T10:02:00.000-04:002013-09-04T10:02:15.551-04:00RantAny vendor requiring registration in order to access online documentation should be condemned to untangle network cables for the rest of the their product life.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.vibrant.com/images/cables/lopsa/green-mess.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="240" src="http://www.vibrant.com/images/cables/lopsa/green-mess.jpg" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-15945726.post-68970586279379674212013-08-07T16:33:00.000-04:002013-08-07T16:33:54.422-04:00Very interesting and popular explanation of Shennon limit: <a href="http://web.mit.edu/newsoffice/2010/explained-shannon-0115.html" target="_blank">Part 1</a> and <a href="http://web.mit.edu/newsoffice/2010/gallager-codes-0121.html" target="_blank">Part 2</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-45384403630365551142013-07-19T12:31:00.000-04:002013-07-19T12:31:30.973-04:00LACP timer and what it meansLACP (IEEE 802.3ad)is protocol used to bundle several physical interfaces to form single logical channel. It has a timer which defines how often devices inter-connected via this bundle exchange LACP PDUs or control messages. Currently, this timer can be set to either "rate fast" - 1 second, or "rate normal" - 30 seconds. What is not always clear is that when you configure "lacp rate <fast normal="">" on Cisco or "set interfaces ae1 aggregated-ether-options lacp periodic fast" on Juniper, you do not configure how often this switch will send LACP PDUs. This command means that switch where this command is applied will expect to receive LACP PDUs with this frequency from the partner on the other side of logical channel.</fast><br />
<fast normal="">Here is quick test. I have Nexus5500 connected to Cat6500. </fast>Let's configure port-channel between them with one physical member interface.<br />
<br />
Cat6500#show run interface TenGigabitEthernet 1/5<br />
!<br />
interface TenGigabitEthernet1/5<br />
switchport<br />
switchport trunk encapsulation dot1q<br />
switchport mode trunk<br />
<span style="background-color: yellow;">lacp rate fast</span><br />
channel-group 5 mode active<br />
<br />
Nexus5500# show running-config interface Ethernet 1/1<br />
!<br />
interface Ethernet1/1<br />
switchport mode trunk<br />
channel-group 1 mode active<br />
<br />
"lacp rate normal" is default setting on Nexus, so this command does not show up in the output, but we can confirm:<br />
<br />
Nexus5500# show running-config interface Ethernet 1/1 all | include lacp<br />
lacp port-priority 32768<br />
<span style="background-color: yellow;">lacp rate normal</span><br />
<span style="background-color: yellow;"><br /></span>
<br />
Cat6500 is configured with rate fast and Nexus5500 - with rate normal. Let's see what's going on behind the scene.<br />
On Catalyst:<br />
Cat6500#show lacp internal<br />
Flags: S - Device is requesting Slow LACPDUs<br />
<span style="background-color: yellow;">F - Device is requesting Fast LACPDUs</span><br />
A - Device is in Active mode P - Device is in Passive mode <br />
<br />
Channel group 3<br />
LACP port Admin Oper Port Port<br />
Port <span style="background-color: yellow;">Flags</span> State Priority Key Key Number State<br />
Te1/5 <span style="background-color: yellow;">FA</span> bndl 32768 0x3 0x3 0x106 0x3F <br />
<br />
F flags says that Cat6500 <b>requesting</b> fast LACP PDUs from its partner.<br />
<br />
On Nexus it's a little bit backwards, the "show" command tells you partner status, not its own.<br />
<br />
Nexus5500# show lacp neighbor interface port-channel 1<br />
Flags: <span style="background-color: yellow;">S - Device is sending Slow LACPDUs</span> F - Device is sending Fast LACPDUs<br />
A - Device is in Active mode P - Device is in Passive mode<br />
port-channel1 neighbors<br />
Partner's information<br />
Partner Partner <span style="background-color: yellow;">Partner</span><br />
Port System ID Port Number Age <span style="background-color: yellow;">Flags</span><br />
Eth1/1 20,0-13-5f-20-63-80 0x106 910 <span style="background-color: yellow;">SA</span><br />
<br />
LACP Partner Partner Partner<br />
Port Priority Oper Key Port State<br />
32768 0x3 0x3f<br />
<div>
<br /></div>
<div>
Nexus5500 says, that its partner - Cat6500 - is sending LACP PDUs every 30 seconds.</div>
<br />
<div>
<br /></div>
<br />Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-15945726.post-77089717764298101342013-07-15T09:11:00.000-04:002013-07-15T09:15:25.148-04:00Find MAC address for IPv4 Multicast groupWhen troubleshooting multicast problem I find myself checking if IGMP snooping works as intended. "show mac address-table multicast" on Cisco switches shows MAC addresses of multicast groups. Tired of converting Multicast IPs into MACs with pencil and paper, I wrote my first ever script in Python which does just that. It takes IP address of multicast group as a parameter. Although I did some testing there might be bugs, so beware.<br />
See Cisco's <a href="http://www.cisco.com/en/US/docs/ios/solutions_docs/ip_multicast/White_papers/mcst_ovr.html" target="_blank">white paper</a> for explanation how the conversion is done.<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=RqjBbz74" style="border: none; height: 850px; width: 100%;"></iframe>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-1856870376900400592013-07-13T14:06:00.002-04:002013-07-13T14:06:35.476-04:00The wait is overFinally, in NX-OS 6.0(2) for Nexus 5000 platform Cisco implemented "default interface" command which lets you return interface to its factory default configuration. It is very-very-very useful feature in the lab environment when one has to do a lot of re-configuration and something does not work as expected simply because of left-over configuration from the previous test. <div>
This command has been available in IOS since 11.1 and in NX-OS for Nexus 7K since 5.1(1)<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-4115035198971633352013-05-29T10:49:00.000-04:002013-05-29T10:49:52.313-04:00Juniper haikuNo wonder Juniper software is so bloated :).<br />
<b>admin@switch> show version and haiku</b><br />
Hostname: switch<br />Model: ex4200-48t<br />JUNOS Base OS boot [10.4R12.1]<br />JUNOS Base OS Software Suite [10.4R12.1]<br />JUNOS Kernel Software Suite [10.4R12.1]<br />JUNOS Crypto Software Suite [10.4R12.1]<br />JUNOS Online Documentation [10.4R12.1]<br />JUNOS Enterprise Software Suite [10.4R12.1]<br />JUNOS Packet Forwarding Engine Enterprise Software Suite [10.4R12.1]<br />JUNOS Routing Software Suite [10.4R12.1]<br />JUNOS Web Management [10.4R12.1]<br /><br /><br /> Now that Zion's safe,<br /> can they find a better place<br /> to get some sweaters?<br /><br /><b>admin@switch> show version and haiku</b><br /><skip></skip><br />
Juniper babies<br /> The next generation starts<br /> Gotta get more sleep<br />
<br /><b>admin@switch> show version and haiku</b><br />
<skip><br /> 3am; darkness;<br /> Maintenance window closing.<br /> Safety net: rollback.</skip><br />
<br />
<b>admin@switch> show version and haiku</b> <br />
<skip><br /></skip>
Shiny leather pants<br /> Why don't they squeak when they kick?<br /> Is _that_ the secret?<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-15945726.post-38485787987870087842013-04-20T13:25:00.000-04:002013-04-20T13:25:15.900-04:00PACL and MAC address learningI have unenviable task to drag legacy application to 21st century. I am talking about 80-the legacy and some of its functions do not even use IP protocols. One of the proposed solution included 2 servers with the same<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipup1i9erbRR9C1V3F6YSJOk10nwIhs4rvLlBH38pbxiMagtZr8pDXRqC6qdODWuadDtDav8Krm137lofQ7ARaIZ8pqxBovSzelJ2ggWKFU0Bl4YLonvKfAiCWhaOFMG-UZI6E/s1600/Legacyapp.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipup1i9erbRR9C1V3F6YSJOk10nwIhs4rvLlBH38pbxiMagtZr8pDXRqC6qdODWuadDtDav8Krm137lofQ7ARaIZ8pqxBovSzelJ2ggWKFU0Bl4YLonvKfAiCWhaOFMG-UZI6E/s200/Legacyapp.png" width="200" /></a></div>
IP and MAC addresses (I know, but splitting network in 2 separate VLANs was not an option) connected to different switches, but in the same VLAN. ClientA and ClientB should be able to talk to each other and server connected to the same switch as client. ServerA and ServerB should not even know about each other's existence, so they won't complain about duplicate IP address. The switches are Cisco 6500s. One of the obvious solutions is to put Port Access Control List on either side of the inter-switch link. PACL successfully blocked the traffic between servers, but switches still learned MAC address of the blocked traffic source and placed it MAC address table. This would cause MAC address flapping on the switch every time clients send ARP query for server's MAC or when both servers need to send traffic to their clients. Why would switch need to keep MAC address of the discarded traffic? Oh, well. Another network mystery.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-53132515534644809972013-04-12T17:44:00.003-04:002013-04-12T17:44:57.981-04:00"private-vlan syncronize" pitfallLet's say you need to configure Private VLAN on Cisco Nexus switch with MSTP. You create new VLAN, make it isolated and map it to primary VLAN. Your newly created VLAN is automatically mapped to MST0 and if your primary happened to be in any other MST instance there is a possibility that your primary and secondary VLANs ended up with different L2 paths. As a bonus, you get <br />annoying message "These secondary vlans are not mapped to the same instance as their primary" every time you run "<b>show spanning-tree configuration</b>". Not a big deal, but things like these irritate me. Not to worry, "<b>private-vlan syncronize</b>" under "spanning-tree mst configuration" will automatically map all secondary VLANs to the same MST instance as primary VLAN. The moment it's done, you MST digest changes and boom, you have brand new MST region and STP convergence on top of it. So, either pick exiting VLAN mapped to the same instance and convert to secondary community or isolated VLAN, or have all your VLANs mapped to MST0 and use other methods to load-share traffic between transit links.<br />I know what you are thinking. No, I did it in the lab, so you won't have to.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-43991696384999747702013-03-05T09:09:00.000-05:002013-03-05T09:09:08.063-05:00How complex systems failA <a href="http://www.ctlab.org/documents/How%20Complex%20Systems%20Fail.pdf" target="_blank">must read paper</a> for every system architect. Also<a href="http://www.youtube.com/watch?v=2S0k12uZR14" target="_blank"> recording of the talk</a> given at Velocity conference by Richard Cook. The main take-away for me is: don't build reliable system, build resilient system.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-15945726.post-49182120642079065462013-02-11T12:33:00.001-05:002013-02-11T12:33:09.278-05:00Port-security side effectI discovered interesting side effect of configuring port-security which can prevent its deployment in certain circumstances. Let's have a look. Here is part of pertaining interface configuration:<br />
<br />
switchport port-security maximum 20<br /> switchport port-security<br /> switchport port-security aging time 1440<br /> switchport port-security violation restrict<br /> switchport port-security aging type inactivity<br />
<br />
You can find explanations of what each command does <a href="http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/port_sec.html" rel="nofollow" target="_blank">here</a>. <br />
<br />
<b>switch# show port-security interface gi 0/45</b><br />Port Security : Enabled<br />Port Status : Secure-up<br />Violation Mode : Restrict<br />Aging Time : 1440 mins<br />Aging Type : Inactivity<br />SecureStatic Address Aging : Disabled<br />Maximum MAC Addresses : 20<br />Total MAC Addresses : 1<br />Configured MAC Addresses : 0<br />Sticky MAC Addresses : 0<br />Last Source Address:Vlan : abcd.ef12.3456:1234<br />
Security Violation Count : 0<br />
<br />
In case you were wondering, the MAC address above is completely made up, but it associated with "floating" IP assigned to active device in cluster. Let's see what happens when active IP has to be moved to other device in cluster due to fail-over:<br />
<br />
<b>port_security-2-psecure_violation: security violation occurred, caused by mac address abcd.ef12.3456 on port gigabitethernet0/46. </b><br />
Oops. Even though port-security configuration allows to learn up to 20 MAC addresses and there are no MAC addresses on Gi0/46, we got port-security violation. Why? Let's see debug port-security:<br />
<br />
<br /> PSECURE: psecure_add_addr_check: Found <i>duplicate</i> mac-address abcd.ef12.3456, It is already secured on Gi0/45<br />
%PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address abcd.ef12.3456 on port GigabitEthernet0/46.<br />
PSECURE: Security violation, TrapCount:346<br />PSECURE: Read:2830, Write:2831<br />PSECURE: swidb = GigabitEthernet0/46 mac_addr = abcd.ef12.3456 vlanid = 1234<br />PSECURE: Adding abcd.ef12.3456 as dynamic on port Gi0/46 for vlan 1234<br />
PSECURE: Violation/duplicate detected upon receiving abcd.ef12.3456 on vlan 1234: port_num_addrs 1 port_max_addrs 20 vlan_addr_ct 1: vlan_addr_max 20 total_addrs 4: max_total_addrs 6144 <br />
<br />
Port-security violation happened because MAC address has not been deleted from original port yet, hence "duplicate mac-address" message. To mitigate, but not completely alleviate the problem, we can reduce aging timer to 1 minute minimum. It still means that in case of fail-over, the floating IP address will not be accessible for another minute, which could be 1 minute too long.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-32551241318449571542012-09-28T10:04:00.001-04:002012-09-28T10:04:50.087-04:00Nexus: peer-switch and STP Bridge IDWith release of NX-OS 5.2, Cisco started supporting peer-switch feature on Nexus 5K. When peer-switch is enabled, both VPC primary and secondary switches originate STP BPDUs on vPC ports and use the same designated bridge ID on vPC ports. This got me wandering what brige ID vPC primary switch uses when peer-switch is not enabled. I set up vPC switch-pair with downstream switch connected via vPC port-channel. The switches are running MST. Here is partial BPDU captured on downstream Nexus switch with command:<br />
<b>ethanalyzer local interface inbound-hi display-filter "stp" limit-captured-frames 20</b><br />
<br />
Spanning Tree Protocol<br />
Protocol Identifier: Spanning Tree Protocol (0x0000)<br />
Protocol Version Identifier: Multiple Spanning Tree (3)<br />
BPDU Type: Rapid/Multiple Spanning Tree (0x02)<br />
BPDU flags: 0x7c (Agreement, Forwarding, Learning, Port Role: Designated)<br />
Root Identifier: 8192 / 0 / 54:7f:ee:01:15:81<br />
Root Path Cost: 0<br />
Bridge Identifier: 8192 / 0 / <span style="background-color: yellow;">54:7f:ee:01:15:81</span><br />
Port identifier: 0x9063<br />
Message Age: 0<br />
Max Age: 20<br />
Hello Time: 2<br />
Forward Delay: 15<br />
Version 1 Length: 0<br />
Version 3 Length: 96<br />
MST Extension<br />
MST Config ID format selector: 0<br />
MST Config name: blp-mst-Region-1<br />
MST Config revision: 2<br />
MST Config digest: d7e7e4984e26acd301b955c5289031ad<br />
CIST Internal Root Path Cost: 0<br />
CIST Bridge Identifier: 8192 / 0 / <span style="background-color: lime;">00:23:04:ee:be:01</span><br />
CIST Bridge Priority: 8192<br />
CIST Bridge Identifier System ID Extension: 0<br />
CIST Bridge Identifier System ID: 00:23:04:ee:be:01<br />
CIST Remaining hops: 20<br />
MSTID 1, Regional Root Identifier 8192 / 54:7f:ee:01:15:81<br />
MSTID 2, Regional Root Identifier 8192 / 54:7f:ee:01:15:81<br />
<br />
Note "Bridge Identifier" and "CIST Bridge Identifier". They are different. The former is "vPC local system-mac" and latter is "vPC system-mac". They can be found in "show vpc role" output:<br />
<br />
nexus-primary# show vpc role <br />
<br />
vPC Role status<br />
----------------------------------------------------<br />
vPC role : primary <br />
Dual Active Detection Status : 0<br />
vPC system-mac : <span style="background-color: lime;">00:23:04:ee:be:01</span> <br />
vPC system-priority : 32667<br />
vPC local system-mac : <span style="background-color: yellow;">54:7f:ee:01:15:81</span> <br />
vPC local role-priority : 8192<br />
<br />
Here we can see, that without peer-switch enabled Nexus switch uses 2 different bridge IDs in the same BPDU. Why does it do it? I reached out to Cisco and will update when I hear anything.<br />
When peer-switch is enabled, both vPC primary and secondary switches originate BPDUs on vPC ports and "Bridge Identifier" and "CIST Bridge Identifier" are the same and equal to "vPC system-mac"Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-15945726.post-52323731797725986222012-02-21T22:54:00.002-05:002012-09-28T09:44:07.752-04:00IPExpert, it's 2012.After reading excellent sample chapter from IPExpert's "<a href="http://www.imakenews.com/ipexpert/e_article002338977.cfm?x=bksVhCk,bjy7P8v7,w">IPv4/IPv6 Multicast Operation and Troubleshooting</a>" book I decided to pre-order it. Today I got the pdf file and was very disappointed. The content is still great, but it can only be read on the PC or MAC - no iPad, Kindle or smartphone. Now, my commute is relatively long and I do most of my reading on the bus. Not be able to read documentation on mobile device is major problem for me. If greedy Hollywood studios found a way to provide their content on mobile platforms, so should IPExpert. Especially given the fact that INE provide PDF files DRM-free. This was my first and last purchase of IPExpert product.<br />
Update: FileOpen released iPad/iPhone app, so now I can read that PDF on my iPad. Unknownnoreply@blogger.com2