Val:~$ whoami

I am Val Glinskiy, network engineer specializing in data center networks. TIME magazine selected me as Person of the Year in 2006.

Search This Blog

Thursday, December 14, 2006

How to generate SYN-flood, linux style

Warning: malicious use of SYN-flood is punishable by law.
I needed to check SYN-Defense (TM) feature of Foundry's SI450. In order to do so I had to generate SYN-flood some how. It can be done with hping

hping --faster -S <dst. host>

The problem is that when SI replies with SYN/ACK packet back to my server, it sends RST in response, hping is not expecting any replies back. And ServerIron terminates the session immediately not letting SYN-Defense feature to kick-in. I had to block outgoing packets with RST flag set and destined to the dst. host above. Iptables to the rescue:

iptables -A OUTPUT -p tcp -d <dst. host> -m tcp --tcp-flags SYN,ACK,FIN,RST RST -j DROP

Enjoy

Monday, October 30, 2006

Centralized TACACS+ management

I've got TACACS+ servers running in every location where I have network equipment. That's fine, until I have to add a user or change password. Then I have to log into every server, edit tacacs.conf file and restart tacacs daemon. Here is what I've done.
1. Install Open LDAP server in every location.

2. Configure replication:

In master LDAP server's slapd.conf:


replica uri=ldap://10.10.10.2:389
binddn="cn=admin,dc=example,dc=com"
bindmethod=simple credentials=SecretPassword


for every replica

in replca LDAP server's slapd.conf:

updatedn "cn=admin,dc=example,dc=com"
updateref ldap://10.10.10.1

Note, that updatedn must match binddn and must have read/write access to LDAP database. See Open LDAP documentation for details.

3. Compile TACACS+ with LDAP support. If you run Debian, simply run "apt-get install tac-plus". It's already compiled with LDAP support.

in tacacs.conf file put

default authentication = ldap "ldap://localhost:389/dc=example,dc=com"

and restart tacacs.
replace "dc=example,dc=com" with your BaseDN

You can use phpLDAPadmin to manage master LDAP server.
Now, if I want to add new user or change my password, I do it on master LDAP server.

For enable password create user $enab15$ in LDAP server. Beware, LDIF format treats dollar sign as line separator.

Wednesday, October 25, 2006

Find a device on Cisco switch

I have a few dozens of Cisco switches running CatOS and IOS and from time to time I need to find the port on the switch where particular network device is connected to. Sometimes I know the MAC address and sometimes I know IP address only. Logging into every switch and running "show cam" or "show mac-address-table" is time consuming, but all my switches have SNMP on. I wrote a script which polls list of switches to find the device either by MAC or IP address.

get_mac_port.pl ([-m <mac-address>]| [-i <ip address> -r <router>]) -f <file> -c <community> [-hV]
Where:
<file> - list of switches (IP or names) in a text file, one per line.
<mac-address> - MAC address of the host. It should be in form of aa:23:cc:dd:ee:ff or aabb.ccdd.eeff
Both lower and upper case letters accepted.
<ip address> - IP address of the host.
<router> - IP address or name of the default gateway for the host's network. This is the best source to get host's MAC address from.
<community> - SNMP community string.

Known problem: If switches are interconnected via trunks, the script will report the device as connected to a trunk port.
Download

Wednesday, September 27, 2006

More Netscreen's OID

netscreen to get name of VPN tunnels
[mrtg@monitor1 etc]$ snmpwalk -v2c -c public 10.44.222.248 1.3.6.1.4.1.3224.4.1.1.1.4
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.0 = STRING: "MCI-PSN"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.1 = STRING: "MCI-EVL"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.2 = STRING: "MCI-NAT"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.3 = STRING: "ten43-ten48"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.4 = STRING: "MCI-DUB"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.5 = STRING: "MCI-EWR"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.6 = STRING: "MCI-CN"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.7 = STRING: "MCI-CITY"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.8 = STRING: "MCI-AN"
SNMPv2-SMI::enterprises.3224.4.1.1.1.4.9 = STRING: "MCI-Village"

netscreen to get remote VPN tunnel destination
[mrtg@monitor1 etc]$ snmpwalk -v2c -c public 10.44.222.248 1.3.6.1.4.1.3224.4.1.1.1.13
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.0 = IpAddress: 10.1.1.2
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.1 = IpAddress: 10.5.5.254
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.2 = IpAddress: z.z.z.125
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.3 = IpAddress: r.r.r.173
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.4 = IpAddress: t.t.t.193
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.5 = IpAddress: u.u.u.5
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.6 = IpAddress: f.f.f.98
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.7 = IpAddress: q.q.q.2
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.8 = IpAddress: c.c.c.5
SNMPv2-SMI::enterprises.3224.4.1.1.1.13.9 = IpAddress: 10.11.11.66

netscreen to get VPN tunnle inbytes counter
[mrtg@monitor1 etc]$ snmpwalk -v2c -c public 10.44.222.248 1.3.6.1.4.1.3224.4.1.1.1.35
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.0 = Counter32: 3574013013
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.1 = Counter32: 3615791047
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.2 = Counter32: 12630018
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.3 = Counter32: 67249688
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.4 = Counter32: 337567075
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.5 = Counter32: 3816042333
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.6 = Counter32: 108651803
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.7 = Counter32: 68491303
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.8 = Counter32: 67229052
SNMPv2-SMI::enterprises.3224.4.1.1.1.35.9 = Counter32: 208267531

netscreen to get VPN tunnel outbytes counter
[mrtg@monitor1 etc]$ snmpwalk -v2c -c public 10.44.222.248 1.3.6.1.4.1.3224.4.1.1.1.36
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.0 = Counter32: 4198953760
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.1 = Counter32: 566033608
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.2 = Counter32: 2140626864
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.3 = Counter32: 159163576
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.4 = Counter32: 2146079992
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.5 = Counter32: 855445376
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.6 = Counter32: 682270400
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.7 = Counter32: 205882792
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.8 = Counter32: 159275584
SNMPv2-SMI::enterprises.3224.4.1.1.1.36.9 = Counter32: 2795282320

In input/output packet counters OIDs last number corresponds to last number in VPN tunnel destination and name OIDs. So, MRTG config for input/output traffic through VPN tunnel "MCI-CITY" will look like this

Target[fw]: 1.3.6.1.4.1.3224.4.1.1.1.35.7&1.3.6.1.4.1.3224.4.1.1.1.36.7:public@10.44.222.248:::::2

Tunnel status OID 1.3.6.1.4.1.3224.4.1.1.1.20

Monday, August 07, 2006

Foundry ServerIron GSLB


gslb protocol
gslb policy
metric-order set health-check weighted-site preference
weighted-site
preference
dns active-only
dns best-only
dns override
dns cname-detect

gslb-host-policy testwww
metric-order set health-check weighted-site preference
weighted-site
preference
dns active-only
dns best-only

gslb site sanjose
weight 100
si vsj-lb1 10.60.0.66 0
gslb site newark
si nwklb01-a 10.50.30.208 250

gslb dns zone jabodo.com
host-info www http
host-info www gslb-policy testwww



gslb log-dns

Thursday, July 13, 2006

MRTG and CPU usage

Juniper Netscreen CPU utilization OID 1.3.6.1.4.1.3224.16.1.4.0

Target[10.50.28.5_cpu]:.1.3.6.1.4.1.3224.16.1.4.0&.1.3.6.1.4.1.3224.16.1.4.0:public@10.50.28.5:
SetEnv[10.50.28.5_cpu]: MRTG_INT_IP="10.50.28.5"
MaxBytes[10.50.28.5_cpu]: 100
Title[10.50.28.5_cpu]: CPU Utilization
PageTop[10.50.28.5_cpu]: CPU Utilization -- netscreen25
Options[10.50.28.5_cpu]: growright,gauge,integer,nopercent
Unscaled[10.50.28.5_cpu]: dwym
YLegend[10.50.28.5_cpu]: CPU load
ShortLegend[10.50.28.5_cpu]: CPU Load
Legend1[10.50.28.5_cpu]: CPU Load
Legend2[10.50.28.5_cpu]: CPU Load
LegendO[10.50.28.5_cpu]:

Foundry's CPU utilization OID 1.3.6.1.4.1.1991.1.1.2.11.1.1.4
There is a catch - Foundry reports utilization in 1/100 of the procent, so number 820 means 8.2%. I have Fondry ServerIron 450 with 1 management CPU and 3 CPUs on Web Switching module and wanted to know 5 min. average for management CPU:

Target[10.50.254.5_cpu]: 1.3.6.1.4.1.1991.1.1.2.11.1.1.4.1.1.300&1.3.6.1.4.1.1991.1.1.2.11.1.1.4.1.1.300:public@10.50.254.5:::::2
SetEnv[10.50.254.5_cpu]: MRTG_INT_IP="10.50.254.5"
MaxBytes[10.50.254.5_cpu]: 10000
Title[10.50.254.5_cpu]: SI450 Management CPU Utilization
PageTop[10.50.254.5_cpu]: Management CPU Utilization -- nwklb01-a
Options[10.50.254.5_cpu]: growright,gauge,integer
#Unscaled[10.50.254.5_cpu]: dwym
YLegend[10.50.254.5_cpu]: 1/100 percent
ShortLegend[10.50.254.5_cpu]: CPU Load
Legend1[10.50.254.5_cpu]: Mangemnt CPU 5min average
LegendO[10.50.254.5_cpu]:

OIDs for WSM CPUs 1-3: 1.3.6.1.4.1.1991.1.1.2.11.1.1.4.[2-4].300

PS: my community string is not really "public"

Thursday, June 22, 2006

Nagios BGP plugin

I converted the script I wrote to monitor BGP on Cisco routers via SNMP into Nagios plugin. I did not like check_bgp plugin which comes with Nagios, because it requires username and password to access a router to be stored in plain text.

Here is how to configure Nagios to use my plugin:
copy check_bgp_snmp.pl into nagios/libexec directory. Then, add the following to Nagios config

define command{       
command_name    check_bgp_snmp
command_line    $USER1$/check_bgp_snmp.pl -r $HOSTADDRESS$ -c $ARG1$
}



define service{
host_name               router1
service_description     BGP CHECK
check_command           check_bgp_snmp!public
max_check_attempts      2
normal_check_interval   5
retry_check_interval    2
check_period            24x7
notification_interval   10
notification_period     24x7
notification_options    w,u,c,r
notifications_enabled   0
contact_groups          admins
}

Download check_bgp_snmp.pl

Wednesday, June 14, 2006

more BGP monitor changes

I added "send alarm" feautre and changed the way it checks router. First, it checks if BGP session with neighbor is established. If not, it proceeds to verify that neighbor is not administratively down and sends e-mail. But if BGP session is established, it checks whether router receives prefixes from that neighbor.
Let me know if you find any bugs.

Download bgpmonitor.pl

Monday, June 12, 2006

BGP monitor changes

I made some changes to the script. The output format is different and it also retreives number of prefixes received from neighbors.
nas-server:~$ ./bgpmonitor.pl router1

=============================
Router: router1  AS  64512
Neighbor                Status          PfxRcd
192.168.1.1           established     2841107
172.16.1.1            established     3445788
10.0.0.2              established     2886739
10.0.0.3              established     3



Download bgpmonitor.pl

Friday, June 09, 2006

BGP monitor

Our ISP had maintenance couple of nights ago and failed to tell us. That maintenace resulted in 2 out of 4 our BGP neighbors going down. That event did not trigger monitor alarms because physical links stayed up. So, I needed to be able to receive information about BGP session state without logging into Cisco router and running "show ip bgp summary" command. Here is the output of this command:
router2#sho ip bgp summary
BGP router identifier 10.0.0.1, local AS number 64512
BGP table version is 14738877, main routing table version 14738877
185644 network entries using 18750044 bytes of memory
556906 path entries using 26731488 bytes of memory
67493 BGP path attribute entries using 3779888 bytes of memory
2 BGP rrinfo entries using 48 bytes of memory
30185 BGP AS-PATH entries using 774560 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
3 BGP filter-list cache entries using 36 bytes of memory
BGP using 50036064 total bytes of memory
BGP activity 870972/685328 prefixes, 6757346/6200440 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.2        4 64512 3358195 3465622 14738877    0    0 11w2d      185628
10.0.0.3        4 64512  114502  114499 14738877    0    0 11w2d           3
192.168.1.1     4 65535 3005050  110420 14738846    0    0 13:02:41   185636
172.16.1.1      4 65535 2987510  114513 14738846    0    0 10w3d      185636


Note, that I replaced AS numbers and IP addresses with fake ones.
I wanted to see state of BPG neighbors only. At first I wanted to write expect script to log into routers and run show ip bgp summary, but there are obvious security implications. I needed to store password somewhere. Another solutions is to wrap snmpwalk command into shell script and parse the output. I decided to shake off some rust of my perl skills and do it in perl.
Download bpgmonitor.pl
Usage example

nas-server:~$ ./bgpmonitor.pl router2

=============================
Router: router2  AS 64512 
Neighbor:       10.0.0.2       Status:        established
Neighbor:       10.0.0.3       Status:        active
Neighbor:       192.168.1.1    Status:        established
Neighbor:       172.16.1.1     Status:        established


Next version of the script will send alarms whenever neighbor status is not "establised" and bgp session is not administratively down.
Thank O'Reilly book for helping me figuring out hash dereferencing.