Smart Port-Channels

Consider the following output.

How is this possible, when no AAA or Privilege Profiles are configured? Have a look at the interface configuration:

Is this a bug/feature/annoyance. Depending on the platform, this is a feature. This test-interface is part of a port-channel. This is a common operational mistake. How many times has it happened in one of your data centers, where an engineer accidentally made a change to an interface which was a member of a port-channel, only to bring the port-channel and possibly any customer data that traversed the link down?

Continue reading “Smart Port-Channels” →

Cisco OTV (Part III)

This is the final follow-on post from OTV (Part I) and OTV (Part II).

In this final post I will go through the configuration steps, some outputs and FHRP isolation.

OTV Lab Setup

I setup a mini lab using two Nexus 7000 switches, each with the four VDCs, two Nexus 5000 switches and a 3750 catalyst switch.
I emulated two data center sites, each with two core switches for typical layer3 breakout, each with two switches dedicated for OTV and each with one access switch to test connectivity. Site1 includes switches 11-14 (four VDCs on N7K-1) and switch 15 (N5K), whereas Site2 includes switches 21-24 (four VDCs on N7K-2) and switch 32 (3750).

To focus on OTV, I removed the complexity from the transport network by using OTV on dedicated VDCs (four of them for redundancy), connected as inline OTV appliances and by connecting the OTV Join interfaces on a single multi-access network.

This is the topology:

Before configuring OTV, the decision must be made how OTV will be integrated part of the data center design.

Recall the OTV/SVI co-existing limitation. If core switches are in place, which are not the Nexus 7000 switches, OTV may be implemented natively on the new Nexus 7000 switch/es or using a VDCs. If the Nexus 7000 switches are providing the core switch functionality, then separate VDCs are required for OTV.

Continue reading “Cisco OTV (Part III)” →

Cisco OTV (Part II)

This is a follow on post from OTV (Part I).

STP Separation

Edge Devices do take part in STP by sending and receiving BPDUs on their internal interface as would any other layer2 switch.

But an OTV Edge Device will not originate or forward BPDUs on the overlay network. OTV thus limits the STP domain to the boundaries of each site. This means a STP problem in the control plane of a given site would not produce any effect on the remote data centers. This is one of the biggest benefits of OTV in comparison to other DCI technologies. This is made possible because MAC reachability information is advertised and learned via the control plane protocol instead of learned using typical MAC flooding behavior.

With the STP separation between sites, the ability for different sites to use different STP technologies is made possible with OTV. I.e., one site can run MSTP while another runs RSTP. In the real world this is a nifty enhancement.

Multi-Homing

OTV allows multiple Edge Devices to co-exist in the same site for load-sharing purposes. (With NX-OS 5.1 that is limited to 2 OTV Edge Devices per site.)

With multiple OTV Edge Devices per site and no STP across the overlay to shut down redundant links, the possibility of an end-to-end site loops are created. The absence of STP between sites holds valuable benefits, but a loop prevention mechanism is still required, so an alternative method was used. The boys who wrote OTV, decided on electing a master device responsible for traffic forwarding (similar to some non-STP protocols).

With OTV this master elected device is called an AED (Authoritative Edge Device).

An AED is an Edge Device that is responsible for forwarding the extended VLAN frames in and out of a site, from and to the overlay network. It is a very important to understand this before carrying on. Only the AED will forward traffic out of the site onto the overlay. With optimal traffic replication in a transport network, a site’s broadcast and multicast traffic will reach every Edge Device in the remote site. Only the AED in the remote site will forward traffic from the overlay into the remote site. The AED thus ensures that traffic crossing the site-overlay boundary does not get duplicated or create loops when a site is multi-homed.

Continue reading “Cisco OTV (Part II)” →

Cisco OTV (Part I)

OTV(Overlay Transport Virtualization) is a technology that provide layer2 extension capabilities between different data centers. In its most simplest form OTV is a new DCI (Data Center Interconnect) technology that routes MAC-based information by encapsulating traffic in normal IP packets for transit.

Cisco has submitted the IETF draft but it is not finalized yet. draft-hasmit-otv-01

OTV Overview

Traditional L2VPN technologies, like EoMPLS and VPLS, rely heavily on tunnels. Rather than creating stateful tunnels, OTV encapsulates layer2 traffic with an IP header and does not create any fixed tunnels.

OTV only requires IP connectivity between remote data center sites, which allows for the transport infrastructures to be layer2 based, layer3 based, or even label switched. IP connectivity as the base requirement along some additional connectivity requirements that will be covered in this post.

OTV requires no changes to existing data centers to work, but it is currently only supported on the Nexus 7000 series switches with M1-Series linecards.

A big enhancement OTV brings to the DCI realm, is its control plane functionality of advertising MAC reachability information instead of relying on the traditional data plane learning of MAC flooding. OTV refers to this concept as MAC routing, aka, MAC-in-IP routinig. The MAC-in-IP routing is done by encapsulating an ethernet frame in an IP packet before forwarded across the transport IP network. The action of encapsulating the traffic between the OTV devices, creates what is called an overlay between the data center sites. Think of an overlay as a logical multipoint bridged network between the sites.

OTV is deployed on devices at the edge of the data center sites, called OTV Edge Devices. These Edge Devices perform typical layer-2 learning and forwarding functions on their site facing interfaces (the Internal Interfaces) and perform IP-based virtualization functions on their core facing interface (the Join Interface) for traffic that is destined via the logical bridge interface between DC sites (the Overlay Interface).

Each Edge Device must have an IP address which is significant in the core/provider network for reachability, but is not required to have any IGP relationship with the core. This allows OTV to be inserted into any type of network in a much simpler fashion.

Lets look at some OTV terminology.

OTV Terminology

Continue reading “Cisco OTV (Part I)” →

Playtime

Its playtime. I am fortunate enough to have the following unboxed and at my disposal for some time.

It is two Cisco Nexus 7010 chassis, meant for a another new 10Gb DC coming online soon.
Each comprise of the following configuration:

2x SUP-1’s: First generation Supervisor.
3x FAB-1: Cross connect Fabric card module.
2x N7K-M132XP: M1-Series 32-Port 1/10Gb Ethernet Module, 80Gb Fabric.
1x N7K-M148GS: M1-Series 48-Port 1Gb Ethernet Modules, 46Gb Fabric.

The the other switches are:

2x Nexus 5010’s
2x Nexus 2224TP (Fabric Extender)
3x Catalyst 3750G
1x Lost Catalyst 2960.

Unfortunately I do not have a F1 series linecard, it would be interesting testing Cisco FabricPath, but I can test OTV (Overlay Transport Virtualization). The messy cable configuration was done for that exact purpose, to test OTV. ;D

So in the next couple days I will cover the theory, configuration, pro’s and con’s of using OTV as a DCI (Data Center Interconnect).

Follow @routingbits

RBAC with AAA Authentication

A earlier post introduced the Cisco Nexus concept of User Roles, which is a local command authorization method. There are some default system user roles.

RBAC (Role-Based Access Control) is the name/ability to create custom user roles locally on a Cisco Nexus. This gives the administrator the flexibility to define a group of certain commands to be allowed or denied for a selected role. Users can then be designated to belong to certain user roles. This designation can either be done locally on each switch or by using TACACS.

As discussed in the earlier post, AAA authorization and the user roles are mutually exclusive, since AAA Authorization overrides the permissions allowed with user roles. But using RBAC along with AAA Authentication (not Authorization), does bring some neat options to the table, depending obviously on a given network design and requirements.

How does RBAC work?

Custom user roles are defined by giving the role a name and by creating rules within the role. Each rule has a number, to decide the order in which the rules are applied. Rules are applied in descending order. I.e., rule 3 is applied before rule 2, which is applied before rule 1. This means a rule with a higher number overrides a rule with a lower number. Each role may have up to 256 rules configured. All the rules combined within a role determine what operations the role allows the associated user to perform.

Rules can be applied for the following parameters:

Command — A command or group of commands defined in a regular expression.
Feature — Commands that apply to a function provided by the Cisco Nexus switch.
Feature group — Default or user-defined group of features.

Continue reading “RBAC with AAA Authentication” →

Cisco 6500 Cosmetic bugs

Ever had this error before on a Cisco 6500 catalyst?

6500#  sh module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    5  Supervisor Engine 720 10GE (Active)    VS-S720-10G        SAL-------
  2   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX    SAL---------
  3   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       SAL----------

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  001d.45e1.ed48 to 001d.45e1.ed4f   2.0   8.5(2)       12.2(33)SXH1 Ok
  2  001f.9ec6.7d70 to 001f.9ec6.7d9f   1.6   8.4(1)       8.7(0.22)BUB Ok
  3  001b.d4ec.ab60 to 001b.d4ec.ab8f   1.12  12.2(14r)S5  12.2(33)SXH1 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Policy Feature Card 3       VS-F6K-PFC3C       SAL----------  1.0    Ok
  1  MSFC3 Daughterboard         VS-F6K-MSFC3       SAL----------  1.0    Ok
  3  Centralized Forwarding Card WS-F6700-CFC        SAL----------  3.1    Ok

Mod  Online Diag Status
---- -------------------
  1  Minor Error
  2  Pass
  3  Pass

Continue reading “Cisco 6500 Cosmetic bugs” →

Cisco Nexus User Roles

IOS relies on privilege levels. Privilege levels (0-15) defines locally what level of access a user has when logged into an IOS device, i.e. what commands are permitted. This only applies in the absence of AAA being configured. There are 3 default privilege levels on IOS, but really only two that are relevant:

Privilege Level 1 — Normal level on Telnet; includes all user-level commands at the router> prompt.
Privilege Level 15 — Includes all enable-level commands at the router# prompt.

NX-OS uses a different concept for the same purpose, known as User Roles. User Roles contain rules that define the operations allowed for a particular user assigned to a role. There are default User Roles:

Network-Admin—Complete read-and-write access to the entire NX-OS device (only available in the default VDC).
Network-Operator—Complete read access to the entire NX-OS device (Default User Role).
VDC-Admin—Read-and-write access limited to a VDC (VDCs are not yet available on Nexus 5000).
VDC-Operator—Read access limited to a VDC (Default User Role).

A VDC (Virtual Device Context) is a logical separation of control plane hardware resources into virtualized layer3 switches. Don’t worry to much about what a VDC is for now, it is not really relevant to the purpose of this post.

When a NX-OS device is setup for the first time, during the first login, a Network-Admin account must be specified and subsequently be used to login. Arguably a bit more secure that IOS. Any additional users created locally after that will by default receive the User Role “Network-Operator“, unless specified differently:

User Roles are local to a switch and only relevant in the absence of AAA Authorization being configured. To see the permissions of a particular User Role use:

N5K-2# sh role name network-operator
Role: network-operator
  Description: Predefined network operator role has access to all read
  commands on the switch
  -------------------------------------------------------------------
  Rule    Perm    Type        Scope               Entity
  -------------------------------------------------------------------
  1       permit  read

Continue reading “Cisco Nexus User Roles” →

Nexus’ improved CLI

The Cisco Nexus Series platform has some good things going. Having spent much of my time recently using them, I have come to appreciate some very neat improvements NX-OS is offering over standard IOS. For the most part driving NX-OS is very similar to IOS, but it’s been greatly improved.

One such example is the output from the most used IOS command “show ip int brief”, which on NX-OS only shows ‘IP’ (being layer 3) interfaces. To see the brief state of all types of interfaces use “sh int brief” instead.

N5K-2(config)# sh ip int brief
IP Interface Status for VRF "default"(1)
Interface            IP Address      Interface Status
Vlan19               10.1.19.6       protocol-up/link-up/admin-up
Vlan22               10.1.22.6       protocol-up/link-up/admin-up

N5K-2(config)# sh int brief
--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        1      eth  trunk  up      none                       1000(D) 51
Eth1/2        22     eth  access up      none                        10G(D) -
Eth1/3        1      eth  trunk  down    SFP not inserted            10G(D) 50
Eth1/4        1      eth  trunk  down    SFP not inserted            10G(D) 50
Eth1/5        1      eth  trunk  down    SFP not inserted            10G(D) -
Eth1/6        19     eth  access down    SFP not inserted            10G(D) -
Eth1/7        1      eth  trunk  down    Link not connected          10G(D) 5
Eth1/8        1      eth  trunk  down    Link not connected          10G(D) 5
Eth1/9        1      eth  fabric down    Administratively down       10G(D) 9
Eth1/10       1      eth  fabric down    FEX identity mismatch       10G(D) 7
Eth1/11       1      eth  fabric down    vpc peerlink is down        10G(D) 34
Eth1/12       1      eth  fabric down    SFP not inserted            10G(D) 12
Eth1/13       1      eth  fabric up      none                        10G(D) 15
Eth1/14       1      eth  fabric down    Administratively down       10G(D) 9

Continue reading “Nexus’ improved CLI” →

Jumbo MTU on Nexus 5000

Setting a per interface MTU (maximum transmission unit) is not supported on the Nexus 5000/2000 series switches.
If a Jumbo packet is required to traverse a Nexus 5000 series switch , the jumbo MTU must be set in a policy-map and applied to the ‘Sytem QOS’.

Configuration:

Configuration, PRE NX-OS 4.1:
policy-map JUMBO
 class class-default
  mtu 9216
system qos
 service-policy JUMBO

Configuration with POST NX-OS 4.1:
policy-map type network-qos JUMBO
 class type network-qos class-default
  mtu 9216
system qos
 service-policy type network-qos JUMBO

Continue reading “Jumbo MTU on Nexus 5000” →

Uptime

Really sad when you have to reboot a production switch that’s been up for this long. Suppose another question is why was the switch never upgraded? Until now not needed. :)

bry-asw1&gt;show version
Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(9)EA1, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Wed 24-Apr-02 06:57 by antonino
Image text-base: 0x80010000, data-base: 0x804E8000
ROM: Bootstrap program is CALHOUN boot loader
bry-asw1 uptime is 7 years, 48 weeks, 6 days, 6 hours, 19 minutes
System returned to ROM by power-on
System restarted at 12:00:24 SAST Thu Feb 13 2003
System image file is "flash:/c2950-i6q4l2-mz.121-9.EA1.bin"
cisco WS-C2950G-24-EI (RC32300) processor (revision D0) with 20815K bytes of memory.
Processor board ID FOC0633Y2T5
....

What is the longest your production devices have been up for?

Nexus defaults to PAP authentication

Ever configured a Nexus switch to use AAA to query a Tacacs+ server? Had some troubles applying standard IOS config to NX-OS?

Possibly if your Tacacs+ server is configured to only allow PAM (Password Authentication Manager) authentication for the users. See when a NX-OS switch sends a AAA authentication packet, by default it is encapsulated using PAP encoding. This is in contrast to normal IOS devices, that use PAM encoding by default.

To illustrate I used the following config:

ip tacacs source-interface mgmt0
tacacs-server host 10.5.0.82 key password
!
aaa group server tacacs+ TAC
server 10.5.0.82
use-vrf management
source-interface mgmt0
!
aaa authentication login default group TAC
aaa authorization config-commands default group TAC
aaa authorization commands default group TAC
aaa accounting default group TAC

Continue reading “Nexus defaults to PAP authentication” →

Troubleshooting random Nexus reboots

November last year, a pair of Cisco Nexus 5010 switches, suddenly started rebooting randomly without user intervention. Since these boxes were a front to a VM environment, stability were of urgent concern. But in order to stabilize the environment, the root cause of the reboots had to be isolated, and quickly.

The Cisco Nexus platform might not be as mature as many would like, but it is quickly becoming a very needed switch in Next-Generation datacenters. Of the things I like most about the Nexus boxes are the readily available local reporting and intuitive system checks. Obviously there are many other features which is making the platform so popular. I’ll cover some of these in time.

Coming back to the rebooting issue. Unlike IOS devices that looses all local logging info, unless a crash dump was saved to NVRAM, the Nexus writes most of its log information to disk. Thus even after the reboot, you have all the information.
Continue reading “Troubleshooting random Nexus reboots” →

Using the iPhone for Out-of-Band access

I frequently use my iPad to console onto routers as per my earlier post. But there are so much more functionality here. The iPhone can be used as a Out-of-Band device.

Why? Because it occasionally happens that a router has no device near it that can provide console access. And if you doing risky changes, this beats having to sit next to the device while doing the changes.

Requirements:

A serial connector cable (30-pin Apple to male DB9 pin RS-232).
A rollover cable.
A jailbroken iPhone.
Terminal application.
Software that supports serial communication.
Inbound connectivity to iPhone Sim.

Steps 1-5 is the same as my previous post. Only difference is with step-4. The app iSSH is not needed here as the SSH connection will not be made locally from the device. So once SSH is loaded via Cydia move along to Step-5.

The last step required is having inbound access to the cellular data IP on your iPhone. This varies between cellular providers. Some providers block inbound access, others allow it by default. If your cellular provider is blocking inbound access, you will have to request them to allow it for you SIM.

All that is left to do, is plugging your phone into the distant router, (preferably locked in the cabinet, to prevent it from being stolen). From you desk SSH to the iPhone and use Minicom to reverse console into your router.

EIGRP adjacency using a secondary IP

Consider the following statement from Cisco.com : “Routers do not form EIGRP neighbors over secondary networks.”

A Routing-BitsHandbook candidate queried this last week, claiming the statement is misleading and that EIGRP will indeed form an adjacency using a secondary IP address under specific conditions.

Consider the following configuration. R1 connects to R2 using a back-to-back serial connection. Both S1/1 interfaces have a primary and a secondary IP address defined. The EIGRP processes only matches the secondary IP addresses.

R1#
interface Serial1/1
ip address 10.1.1.1 255.255.255.0 secondary
ip address 10.5.1.1 255.255.255.0
!
router eigrp 1
network 10.1.1.1 0.0.0.0
no auto-summary

R2#
interface Serial1/1
ip address 10.0.1.2 255.255.255.0 secondary
ip address 10.5.1.2 255.255.255.0
!
router eigrp 1
network 10.0.1.2 0.0.0.0
no auto-summary

So what do you think will happen in this scenario? Will R1 and R2 become adjacent? Cisco explicitly mentions that a secondary IP address is not used in the EIGRP hello packets, therefore EIGRP neighbors will not become adjacent using secondary IP addresses.
Continue reading “EIGRP adjacency using a secondary IP” →