Cisco OTV (Part III)

July 6, 2011

This is the final follow-on post from OTV (Part I) and OTV (Part II).

In this final post I will go through the configuration steps, some outputs and FHRP isolation.

OTV Lab Setup

I setup a mini lab using two Nexus 7000 switches, each with the four VDCs, two Nexus 5000 switches and a 3750 catalyst switch.
I emulated two data center sites, each with two core switches for typical layer3 breakout, each with two switches dedicated for OTV and each with one access switch to test connectivity. Site1 includes switches 11-14 (four VDCs on N7K-1) and switch 15 (N5K), whereas Site2 includes switches 21-24 (four VDCs on N7K-2) and switch 32 (3750).

To focus on OTV, I removed the complexity from the transport network by using OTV on dedicated VDCs (four of them for redundancy), connected as inline OTV appliances and by connecting the OTV Join interfaces on a single multi-access network.

This is the topology:

Before configuring OTV, the decision must be made how OTV will be integrated part of the data center design.

Recall the OTV/SVI co-existing limitation. If core switches are in place, which are not the Nexus 7000 switches, OTV may be implemented natively on the new Nexus 7000 switch/es or using a VDCs. If the Nexus 7000 switches are providing the core switch functionality, then separate VDCs are required for OTV.

OTV Configuration

The intra site connectivity is using Cisco’s VPC technology as the MLAG (Multi-chassis Link Aggregation) protocol. The VPC configuration is not covered here. I will cover VPCs in more detail in future posts. The configuration steps covered will focus specifically on OTV.

Step-1: OTV Feature

Enable the OTV feature within the system VDC. OTV requires a license. The Transport_Services_PKG license could be purchased and loaded. Alternatively if just for testing or POC, enable the grace period which will allow a 120 day demo/grace period to use and test OTV.

N7K-11(config)# license grace-period
N7K-11(config)# feature otv
 N7K-11 %LICMGR-2-LOG_LIC_NO_LIC: No license(s) present for feature TRANSPORT_SERVICES_PKG.
 Application(s) shut down in 120 days. TRANSPORT_SERVICES_PKG license not installed.
 OTV feature will be shutdown after grace period of approximately 120 day(s)

Step-2: VLANs

Identify/Create the VLANs that should be extended between the sites across the overlay. Make sure all these VLANs are active.

N7K-13(config)# vlan 10-15
N7K-13(config-vlan)# name DATA-VLANS

Step-3: OTV Site VLAN

By default VLAN 1 will be used. It is recommended to use a dedicated VLAN. The allocated Site VLAN must not be extended across the overlay. As a result the same Site VLAN could be used at both sites. Even if only one OTV Edge Device, the Site VLAN must still be defined. Create a new VLAN to be used as the OTV Site VLAN and define this VLAN as the OTV Site VLAN.

N7K-13(config-vlan)# vlan 55
N7K-13(config-vlan)# name OTV-SITE-VLAN
N7K-13(config-vlan)# exit
N7K-13(config)# otv site-vlan 55

Step-4: Join Interfaces

Only one join interface can be specified per overlay on each device. Refer to the limitations here. Configure an IP address on each physical interface connected to the transport network. Enable IGMP v3 (required to join the SSM groups). Do not enable PIM on the Join interface. Test IP reachability between the Join interfaces.

N7K-13(config)#interface Ethernet2/9
N7K-13(config-int)#description otv-join-interface
N7K-13(config-int)#no switchport
N7K-13(config-int)#mtu 9216
N7K-13(config-int)#ip address
N7K-13(config-int)#ip igmp version 3
N7K-13(config-int)#no shutdown

Step-5: Internal interfaces

Configure the layer2 interfaces, that faces each site. These are the interfaces that will participate in STP and learning the MAC addresses from the local data center. I would recommended allowing only the relevant VLANs on these interfaces, i.e., the VLANs to be extended and the Site VLAN. Configure any further STP configurations if required.

N7K-13(config)# interface Ethernet2/10-11
N7K-13(config-int)#description otv-internal-interface
N7K-13(config-int)#mtu 9216
N7K-13(config-int)#switchport mode trunk
N7K-13(config-int)#switchport trunk allowed vlan 10-15,55
N7K-13(config-int)#channel-group 14 mode active
N7K-13(config-int)#no shutdown

Step-6: Overlay interface

Create the logical Overlay interface. Multiple Overlay interfaces can be used to allow different VLANs to use different paths in the transport network, but two conditions are required for this. The overlay number must match between sites and a VLAN can only be assigned to one overlay. Shutdown the Overlay interface before configuring it. Specify the OTV control and data multicast groups.

N7K-13(config)#interface Overlay1
N7K-13(config-if-overlay)#desc overlay-interface
N7K-13(config-if-overlay)#otv join-interface Ethernet1/9
N7K-13(config-if-overlay)#otv control-group
N7K-13(config-if-overlay)#otv data-group

Step-7: Extended VLANs

Add the VLANs to be transported across the overlay. Notice the Site VLAN is not extended.

N7K-13(config)#interface Overlay1
N7K-13(config-if-overlay)#otv extend-vlan 10-15

Step-8: MTU values

Configuring the MTU values within the data center sites is less important, but in the transport network the correct values are vital. Recall that the DF bit is set on all OTV packets leaving an Edge Device.

Transport-Devices(config)#interface e1/7
Transport-Devices(config-if-overlay)#mtu 9216

Step-9: Unshut the overlay interface

If there is only one OTV Edge Device present per site, this step is trivial. If there are two OTV Edge Devices per site for load-sharing purposes, for the sake of stability, bring one Edge Device up in each site, confirm connectivity is working before bringing up the redundant OTV devices.

N7K-13(config)#interface Overlay1
N7K-13(config-if-overlay)#no shut
N7K-23(config)#interface Overlay1
N7K-23(config-if-overlay)#no shut

Step-10: Test Connectivity

A simple ping from one site to another should be sufficient. This however can’t be done from the OTV Edge Devices, since they have no layer3 interfaces for the extended VLANs. At first it is normal for the first/second ECHO request to time out. This is the time it takes for the IP ARP request process to complete and allow OTV to advertise the newly learned MAC addresses from both sites. Subsequent PINGs should be fine.

N5K-15# ping
PING ( 56 data bytes
Request 0 timed out
64 bytes from icmp_seq=1 ttl=58 time=11.367 ms
64 bytes from icmp_seq=2 ttl=58 time=2.001 ms
64 bytes from icmp_seq=3 ttl=58 time=3.213 ms
64 bytes from icmp_seq=4 ttl=58 time=2.894 ms

--- ping statistics ---
5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 2.001/4.431/11.367 ms


Output and Verification

The first useful command to verify the state of the overlay is “show otv“.

n7k13# sh otv
OTV Overlay Information
Overlay interface Overlay1
 VPN name            : Overlay1
 VPN state           : UP
 Extended vlans      : 10-15 (Total:6)
 Control group       :
 Data group range(s) :
 Join interface(s)   : Eth1/9 (
 Site vlan           : 55 (up)

The next command shows the adjacencies formed between all Edge Devices and the state.

n7k13# sh otv adjacency
Overlay Adjacency database
Overlay-Interface Overlay1  :
Hostname                         System-ID      Dest Addr       Up Time   State
n7k14                            0026.9812.2244       02:36:05  UP
n7k23                            0026.9810.91c3       02:41:40  UP
n7k24                            0026.9810.91c4       02:35:33  UP

The next command is very useful. It shows the AED for each VLAN. Recall that the Edge Device with a lower System-ID will become authoritative for all the even extended VLANs, whereas the Edge Device with higher System-ID will be authoritative the odd extended VLANs. n7k13 system-ID was 0026.9812.2243 and n7k14 system-ID was 0026.9812.2244.

n7k14# sh otv vlan
OTV Extended VLANs and Edge Device State Information (* - AED)
VLAN   Auth. Edge Device                     Vlan State       Overlay
----   -----------------------------------   ----------       -------
  10   n7k13                                 inactive(Non AED)Overlay1
  11*  n7k14                                 active           Overlay1
  12   n7k13                                 inactive(Non AED)Overlay1
  13*  n7k14                                 active           Overlay1
  14   n7k13                                 inactive(Non AED)Overlay1
  15*  n7k14                                 active           Overlay1

To see the ARP-ND table use the next command:

n7k14# sh otv arp-nd-cache
OTV ARP/ND L3->L2 Address Mapping Cache
Overlay Interface Overlay1
VLAN  MAC Address         Layer-3 Address     Age       Expires In
15    001a.a1ff.7d46           00:03:42  00:04:17

The following command shows why the OTV is called MAC-in-IP routing:
To reach the first MAC address, the Next-Hop is an interface local within the site. To reach the last MAC address the Next-Hop in via the overlay as advertised by switch 24.

n7k14# sh otv route
OTV Unicast MAC Routing Table For Overlay1
VLAN MAC-Address     Metric  Uptime    Owner      Next-hop(s)
---- --------------  ------  --------  ---------  -----------
  15 000d.ecfe.077c  1       01:38:57  site       port-channel14
  15 0011.0000.0015  1       02:35:07  site       port-channel14
  15 0012.0000.0015  1       00:12:15  site       port-channel14
  15 001a.a1ff.7d46  42      01:44:44  overlay    n7k24
  15 0021.0000.0015  42      01:45:12  overlay    n7k24
  15 0022.0000.0015  42      00:12:16  overlay    n7k24

There are more commands with useful information, but I’m not going to show all examples here. Some other commands I found useful:

#sh otv site
#sh otv internal overlay
#sh otv internal adjacency
#sh tunnel internal implicit otv detail


FHRP Isolation

The last item I would like to cover before concluding this OTV trilogy post is FHRP Isolation. Let’s first look at why this is necessary.

I configured HSRP for VLAN-10 in both Site-1 (between switch 11 and 12) and Site-2 (between switch 21 and 22). Because it is a contiguous subnet all hosts in VLAN-10 has their gateway set to,  which is the same virtual gateway IP address in both sites. Now ideally traffic from Site-1 should exit via switch 11 and traffic from Site-2 should exit via switch 21 since both of these switches where setup with a HSRP priority of 105. But this is not what happens by default. Have a look at the output below:

On switch 11, the active HSRP device for VLAN-10 is switch 21. This would be the same on switch 12 and 22. Why would this be? Since both switch 11 and 21 have their HSRP priorities set to 105, the router with the higher interface IP address will be elected as the Active HSRP device. In this case switch 21. This means that all traffic towards the gateway of will be forward to Site-2. This is the problem that we will now correct. The next output shows the traffic from Site-1 is destined via the overlay. Not cool.

FHRP isolation is the act of filtering HSRP, VRRP or GLBP traffic from going across the overlay, and thereby forcing localized FHRP elections. There are two parts to filter.

  1. The election process should be contained within each site to elect a local active devices.
  2. The virtual MAC addresses will still be advertised, which would cause constant MAC move moves. I.e., local site, remote site, local site, etc.

Point number 1 is accomplished using a VLAN ACL on the OTV Edge Devices to filter the respective traffic depending on which FHRP protocol used. The example below show how to filter all of them:

ip access-list HSRPv1-IP
10 permit udp any eq 1985
ip access-list HSRPv2-IP
10 permit udp any eq 1985
ip access-list VRRP-IP
10 permit udp any
20 permit 112 any any
ip access-list GLBP-IP
10 permit udp any eq 3222
ip access-list ALL-IPs
10 permit ip any any
vlan access-map FHRP-FILTER 10
 match ip address HSRPv1-IP
 action drop
vlan access-map FHRP-FILTER 20
 match ip address HSRPv2-IP
 action drop
vlan access-map FHRP-FILTER 30
 match ip address VRRP-IP
 action drop
vlan access-map FHRP-FILTER 40
 match ip address GLBP-IP
 action drop
vlan access-map FHRP-FILTER 50
 match ip address ALL-IPs
 action forward
vlan filter FHRP-FILTER vlan-list 10-15

To prevent the virtual MAC addresses from causing MAC moves and allow for a cleaner design, an OTV route-map must be configured. This route-map must match the virtual MAC of the FHRP protocol used. For example HSRP v1 uses the virtual MAC: 0000.0c07.acXX where the last byte (XX) is the HSRP group number in HEX. Similar formats apply for VRRP and GLBP. The configuration below shows how to filter all FHRP protocols and should be applied on all OTV Edge Devices.

!! Filters HSRPv1 MAC addresses
mac-list OTV-FHRP-MAC seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00
!! Filters HSRPv2 MAC addresses
mac-list OTV-FHRP-MAC seq 11 deny 0000.0c9f.f000 ffff.ffff.ff00
!! Filters VRRP MAC addresses
mac-list OTV-FHRP-MAC seq 12 deny 0000.5e00.0100 ffff.ffff.ff00
!! Filters GLBP MAC addresses
mac-list OTV-FHRP-MAC seq 13 deny 0007.b400.0000 ffff.ff00.0000
mac-list OTV-FHRP-MAC seq 20 permit 0000.0000.0000 0000.0000.0000
route-map OTV-FHRP-FILTER permit 10
match mac-list OTV-FHRP-MAC
otv-isis default
  vpn Overlay1
    redistribute filter route-map OTV-FHRP-FILTER

Once the filters is applied, have a look at the same output as previously on both the intended FHRP breakout devices.


OTV Troubleshooting

Here is some good troubleshooting documents I found that might additionally be useful.

Troubleshooting OTV Adjacency
Troubleshooting ARP issues across OTV
OTV Silent Host Connectivity Problem



  1. […] OTV Enhancements, most notably the support for Unicast as the transport. Feel free to read my earlier OTV posts. […]

  2. Thank you, excellent series on OTV.

    One question, doesn’t OTV provide built-in filtering of FHRP?

    • Currently OTV does not, but the Cisco Documentation eludes to promises that it will be in future NX-OS releases.
      But on the versions available today, manual filtering is required. I tested both HSRP and GLBP. Definitely required.
      Nice blog btw.
      hth :)

  3. […] Three Articles on Routing-Bits about Cisco’s OTV(Overlay Transport Virtualization): Cisco OTV (Part I) Cisco OTV (Part II) Cisco OTV (Part III) […]

  4. Shouldn’t this:
    “mac-list OTV-FHRP-MAC seq 20 permit 0000.0000.0000 0000.0000.0000”

    Be this:
    “mac-list OTV-FHRP-MAC seq 20 permit 0000.0000.0000 ffff.ffff.ffff”?

    May be?

    • nope, that is correct. :)
      0 means any…

  5. […] Part III […]

  6. […] OTV Part III Share this:ShareEmailPrintFacebookTwitterDiggRedditStumbleUponLike this:LikeBe the first to like this post. […]

  7. Hi,

    I see in Cisco documentation (OTV best parctices) that we can use Cisco ASR in one Side and the N7k in the other side.
    Does the configuration is the same as when we configure N7k to N7k or ASR to ASR?
    If i configure ASR with the same configuration used when i have the same device in the other Side, does this configuration will work?

    Thanks for your help.

Please leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s