Cisco OTV (Part III)

This is the final follow-on post from OTV (Part I) and OTV (Part II).

In this final post I will go through the configuration steps, some outputs and FHRP isolation.

OTV Lab Setup

I setup a mini lab using two Nexus 7000 switches, each with the four VDCs, two Nexus 5000 switches and a 3750 catalyst switch.
I emulated two data center sites, each with two core switches for typical layer3 breakout, each with two switches dedicated for OTV and each with one access switch to test connectivity. Site1 includes switches 11-14 (four VDCs on N7K-1) and switch 15 (N5K), whereas Site2 includes switches 21-24 (four VDCs on N7K-2) and switch 32 (3750).

To focus on OTV, I removed the complexity from the transport network by using OTV on dedicated VDCs (four of them for redundancy), connected as inline OTV appliances and by connecting the OTV Join interfaces on a single multi-access network.

This is the topology:

Before configuring OTV, the decision must be made how OTV will be integrated part of the data center design.

Recall the OTV/SVI co-existing limitation. If core switches are in place, which are not the Nexus 7000 switches, OTV may be implemented natively on the new Nexus 7000 switch/es or using a VDCs. If the Nexus 7000 switches are providing the core switch functionality, then separate VDCs are required for OTV.

OTV Configuration

The intra site connectivity is using Cisco’s VPC technology as the MLAG (Multi-chassis Link Aggregation) protocol. The VPC configuration is not covered here. I will cover VPCs in more detail in future posts. The configuration steps covered will focus specifically on OTV.

Step-1: OTV Feature

Enable the OTV feature within the system VDC. OTV requires a license. The Transport_Services_PKG license could be purchased and loaded. Alternatively if just for testing or POC, enable the grace period which will allow a 120 day demo/grace period to use and test OTV.

N7K-11(config)# license grace-period
N7K-11(config)# feature otv
 N7K-11 %LICMGR-2-LOG_LIC_NO_LIC: No license(s) present for feature TRANSPORT_SERVICES_PKG.
 Application(s) shut down in 120 days. TRANSPORT_SERVICES_PKG license not installed.
 OTV feature will be shutdown after grace period of approximately 120 day(s)

Step-2: VLANs

Identify/Create the VLANs that should be extended between the sites across the overlay. Make sure all these VLANs are active.

N7K-13(config)# vlan 10-15
N7K-13(config-vlan)# name DATA-VLANS

Step-3: OTV Site VLAN

By default VLAN 1 will be used. It is recommended to use a dedicated VLAN. The allocated Site VLAN must not be extended across the overlay. As a result the same Site VLAN could be used at both sites. Even if only one OTV Edge Device, the Site VLAN must still be defined. Create a new VLAN to be used as the OTV Site VLAN and define this VLAN as the OTV Site VLAN.

N7K-13(config-vlan)# vlan 55
N7K-13(config-vlan)# name OTV-SITE-VLAN
N7K-13(config-vlan)# exit
N7K-13(config)# otv site-vlan 55

Step-4: Join Interfaces

Only one join interface can be specified per overlay on each device. Refer to the limitations here. Configure an IP address on each physical interface connected to the transport network. Enable IGMP v3 (required to join the SSM groups). Do not enable PIM on the Join interface. Test IP reachability between the Join interfaces.

N7K-13(config)#interface Ethernet2/9
N7K-13(config-int)#description otv-join-interface
N7K-13(config-int)#no switchport
N7K-13(config-int)#mtu 9216
N7K-13(config-int)#ip address 55.1.1.13/24
N7K-13(config-int)#ip igmp version 3
N7K-13(config-int)#no shutdown

Step-5: Internal interfaces

Configure the layer2 interfaces, that faces each site. These are the interfaces that will participate in STP and learning the MAC addresses from the local data center. I would recommended allowing only the relevant VLANs on these interfaces, i.e., the VLANs to be extended and the Site VLAN. Configure any further STP configurations if required.

N7K-13(config)# interface Ethernet2/10-11
N7K-13(config-int)#description otv-internal-interface
N7K-13(config-int)#mtu 9216
N7K-13(config-int)#switchport
N7K-13(config-int)#switchport mode trunk
N7K-13(config-int)#switchport trunk allowed vlan 10-15,55
N7K-13(config-int)#channel-group 14 mode active
N7K-13(config-int)#no shutdown

Step-6: Overlay interface

Create the logical Overlay interface. Multiple Overlay interfaces can be used to allow different VLANs to use different paths in the transport network, but two conditions are required for this. The overlay number must match between sites and a VLAN can only be assigned to one overlay. Shutdown the Overlay interface before configuring it. Specify the OTV control and data multicast groups.

N7K-13(config)#interface Overlay1
N7K-13(config-if-overlay)#desc overlay-interface
N7K-13(config-if-overlay)#shutdown
N7K-13(config-if-overlay)#otv join-interface Ethernet1/9
N7K-13(config-if-overlay)#otv control-group 239.12.34.5
N7K-13(config-if-overlay)#otv data-group 232.5.5.0/28

Step-7: Extended VLANs

Add the VLANs to be transported across the overlay. Notice the Site VLAN is not extended.

N7K-13(config)#interface Overlay1
N7K-13(config-if-overlay)#otv extend-vlan 10-15

Step-8: MTU values

Configuring the MTU values within the data center sites is less important, but in the transport network the correct values are vital. Recall that the DF bit is set on all OTV packets leaving an Edge Device.

Transport-Devices(config)#interface e1/7
Transport-Devices(config-if-overlay)#mtu 9216

Step-9: Unshut the overlay interface

If there is only one OTV Edge Device present per site, this step is trivial. If there are two OTV Edge Devices per site for load-sharing purposes, for the sake of stability, bring one Edge Device up in each site, confirm connectivity is working before bringing up the redundant OTV devices.

N7K-13(config)#interface Overlay1
N7K-13(config-if-overlay)#no shut
!
N7K-23(config)#interface Overlay1
N7K-23(config-if-overlay)#no shut

Step-10: Test Connectivity

A simple ping from one site to another should be sufficient. This however can’t be done from the OTV Edge Devices, since they have no layer3 interfaces for the extended VLANs. At first it is normal for the first/second ECHO request to time out. This is the time it takes for the IP ARP request process to complete and allow OTV to advertise the newly learned MAC addresses from both sites. Subsequent PINGs should be fine.

N5K-15# ping 15.1.1.32
PING 15.1.1.32 (15.1.1.32): 56 data bytes
Request 0 timed out
64 bytes from 15.1.1.32: icmp_seq=1 ttl=58 time=11.367 ms
64 bytes from 15.1.1.32: icmp_seq=2 ttl=58 time=2.001 ms
64 bytes from 15.1.1.32: icmp_seq=3 ttl=58 time=3.213 ms
64 bytes from 15.1.1.32: icmp_seq=4 ttl=58 time=2.894 ms

--- 15.1.1.32 ping statistics ---
5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 2.001/4.431/11.367 ms

.

Output and Verification

The first useful command to verify the state of the overlay is “show otv“.

n7k13# sh otv
OTV Overlay Information
Overlay interface Overlay1
 VPN name            : Overlay1
 VPN state           : UP
 Extended vlans      : 10-15 (Total:6)
 Control group       : 239.12.34.5
 Data group range(s) : 232.5.5.0/28
 Join interface(s)   : Eth1/9 (55.1.1.13)
 Site vlan           : 55 (up)

The next command shows the adjacencies formed between all Edge Devices and the state.

n7k13# sh otv adjacency
Overlay Adjacency database
Overlay-Interface Overlay1  :
Hostname                         System-ID      Dest Addr       Up Time   State
n7k14                            0026.9812.2244 55.1.1.14       02:36:05  UP
n7k23                            0026.9810.91c3 55.1.1.23       02:41:40  UP
n7k24                            0026.9810.91c4 55.1.1.24       02:35:33  UP

The next command is very useful. It shows the AED for each VLAN. Recall that the Edge Device with a lower System-ID will become authoritative for all the even extended VLANs, whereas the Edge Device with higher System-ID will be authoritative the odd extended VLANs. n7k13 system-ID was 0026.9812.2243 and n7k14 system-ID was 0026.9812.2244.

n7k14# sh otv vlan
OTV Extended VLANs and Edge Device State Information (* - AED)
VLAN   Auth. Edge Device                     Vlan State       Overlay
----   -----------------------------------   ----------       -------
  10   n7k13                                 inactive(Non AED)Overlay1
  11*  n7k14                                 active           Overlay1
  12   n7k13                                 inactive(Non AED)Overlay1
  13*  n7k14                                 active           Overlay1
  14   n7k13                                 inactive(Non AED)Overlay1
  15*  n7k14                                 active           Overlay1

To see the ARP-ND table use the next command:

n7k14# sh otv arp-nd-cache
OTV ARP/ND L3->L2 Address Mapping Cache
Overlay Interface Overlay1
VLAN  MAC Address         Layer-3 Address     Age       Expires In
15    001a.a1ff.7d46      15.1.1.32           00:03:42  00:04:17

The following command shows why the OTV is called MAC-in-IP routing:
To reach the first MAC address, the Next-Hop is an interface local within the site. To reach the last MAC address the Next-Hop in via the overlay as advertised by switch 24.

n7k14# sh otv route
OTV Unicast MAC Routing Table For Overlay1
VLAN MAC-Address     Metric  Uptime    Owner      Next-hop(s)
---- --------------  ------  --------  ---------  -----------
  15 000d.ecfe.077c  1       01:38:57  site       port-channel14
  15 0011.0000.0015  1       02:35:07  site       port-channel14
  15 0012.0000.0015  1       00:12:15  site       port-channel14
  15 001a.a1ff.7d46  42      01:44:44  overlay    n7k24
  15 0021.0000.0015  42      01:45:12  overlay    n7k24
  15 0022.0000.0015  42      00:12:16  overlay    n7k24

There are more commands with useful information, but I’m not going to show all examples here. Some other commands I found useful:

#sh otv site
#sh otv internal overlay
#sh otv internal adjacency
#sh tunnel internal implicit otv detail

.

FHRP Isolation

The last item I would like to cover before concluding this OTV trilogy post is FHRP Isolation. Let’s first look at why this is necessary.

I configured HSRP for VLAN-10 in both Site-1 (between switch 11 and 12) and Site-2 (between switch 21 and 22). Because it is a contiguous subnet all hosts in VLAN-10 has their gateway set to 10.1.1.1,  which is the same virtual gateway IP address in both sites. Now ideally traffic from Site-1 should exit via switch 11 and traffic from Site-2 should exit via switch 21 since both of these switches where setup with a HSRP priority of 105. But this is not what happens by default. Have a look at the output below:

On switch 11, the active HSRP device for VLAN-10 is switch 21. This would be the same on switch 12 and 22. Why would this be? Since both switch 11 and 21 have their HSRP priorities set to 105, the router with the higher interface IP address will be elected as the Active HSRP device. In this case switch 21. This means that all traffic towards the gateway of 10.1.1.1 will be forward to Site-2. This is the problem that we will now correct. The next output shows the traffic from Site-1 is destined via the overlay. Not cool.

FHRP isolation is the act of filtering HSRP, VRRP or GLBP traffic from going across the overlay, and thereby forcing localized FHRP elections. There are two parts to filter.

1. The election process should be contained within each site to elect a local active devices.
2. The virtual MAC addresses will still be advertised, which would cause constant MAC move moves. I.e., local site, remote site, local site, etc.

Point number 1 is accomplished using a VLAN ACL on the OTV Edge Devices to filter the respective traffic depending on which FHRP protocol used. The example below show how to filter all of them:

ip access-list HSRPv1-IP
10 permit udp any 224.0.0.2/32 eq 1985
!
ip access-list HSRPv2-IP
10 permit udp any 224.0.0.102/32 eq 1985
!
ip access-list VRRP-IP
10 permit udp any 224.0.0.18/32
20 permit 112 any any
!
ip access-list GLBP-IP
10 permit udp any 224.0.0.102/32 eq 3222
!
ip access-list ALL-IPs
10 permit ip any any
!
vlan access-map FHRP-FILTER 10
 match ip address HSRPv1-IP
 action drop
vlan access-map FHRP-FILTER 20
 match ip address HSRPv2-IP
 action drop
vlan access-map FHRP-FILTER 30
 match ip address VRRP-IP
 action drop
vlan access-map FHRP-FILTER 40
 match ip address GLBP-IP
 action drop
vlan access-map FHRP-FILTER 50
 match ip address ALL-IPs
 action forward
!
vlan filter FHRP-FILTER vlan-list 10-15

To prevent the virtual MAC addresses from causing MAC moves and allow for a cleaner design, an OTV route-map must be configured. This route-map must match the virtual MAC of the FHRP protocol used. For example HSRP v1 uses the virtual MAC: 0000.0c07.acXX where the last byte (XX) is the HSRP group number in HEX. Similar formats apply for VRRP and GLBP. The configuration below shows how to filter all FHRP protocols and should be applied on all OTV Edge Devices.

!! Filters HSRPv1 MAC addresses
mac-list OTV-FHRP-MAC seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00
!
!! Filters HSRPv2 MAC addresses
mac-list OTV-FHRP-MAC seq 11 deny 0000.0c9f.f000 ffff.ffff.ff00
!
!! Filters VRRP MAC addresses
mac-list OTV-FHRP-MAC seq 12 deny 0000.5e00.0100 ffff.ffff.ff00
!
!! Filters GLBP MAC addresses
mac-list OTV-FHRP-MAC seq 13 deny 0007.b400.0000 ffff.ff00.0000
mac-list OTV-FHRP-MAC seq 20 permit 0000.0000.0000 0000.0000.0000
!
route-map OTV-FHRP-FILTER permit 10
match mac-list OTV-FHRP-MAC
!
otv-isis default
  vpn Overlay1
    redistribute filter route-map OTV-FHRP-FILTER

Once the filters is applied, have a look at the same output as previously on both the intended FHRP breakout devices.

.

OTV Troubleshooting

Here is some good troubleshooting documents I found that might additionally be useful.

Troubleshooting OTV Adjacency
Troubleshooting ARP issues across OTV
OTV Silent Host Connectivity Problem