In this final post I will go through the configuration steps, some outputs and FHRP isolation.
OTV Lab Setup
I setup a mini lab using two Nexus 7000 switches, each with the four VDCs, two Nexus 5000 switches and a 3750 catalyst switch.
I emulated two data center sites, each with two core switches for typical layer3 breakout, each with two switches dedicated for OTV and each with one access switch to test connectivity. Site1 includes switches 11-14 (four VDCs on N7K-1) and switch 15 (N5K), whereas Site2 includes switches 21-24 (four VDCs on N7K-2) and switch 32 (3750).
To focus on OTV, I removed the complexity from the transport network by using OTV on dedicated VDCs (four of them for redundancy), connected as inline OTV appliances and by connecting the OTV Join interfaces on a single multi-access network.
This is the topology:
Before configuring OTV, the decision must be made how OTV will be integrated part of the data center design.
Recall the OTV/SVI co-existing limitation. If core switches are in place, which are not the Nexus 7000 switches, OTV may be implemented natively on the new Nexus 7000 switch/es or using a VDCs. If the Nexus 7000 switches are providing the core switch functionality, then separate VDCs are required for OTV.
The intra site connectivity is using Cisco’s VPC technology as the MLAG (Multi-chassis Link Aggregation) protocol. The VPC configuration is not covered here. I will cover VPCs in more detail in future posts. The configuration steps covered will focus specifically on OTV.
Step-1: OTV Feature
Enable the OTV feature within the system VDC. OTV requires a license. The Transport_Services_PKG license could be purchased and loaded. Alternatively if just for testing or POC, enable the grace period which will allow a 120 day demo/grace period to use and test OTV.
N7K-11(config)# license grace-period N7K-11(config)# feature otv N7K-11 %LICMGR-2-LOG_LIC_NO_LIC: No license(s) present for feature TRANSPORT_SERVICES_PKG. Application(s) shut down in 120 days. TRANSPORT_SERVICES_PKG license not installed. OTV feature will be shutdown after grace period of approximately 120 day(s)
Identify/Create the VLANs that should be extended between the sites across the overlay. Make sure all these VLANs are active.
N7K-13(config)# vlan 10-15 N7K-13(config-vlan)# name DATA-VLANS
Step-3: OTV Site VLAN
By default VLAN 1 will be used. It is recommended to use a dedicated VLAN. The allocated Site VLAN must not be extended across the overlay. As a result the same Site VLAN could be used at both sites. Even if only one OTV Edge Device, the Site VLAN must still be defined. Create a new VLAN to be used as the OTV Site VLAN and define this VLAN as the OTV Site VLAN.
N7K-13(config-vlan)# vlan 55 N7K-13(config-vlan)# name OTV-SITE-VLAN N7K-13(config-vlan)# exit N7K-13(config)# otv site-vlan 55
Step-4: Join Interfaces
Only one join interface can be specified per overlay on each device. Refer to the limitations here. Configure an IP address on each physical interface connected to the transport network. Enable IGMP v3 (required to join the SSM groups). Do not enable PIM on the Join interface. Test IP reachability between the Join interfaces.
N7K-13(config)#interface Ethernet2/9 N7K-13(config-int)#description otv-join-interface N7K-13(config-int)#no switchport N7K-13(config-int)#mtu 9216 N7K-13(config-int)#ip address 188.8.131.52/24 N7K-13(config-int)#ip igmp version 3 N7K-13(config-int)#no shutdown
Step-5: Internal interfaces
Configure the layer2 interfaces, that faces each site. These are the interfaces that will participate in STP and learning the MAC addresses from the local data center. I would recommended allowing only the relevant VLANs on these interfaces, i.e., the VLANs to be extended and the Site VLAN. Configure any further STP configurations if required.
N7K-13(config)# interface Ethernet2/10-11 N7K-13(config-int)#description otv-internal-interface N7K-13(config-int)#mtu 9216 N7K-13(config-int)#switchport N7K-13(config-int)#switchport mode trunk N7K-13(config-int)#switchport trunk allowed vlan 10-15,55 N7K-13(config-int)#channel-group 14 mode active N7K-13(config-int)#no shutdown
Step-6: Overlay interface
Create the logical Overlay interface. Multiple Overlay interfaces can be used to allow different VLANs to use different paths in the transport network, but two conditions are required for this. The overlay number must match between sites and a VLAN can only be assigned to one overlay. Shutdown the Overlay interface before configuring it. Specify the OTV control and data multicast groups.
N7K-13(config)#interface Overlay1 N7K-13(config-if-overlay)#desc overlay-interface N7K-13(config-if-overlay)#shutdown N7K-13(config-if-overlay)#otv join-interface Ethernet1/9 N7K-13(config-if-overlay)#otv control-group 184.108.40.206 N7K-13(config-if-overlay)#otv data-group 220.127.116.11/28
Step-7: Extended VLANs
Add the VLANs to be transported across the overlay. Notice the Site VLAN is not extended.
N7K-13(config)#interface Overlay1 N7K-13(config-if-overlay)#otv extend-vlan 10-15
Step-8: MTU values
Configuring the MTU values within the data center sites is less important, but in the transport network the correct values are vital. Recall that the DF bit is set on all OTV packets leaving an Edge Device.
Transport-Devices(config)#interface e1/7 Transport-Devices(config-if-overlay)#mtu 9216
Step-9: Unshut the overlay interface
If there is only one OTV Edge Device present per site, this step is trivial. If there are two OTV Edge Devices per site for load-sharing purposes, for the sake of stability, bring one Edge Device up in each site, confirm connectivity is working before bringing up the redundant OTV devices.
N7K-13(config)#interface Overlay1 N7K-13(config-if-overlay)#no shut ! N7K-23(config)#interface Overlay1 N7K-23(config-if-overlay)#no shut
Step-10: Test Connectivity
A simple ping from one site to another should be sufficient. This however can’t be done from the OTV Edge Devices, since they have no layer3 interfaces for the extended VLANs. At first it is normal for the first/second ECHO request to time out. This is the time it takes for the IP ARP request process to complete and allow OTV to advertise the newly learned MAC addresses from both sites. Subsequent PINGs should be fine.
N5K-15# ping 18.104.22.168 PING 22.214.171.124 (126.96.36.199): 56 data bytes Request 0 timed out 64 bytes from 188.8.131.52: icmp_seq=1 ttl=58 time=11.367 ms 64 bytes from 184.108.40.206: icmp_seq=2 ttl=58 time=2.001 ms 64 bytes from 220.127.116.11: icmp_seq=3 ttl=58 time=3.213 ms 64 bytes from 18.104.22.168: icmp_seq=4 ttl=58 time=2.894 ms --- 22.214.171.124 ping statistics --- 5 packets transmitted, 4 packets received, 20.00% packet loss round-trip min/avg/max = 2.001/4.431/11.367 ms
Output and Verification
The first useful command to verify the state of the overlay is “show otv“.
n7k13# sh otv OTV Overlay Information Overlay interface Overlay1 VPN name : Overlay1 VPN state : UP Extended vlans : 10-15 (Total:6) Control group : 126.96.36.199 Data group range(s) : 188.8.131.52/28 Join interface(s) : Eth1/9 (184.108.40.206) Site vlan : 55 (up)
The next command shows the adjacencies formed between all Edge Devices and the state.
n7k13# sh otv adjacency Overlay Adjacency database Overlay-Interface Overlay1 : Hostname System-ID Dest Addr Up Time State n7k14 0026.9812.2244 220.127.116.11 02:36:05 UP n7k23 0026.9810.91c3 18.104.22.168 02:41:40 UP n7k24 0026.9810.91c4 22.214.171.124 02:35:33 UP
The next command is very useful. It shows the AED for each VLAN. Recall that the Edge Device with a lower System-ID will become authoritative for all the even extended VLANs, whereas the Edge Device with higher System-ID will be authoritative the odd extended VLANs. n7k13 system-ID was 0026.9812.2243 and n7k14 system-ID was 0026.9812.2244.
n7k14# sh otv vlan OTV Extended VLANs and Edge Device State Information (* - AED) VLAN Auth. Edge Device Vlan State Overlay ---- ----------------------------------- ---------- ------- 10 n7k13 inactive(Non AED)Overlay1 11* n7k14 active Overlay1 12 n7k13 inactive(Non AED)Overlay1 13* n7k14 active Overlay1 14 n7k13 inactive(Non AED)Overlay1 15* n7k14 active Overlay1
To see the ARP-ND table use the next command:
n7k14# sh otv arp-nd-cache OTV ARP/ND L3->L2 Address Mapping Cache Overlay Interface Overlay1 VLAN MAC Address Layer-3 Address Age Expires In 15 001a.a1ff.7d46 126.96.36.199 00:03:42 00:04:17
The following command shows why the OTV is called MAC-in-IP routing:
To reach the first MAC address, the Next-Hop is an interface local within the site. To reach the last MAC address the Next-Hop in via the overlay as advertised by switch 24.
n7k14# sh otv route OTV Unicast MAC Routing Table For Overlay1 VLAN MAC-Address Metric Uptime Owner Next-hop(s) ---- -------------- ------ -------- --------- ----------- 15 000d.ecfe.077c 1 01:38:57 site port-channel14 15 0011.0000.0015 1 02:35:07 site port-channel14 15 0012.0000.0015 1 00:12:15 site port-channel14 15 001a.a1ff.7d46 42 01:44:44 overlay n7k24 15 0021.0000.0015 42 01:45:12 overlay n7k24 15 0022.0000.0015 42 00:12:16 overlay n7k24
There are more commands with useful information, but I’m not going to show all examples here. Some other commands I found useful:
#sh otv site #sh otv internal overlay #sh otv internal adjacency #sh tunnel internal implicit otv detail
The last item I would like to cover before concluding this OTV trilogy post is FHRP Isolation. Let’s first look at why this is necessary.
I configured HSRP for VLAN-10 in both Site-1 (between switch 11 and 12) and Site-2 (between switch 21 and 22). Because it is a contiguous subnet all hosts in VLAN-10 has their gateway set to 10.1.1.1, which is the same virtual gateway IP address in both sites. Now ideally traffic from Site-1 should exit via switch 11 and traffic from Site-2 should exit via switch 21 since both of these switches where setup with a HSRP priority of 105. But this is not what happens by default. Have a look at the output below:
On switch 11, the active HSRP device for VLAN-10 is switch 21. This would be the same on switch 12 and 22. Why would this be? Since both switch 11 and 21 have their HSRP priorities set to 105, the router with the higher interface IP address will be elected as the Active HSRP device. In this case switch 21. This means that all traffic towards the gateway of 10.1.1.1 will be forward to Site-2. This is the problem that we will now correct. The next output shows the traffic from Site-1 is destined via the overlay. Not cool.
FHRP isolation is the act of filtering HSRP, VRRP or GLBP traffic from going across the overlay, and thereby forcing localized FHRP elections. There are two parts to filter.
- The election process should be contained within each site to elect a local active devices.
- The virtual MAC addresses will still be advertised, which would cause constant MAC move moves. I.e., local site, remote site, local site, etc.
Point number 1 is accomplished using a VLAN ACL on the OTV Edge Devices to filter the respective traffic depending on which FHRP protocol used. The example below show how to filter all of them:
ip access-list HSRPv1-IP 10 permit udp any 188.8.131.52/32 eq 1985 ! ip access-list HSRPv2-IP 10 permit udp any 184.108.40.206/32 eq 1985 ! ip access-list VRRP-IP 10 permit udp any 220.127.116.11/32 20 permit 112 any any ! ip access-list GLBP-IP 10 permit udp any 18.104.22.168/32 eq 3222 ! ip access-list ALL-IPs 10 permit ip any any ! vlan access-map FHRP-FILTER 10 match ip address HSRPv1-IP action drop vlan access-map FHRP-FILTER 20 match ip address HSRPv2-IP action drop vlan access-map FHRP-FILTER 30 match ip address VRRP-IP action drop vlan access-map FHRP-FILTER 40 match ip address GLBP-IP action drop vlan access-map FHRP-FILTER 50 match ip address ALL-IPs action forward ! vlan filter FHRP-FILTER vlan-list 10-15
To prevent the virtual MAC addresses from causing MAC moves and allow for a cleaner design, an OTV route-map must be configured. This route-map must match the virtual MAC of the FHRP protocol used. For example HSRP v1 uses the virtual MAC: 0000.0c07.acXX where the last byte (XX) is the HSRP group number in HEX. Similar formats apply for VRRP and GLBP. The configuration below shows how to filter all FHRP protocols and should be applied on all OTV Edge Devices.
!! Filters HSRPv1 MAC addresses mac-list OTV-FHRP-MAC seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00 ! !! Filters HSRPv2 MAC addresses mac-list OTV-FHRP-MAC seq 11 deny 0000.0c9f.f000 ffff.ffff.ff00 ! !! Filters VRRP MAC addresses mac-list OTV-FHRP-MAC seq 12 deny 0000.5e00.0100 ffff.ffff.ff00 ! !! Filters GLBP MAC addresses mac-list OTV-FHRP-MAC seq 13 deny 0007.b400.0000 ffff.ff00.0000 mac-list OTV-FHRP-MAC seq 20 permit 0000.0000.0000 0000.0000.0000 ! route-map OTV-FHRP-FILTER permit 10 match mac-list OTV-FHRP-MAC ! otv-isis default vpn Overlay1 redistribute filter route-map OTV-FHRP-FILTER
Once the filters is applied, have a look at the same output as previously on both the intended FHRP breakout devices.
Here is some good troubleshooting documents I found that might additionally be useful.