h1

Jumbo MTU on Nexus 5000

April 5, 2011

Setting a per interface MTU (maximum transmission unit) is not supported on the Nexus 5000/2000 series switches.
If a Jumbo packet is required to traverse a Nexus 5000 series switch , the jumbo MTU must be set in a policy-map and applied to the ‘Sytem QOS’.

Configuration:

Configuration, PRE NX-OS 4.1:
policy-map JUMBO
 class class-default
  mtu 9216
system qos
 service-policy JUMBO

Configuration with POST NX-OS 4.1:
policy-map type network-qos JUMBO
 class type network-qos class-default
  mtu 9216
system qos
 service-policy type network-qos JUMBO

Verification:
First useful command is “sh queuing”. For clarity to understand the output of the command, by default QOS-Group 0 is for IP traffic and QOS-Group 1 is for FCOE traffic.  Thus look at the MTU setting for QOS-Group 0. As you can see it is set to 9216.

Nexus5k-lab2# sh queuing int e1/1
Ethernet1/1 queuing information:
TX Queuing
qos-group  sched-type  oper-bandwidth
0       WRR             50
1       WRR             50

RX Queuing
qos-group 0
q-size: 243200, HW MTU: 9280 (9216 configured)
drop-type: drop, xon: 0, xoff: 1520
Statistics:
Pkts received over the port             : 86
Ucast pkts sent to the cross-bar        : 77
Mcast pkts sent to the cross-bar        : 9
Ucast pkts received from the cross-bar  : 0
Pkts sent to the port                   : 0
Pkts discarded on ingress               : 0
Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

qos-group 1
q-size: 76800, HW MTU: 2240 (2158 configured)
drop-type: no-drop, xon: 128, xoff: 240
Statistics:
Pkts received over the port             : 0
Ucast pkts sent to the cross-bar        : 0
Mcast pkts sent to the cross-bar        : 0
Ucast pkts received from the cross-bar  : 0
Pkts sent to the port                   : 0
Pkts discarded on ingress               : 0
Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

Total Multicast crossbar statistics:
Mcast pkts received from the cross-bar      : 23

Assuming you’ve had jumbo traffic,  the jumbo counters with “show int” and “show interface {int} counter detail” will be non-zero.

Nexus-5k-SW1# sh int e1/2 | i jumbo
    31 jumbo packets  0 storm suppression packets
    10 jumbo packets

Nexus5k-lab1# sh int counters detailed | i Jumbo
 Rx Jumbo Packets:                                    10
 Tx Jumbo Packets:                                    51
 Rx Jumbo Packets:                                    10
 Tx Jumbo Packets:                                    51

It is important to note that the command “show interface” does not display the correct MTU value. This is a cosmetic bug which has been logged:

CSCsl21529, Symptom: An incorrect MTU value is displayed in the show interface command output. The Cisco Nexus 5000 Series switch only supports class-based MTU. Per-interface level MTU configuration is not supported. The switch supports jumbo frames by default. However, the show interface command output currently displays an incorrect MTU value of 1500 bytes.

About these ads

4 comments

  1. Excellent post. I’ve had many customers ask why they can’t see MTU under show interface and I’ve always referenced the bugID. Now I will reference your blog instead :)


  2. oho man you are lucky , getting experience in next generation nexus switches , thanks for sharing the knowledge


    • hehe, its cool, but takes time to test each and every little thing :)


  3. Great article and insight on class based MTU for Jumbo packets on the FEX (2K) and 5K Nexus switches. Bravo Sir!



Please leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,522 other followers

%d bloggers like this: