Troubleshooting MAC-Flushes on NX-OS

An interesting client problem in one of our multi-tenant data centers came to my attention the other day. A delay sensitive client noticed a slight increase in latency (20 ms) at very intermittent intervals from his servers in our data center to specific off-net destinations. The increase in latency was localized to the pair of Nexus 7000’s functioning as the core switch layer (CSW) and the layer3 edge for this particular data center. Beyond that all appeared normal on the N7K CSWs.

A TCP dump from a normal trunk interface attached to the N7Ks, showed unicast traffic on the N7K-2 device when the N7K-1 device was setup to receive internet traffic inbound and forward it into the data center client VLANs.  The N7Ks are setup using the Cisco VPC (Virtual Port Channels).

Continue reading “Troubleshooting MAC-Flushes on NX-OS”


Detecting Layer2 Loops

We all too familiar with the devastating impact a talented layer 2 loop could have on a data center lacking sufficient controls and processes. If you are using Cisco Nexus switches in your data center, you would be happy to know that NX-OS offers an interesting new tool you should add to your loop detection list. The somewhat undocumented feature is known as (for the lack of a better name)  FWM-Loop Detection. FWM refers to the NX-OS Forwarding Manager. In Syslog it is seen as:


Continue reading “Detecting Layer2 Loops”

N5K Stuck in Boot Mode

Another trivial post. The upcoming posts following this one will take a more in-depth look at the Nexus technologies.

So you do an non-ISSU NX-OS upgrade on a Nexus 5000 switch and something goes wrong. After reload you get the following prompt:

...Loader Version pr-1.3

The switch did not successfully boot from the images it was suppose to. How to go about restoring it?

Continue reading “N5K Stuck in Boot Mode”

Nexus load intervals

This is a interesting but a trivial post. Everybody know about the interface command “load-interval” that changes the time period over which the interface packet-rate and throughput statistics are averaged.

I discovered an addition to this command on the Nexus the other day while poking around. NX-OS allows multiple counter intervals to be configured on the same interface. This allows different sampled intervals to be listed at the same time.

The configuration is easy:

#interface Ethernet1/19
  load-interval counter 1 40
  load-interval counter 2 60
  load-interval counter 3 180

Continue reading “Nexus load intervals”

Cisco Nexus 7000 upgrade to 8Gb

When upgrading a Nexus 7000 to NX-OS version 5.2 (using more than 1 VDC) or to NX-OS v6+, Cisco claims the need to upgrade the system memory to 8Gb.

Note I have run on v5.2 using only 4Gb per SUP using 2 VDCs and it has worked just fine, but I should mention that the box was not under heavy load.

See how much memory your N7K has on a SUP by using the following command:

N7K# show system resources
Load average:   1 minute: 0.47   5 minutes: 0.24   15 minutes: 0.15
Processes   :   959 total, 1 running
CPU states  :   3.0% user,   3.5% kernel,   93.5% idle
Memory usage:   4115776K total,   2793428K used,   1322348K free

The upgrade per SUP would need the Cisco Bundle upgrade package (Product code: N7K-SUP1-8GBUPG=). One package has one 4Gb module. (see picture below) If you have two SUPs you would need two bundles. Notice the 8Gb sticker on module in the red block.

Continue reading “Cisco Nexus 7000 upgrade to 8Gb”