Load-Sharing across ASICs

Port-channels have become an acceptable solution in data centers to both mitigate STP footprints and extend physical interface limits.

One of the biggest drawbacks with port-channels is the single point of failure.

Scenario 1- Failure of an ASIC on one switch, which could potentially bring the port-channel down, if all member interfaces were connected on one ASIC.

Scenario 2- Failure of one switch on either side. The obvious solution available today is multi-chassis port-channels which addresses the problem 95%.

Consider the following topology:

Even with multi-chassis port-channel there is the still the possibility of an ASIC failure.  Although not as detrimental as Scenario-1, there will still be some impact (depending on the traffic load) if both interfaces on one switch happen to connect to the same ASIC.

Thus it only makes sense that the ports used on the same switch, uses different ASICs. How would confirm this on the Nexus 5000 and Nexus 7000?

Nexus 50xx Port-to-ASIC mapping

The Nexus 5000 series architecture (Models 5010 and 5020) uses a UPC (Unified Port Controller) ASIC. Each UPC ASIC a.k.a. gatos have 4 hardwired ports. This can be seen with the following command:

N5010# sh hardware internal gatos all-ports

Gatos Port Info:
name   |log|gat|mac|flag|adm|opr|c:m:s:l|ipt|fab|xgat|xpt|if_index|diag
-------+---+---+---+----+---+---+-------+---+---+----+---+--------+----
1gb1/8 |7  |0  |0  |b7  |en |dn |0:0:0:0|0  |55 |0   |2  |1a007000|pass
1gb1/7 |6  |0  |1  |b7  |en |dn |0:1:1:0|1  |54 |0   |0  |1a006000|pass
1gb1/3 |2  |0  |2  |b7  |en |up |1:2:2:0|2  |56 |0   |4  |1a002000|pass
1gb1/4 |3  |0  |3  |b7  |en |up |1:3:3:0|3  |57 |0   |6  |1a003000|pass
1gb1/16|15 |1  |0  |b7  |en |up |0:0:0:0|0  |50 |1   |2  |1a00f000|pass
1gb1/15|14 |1  |1  |b7  |dis|dn |0:1:1:0|1  |51 |1   |0  |1a00e000|pass
1gb1/11|10 |1  |2  |b7  |en |dn |1:2:2:0|2  |53 |1   |4  |1a00a000|pass
1gb1/12|11 |1  |3  |b7  |en |dn |1:3:3:0|3  |52 |1   |6  |1a00b000|pass
xgb1/39|38 |2  |0  |b7  |en |up |0:0:0:f|0  |45 |4   |0  |1a026000|pass
xgb1/23|22 |2  |1  |b7  |dis|dn |0:1:1:f|1  |44 |2   |0  |1a016000|pass
xgb1/40|39 |2  |2  |b7  |en |up |1:2:2:f|2  |49 |4   |2  |1a027000|pass
xgb1/24|23 |2  |3  |b7  |dis|dn |1:3:3:f|3  |48 |2   |2  |1a017000|pass
xgb1/36|35 |3  |0  |b7  |en |dn |0:0:0:f|0  |41 |4   |6  |1a023000|pass
xgb1/20|19 |3  |1  |b7  |en |up |0:1:1:f|1  |40 |2   |6  |1a013000|pass
xgb1/35|34 |3  |2  |b7  |en |dn |1:2:2:f|2  |46 |4   |4  |1a022000|pass
xgb1/19|18 |3  |3  |b7  |en |up |1:3:3:f|3  |47 |2   |4  |1a012000|pass
xgb1/31|30 |4  |1  |b7  |dis|dn |0:1:1:f|1  |36 |3   |0  |1a01e000|pass
xgb1/32|31 |4  |3  |b7  |dis|dn |1:3:3:f|3  |42 |3   |2  |1a01f000|pass
sup0   |40 |5  |0  |b7  |en |dn |0:0:0:0|0  |33 |0   |0  |15020000|pass
sup1   |41 |5  |1  |b7  |en |dn |0:0:0:1|0  |33 |0   |0  |15010000|pass
--snip--

Column 1- Indicates the physical ports e.g., Eth1/8, Eth1/7, Eth1/3, Eth1/4, etc.
Column 3- Indicates the Gatos (UPC ASIC) number.
Column 4- Indicates the physical connection on the ASIC. 0-3 are the 4 port available on the ASIC.

Nexus 55xx Port-to-ASIC mapping

The Nexus 5500 series architecture (Models 5548 and 5596) uses a UPC (Unified Port Controller) ASIC. Each UPC ASIC a.k.a. carmel on the 55 range have 8 ports mapped. This can be seen with the following command:

Nexus5548# show hardware internal carmel all

Carmel Port Info:
name   |log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer
-------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+-----
xgb1/1 |0  |0  |0 -|b7  |dis|dn |0:0:f|0  |92 |0   |0  |1a000000|pass| 0.00
xgb1/2 |1  |0  |1 -|b7  |dis|dn |1:1:f|1  |88 |0   |0  |1a001000|pass| 0.00
xgb1/3 |2  |0  |2 -|b7  |dis|dn |2:2:f|2  |93 |0   |0  |1a002000|pass| 0.00
xgb1/4 |3  |0  |3 -|b7  |dis|dn |3:3:f|3  |89 |0   |0  |1a003000|pass| 0.00
xgb1/5 |4  |0  |4 -|b7  |dis|dn |4:4:f|4  |90 |0   |0  |1a004000|pass| 0.00
xgb1/6 |5  |0  |5 -|b7  |dis|dn |5:5:f|5  |94 |0   |0  |1a005000|pass| 0.00
xgb1/7 |6  |0  |6 -|b7  |dis|dn |6:6:f|6  |95 |0   |0  |1a006000|pass| 0.00
xgb1/8 |7  |0  |7 -|b7  |dis|dn |7:7:f|7  |91 |0   |0  |1a007000|pass| 0.00
xgb1/9 |8  |1  |0 -|b7  |dis|dn |0:0:f|0  |80 |0   |0  |1a008000|pass| 0.00
xgb1/10|9  |1  |1 p|b7  |en |up |1:1:f|1  |87 |0   |0  |1a009000|pass| 0.00
xgb1/11|10 |1  |2 -|b7  |dis|dn |2:2:f|2  |81 |0   |0  |1a00a000|pass| 0.00
xgb1/12|11 |1  |3 -|b7  |dis|dn |3:3:f|3  |86 |0   |0  |1a00b000|pass| 0.00
xgb1/13|12 |1  |4 -|b7  |dis|dn |4:4:f|4  |82 |0   |0  |1a00c000|pass| 0.00
xgb1/14|13 |1  |5 -|b7  |dis|dn |5:5:f|5  |85 |0   |0  |1a00d000|pass| 0.00
xgb1/15|14 |1  |6 -|b7  |dis|dn |6:6:f|6  |83 |0   |0  |1a00e000|pass| 0.00
xgb1/16|15 |1  |7 -|b7  |dis|dn |7:7:f|7  |84 |0   |0  |1a00f000|pass| 0.00
xgb1/17|16 |2  |0 -|b7  |dis|dn |0:0:f|0  |75 |0   |0  |1a010000|pass| 0.00
--snip--

Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/2, Eth1/3, Eth1/4, etc.
Column 3- Indicates the Carmel (UPC ASIC) number.
Column 4- Indicates the physical connection on the ASIC. 0-3 are the 4 port available on the ASIC.

Nexus 7000 Port-to-ASIC mapping

The Nexus 7000 being a modular switch uses line cards. The port-to-ASIC mapping does differ between the different types of line cards. The following command however will indicate the port mapping:

N7010# slot 1 show hardware internal dev-port-map | begin FP
FP port|PHYS |MAC_0 |MAC_1 |MAC_2 |RWR_0 |RWR_1 |L2LKP |L3LKP |QUEUE |SWICHF
   1    15      7     15      7      3      0      0      0      1      0
   2     0      0      0      0      0      1      0      0      0      0
   3    15      7     15      7      3      0      0      0      1      0
   4     0      0      0      0      0      1      0      0      0      0
   5    14      7     14      7      3      0      0      0      1      0
   6     1      0      1      0      0      1      0      0      0      0
   7    14      7     14      7      3      0      0      0      1      0
   8     1      0      1      0      0      1      0      0      0      0
   9    13      6     13      6      3      0      0      0      1      0
  10     2      1      2      1      0      1      0      0      0      0
  11    13      6     13      6      3      0      0      0      1      0
  12     2      1      2      1      0      1      0      0      0      0
--snip--

Column 1- Indicates the Ethernet ports on line card 1 e.g., Eth1/1, Eth1/2, Eth1/3, Eth1/4, etc.
Column 2- Indicates the physical ASIC number. across

Above is the output from a N7K-M132XP-12 line card showing 2 ports per ASIC.

–Update 13-Jun–

Santiago correctly pointed out that the command structure on the Nexus 55xx series architecture is diffrent. Post updated accordingly. thx

Advertisements

4 thoughts on “Load-Sharing across ASICs

  1. I’ve also seen a scenario where the portchannel was split across different chassis and an ASIC in one failed, but it didn’t fail to the point of the link going down and the switch still saw it as active. This led to half of the traffic working fine across one half of the channel, and the rest going nowhere. Not fun at all to troubleshoot

  2. Command line for Nexus 5500

    switch#show hardware internal carmel all-ports

    also you can use

    switch# show hardware internal carmel detail

    Cheers

  3. Nexus 3548-vpc-3548 with portchannel on same asic, connected to AIX host fails lacp negotiation. Port buffers dropping unicast packed on port-channel member port.

    show queuing interface e1/22

    Ethernet1/22 queuing information:
    TX Queuing
    qos-group sched-type oper-bandwidth
    0 WRR 100

    RX Queuing
    Multicast statistics:
    Mcast pkts dropped : 38010
    Unicast statistics:
    qos-group 0
    HW MTU: 9216 (9216 configured)
    drop-type: drop, xon: 0, xoff: 0
    Statistics:
    Ucast pkts dropped : 1886498

    show hardware internal mtc-usd info port-mapping
    OB Ports to Front Ports:
    ========= OB0 ========= ========= OB1 ========= ========= OB2 =========
    45 47 21 23 09 11 33 35 17 19 05 07 41 43 29 31 13 15 37 39 25 27 01 03
    46 48 22 24 10 12 34 36 18 20 06 08 42 44 30 32 14 16 38 40 26 28 02 04

    Front Ports to OB Ports:
    =OB2= =OB1= =OB0= =OB2= =OB1= =OB0= =OB2= =OB1= =OB0= =OB2= =OB1= =OB0=
    12 14 04 06 08 10 00 02 00 02 04 06 08 10 12 14 12 14 04 06 08 10 00 02
    13 15 05 07 09 11 01 03 01 03 05 07 09 11 13 15 13 15 05 07 09 11 01 03

Please leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s