Port-channels have become an acceptable solution in data centers to both mitigate STP footprints and extend physical interface limits.
One of the biggest drawbacks with port-channels is the single point of failure.
Scenario 1- Failure of an ASIC on one switch, which could potentially bring the port-channel down, if all member interfaces were connected on one ASIC.
Scenario 2- Failure of one switch on either side. The obvious solution available today is multi-chassis port-channels which addresses the problem 95%.
Consider the following topology:
Even with multi-chassis port-channel there is the still the possibility of an ASIC failure. Although not as detrimental as Scenario-1, there will still be some impact (depending on the traffic load) if both interfaces on one switch happen to connect to the same ASIC.
Thus it only makes sense that the ports used on the same switch, uses different ASICs. How would confirm this on the Nexus 5000 and Nexus 7000?
Nexus 50xx Port-to-ASIC mapping
The Nexus 5000 series architecture (Models 5010 and 5020) uses a UPC (Unified Port Controller) ASIC. Each UPC ASIC a.k.a. gatos have 4 hardwired ports. This can be seen with the following command:
N5010# sh hardware internal gatos all-ports Gatos Port Info: name |log|gat|mac|flag|adm|opr|c:m:s:l|ipt|fab|xgat|xpt|if_index|diag -------+---+---+---+----+---+---+-------+---+---+----+---+--------+---- 1gb1/8 |7 |0 |0 |b7 |en |dn |0:0:0:0|0 |55 |0 |2 |1a007000|pass 1gb1/7 |6 |0 |1 |b7 |en |dn |0:1:1:0|1 |54 |0 |0 |1a006000|pass 1gb1/3 |2 |0 |2 |b7 |en |up |1:2:2:0|2 |56 |0 |4 |1a002000|pass 1gb1/4 |3 |0 |3 |b7 |en |up |1:3:3:0|3 |57 |0 |6 |1a003000|pass 1gb1/16|15 |1 |0 |b7 |en |up |0:0:0:0|0 |50 |1 |2 |1a00f000|pass 1gb1/15|14 |1 |1 |b7 |dis|dn |0:1:1:0|1 |51 |1 |0 |1a00e000|pass 1gb1/11|10 |1 |2 |b7 |en |dn |1:2:2:0|2 |53 |1 |4 |1a00a000|pass 1gb1/12|11 |1 |3 |b7 |en |dn |1:3:3:0|3 |52 |1 |6 |1a00b000|pass xgb1/39|38 |2 |0 |b7 |en |up |0:0:0:f|0 |45 |4 |0 |1a026000|pass xgb1/23|22 |2 |1 |b7 |dis|dn |0:1:1:f|1 |44 |2 |0 |1a016000|pass xgb1/40|39 |2 |2 |b7 |en |up |1:2:2:f|2 |49 |4 |2 |1a027000|pass xgb1/24|23 |2 |3 |b7 |dis|dn |1:3:3:f|3 |48 |2 |2 |1a017000|pass xgb1/36|35 |3 |0 |b7 |en |dn |0:0:0:f|0 |41 |4 |6 |1a023000|pass xgb1/20|19 |3 |1 |b7 |en |up |0:1:1:f|1 |40 |2 |6 |1a013000|pass xgb1/35|34 |3 |2 |b7 |en |dn |1:2:2:f|2 |46 |4 |4 |1a022000|pass xgb1/19|18 |3 |3 |b7 |en |up |1:3:3:f|3 |47 |2 |4 |1a012000|pass xgb1/31|30 |4 |1 |b7 |dis|dn |0:1:1:f|1 |36 |3 |0 |1a01e000|pass xgb1/32|31 |4 |3 |b7 |dis|dn |1:3:3:f|3 |42 |3 |2 |1a01f000|pass sup0 |40 |5 |0 |b7 |en |dn |0:0:0:0|0 |33 |0 |0 |15020000|pass sup1 |41 |5 |1 |b7 |en |dn |0:0:0:1|0 |33 |0 |0 |15010000|pass --snip--
Column 1- Indicates the physical ports e.g., Eth1/8, Eth1/7, Eth1/3, Eth1/4, etc.
Column 3- Indicates the Gatos (UPC ASIC) number.
Column 4- Indicates the physical connection on the ASIC. 0-3 are the 4 port available on the ASIC.
Nexus 55xx Port-to-ASIC mapping
The Nexus 5500 series architecture (Models 5548 and 5596) uses a UPC (Unified Port Controller) ASIC. Each UPC ASIC a.k.a. carmel on the 55 range have 8 ports mapped. This can be seen with the following command:
Nexus5548# show hardware internal carmel all Carmel Port Info: name |log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer -------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+----- xgb1/1 |0 |0 |0 -|b7 |dis|dn |0:0:f|0 |92 |0 |0 |1a000000|pass| 0.00 xgb1/2 |1 |0 |1 -|b7 |dis|dn |1:1:f|1 |88 |0 |0 |1a001000|pass| 0.00 xgb1/3 |2 |0 |2 -|b7 |dis|dn |2:2:f|2 |93 |0 |0 |1a002000|pass| 0.00 xgb1/4 |3 |0 |3 -|b7 |dis|dn |3:3:f|3 |89 |0 |0 |1a003000|pass| 0.00 xgb1/5 |4 |0 |4 -|b7 |dis|dn |4:4:f|4 |90 |0 |0 |1a004000|pass| 0.00 xgb1/6 |5 |0 |5 -|b7 |dis|dn |5:5:f|5 |94 |0 |0 |1a005000|pass| 0.00 xgb1/7 |6 |0 |6 -|b7 |dis|dn |6:6:f|6 |95 |0 |0 |1a006000|pass| 0.00 xgb1/8 |7 |0 |7 -|b7 |dis|dn |7:7:f|7 |91 |0 |0 |1a007000|pass| 0.00 xgb1/9 |8 |1 |0 -|b7 |dis|dn |0:0:f|0 |80 |0 |0 |1a008000|pass| 0.00 xgb1/10|9 |1 |1 p|b7 |en |up |1:1:f|1 |87 |0 |0 |1a009000|pass| 0.00 xgb1/11|10 |1 |2 -|b7 |dis|dn |2:2:f|2 |81 |0 |0 |1a00a000|pass| 0.00 xgb1/12|11 |1 |3 -|b7 |dis|dn |3:3:f|3 |86 |0 |0 |1a00b000|pass| 0.00 xgb1/13|12 |1 |4 -|b7 |dis|dn |4:4:f|4 |82 |0 |0 |1a00c000|pass| 0.00 xgb1/14|13 |1 |5 -|b7 |dis|dn |5:5:f|5 |85 |0 |0 |1a00d000|pass| 0.00 xgb1/15|14 |1 |6 -|b7 |dis|dn |6:6:f|6 |83 |0 |0 |1a00e000|pass| 0.00 xgb1/16|15 |1 |7 -|b7 |dis|dn |7:7:f|7 |84 |0 |0 |1a00f000|pass| 0.00 xgb1/17|16 |2 |0 -|b7 |dis|dn |0:0:f|0 |75 |0 |0 |1a010000|pass| 0.00 --snip--
Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/2, Eth1/3, Eth1/4, etc.
Column 3- Indicates the Carmel (UPC ASIC) number.
Column 4- Indicates the physical connection on the ASIC. 0-3 are the 4 port available on the ASIC.
Nexus 7000 Port-to-ASIC mapping
The Nexus 7000 being a modular switch uses line cards. The port-to-ASIC mapping does differ between the different types of line cards. The following command however will indicate the port mapping:
N7010# slot 1 show hardware internal dev-port-map | begin FP FP port|PHYS |MAC_0 |MAC_1 |MAC_2 |RWR_0 |RWR_1 |L2LKP |L3LKP |QUEUE |SWICHF 1 15 7 15 7 3 0 0 0 1 0 2 0 0 0 0 0 1 0 0 0 0 3 15 7 15 7 3 0 0 0 1 0 4 0 0 0 0 0 1 0 0 0 0 5 14 7 14 7 3 0 0 0 1 0 6 1 0 1 0 0 1 0 0 0 0 7 14 7 14 7 3 0 0 0 1 0 8 1 0 1 0 0 1 0 0 0 0 9 13 6 13 6 3 0 0 0 1 0 10 2 1 2 1 0 1 0 0 0 0 11 13 6 13 6 3 0 0 0 1 0 12 2 1 2 1 0 1 0 0 0 0 --snip--
Column 1- Indicates the Ethernet ports on line card 1 e.g., Eth1/1, Eth1/2, Eth1/3, Eth1/4, etc.
Column 2- Indicates the physical ASIC number. across
Above is the output from a N7K-M132XP-12 line card showing 2 ports per ASIC.
–Update 13-Jun–
Santiago correctly pointed out that the command structure on the Nexus 55xx series architecture is diffrent. Post updated accordingly. thx
I’ve also seen a scenario where the portchannel was split across different chassis and an ASIC in one failed, but it didn’t fail to the point of the link going down and the switch still saw it as active. This led to half of the traffic working fine across one half of the channel, and the rest going nowhere. Not fun at all to troubleshoot
Partial hardware failures are the absolute worst to troubleshoot :(
Command line for Nexus 5500
switch#show hardware internal carmel all-ports
also you can use
switch# show hardware internal carmel detail
Cheers
Nexus 3548-vpc-3548 with portchannel on same asic, connected to AIX host fails lacp negotiation. Port buffers dropping unicast packed on port-channel member port.
show queuing interface e1/22
Ethernet1/22 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
Multicast statistics:
Mcast pkts dropped : 38010
Unicast statistics:
qos-group 0
HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Ucast pkts dropped : 1886498
show hardware internal mtc-usd info port-mapping
OB Ports to Front Ports:
========= OB0 ========= ========= OB1 ========= ========= OB2 =========
45 47 21 23 09 11 33 35 17 19 05 07 41 43 29 31 13 15 37 39 25 27 01 03
46 48 22 24 10 12 34 36 18 20 06 08 42 44 30 32 14 16 38 40 26 28 02 04
Front Ports to OB Ports:
=OB2= =OB1= =OB0= =OB2= =OB1= =OB0= =OB2= =OB1= =OB0= =OB2= =OB1= =OB0=
12 14 04 06 08 10 00 02 00 02 04 06 08 10 12 14 12 14 04 06 08 10 00 02
13 15 05 07 09 11 01 03 01 03 05 07 09 11 13 15 13 15 05 07 09 11 01 03