Upgrading a Cisco 6500

Upgrading a 6500 is pretty straight forward, provided the necessary is done in the right order. I’ve listed the steps I would typically take to fully upgrade a single Cisco-6509-E (single Route-Processor) with a IPSEC VPN SPA blade.

Please lab this if possible BEFORE trying it in a production network. I have illustrated the steps to be taken if some of the known funnies occur during an upgrade. Feel free to use this as a guideline.

Firstly download the IOS and image versions, you need. Obviously do a little homework and check the specific IOS for known bugs using the Bug Toolkit. Don’t just pick any IOS. Make sure all the required features are relatively bug free.

Copy the downloaded files to the following locations:

ROMMON firmware to sup-bootflash
BOOTLDR to bootflash
IOS to flash disk

I always use FTP if possible, due to the higher transfer rates. 10.3.29.239 is connected to the switch and is running a FTP server, expecting a username:password of cisco:pass.

copy ftp://cisco:pass@10.3.29.239/c6msfc3-rm2.srec.122-17r.SX5 sup-bootflash:
copy ftp://cisco:pass@10.3.29.239/s72033-boot-mz.122-33.SXI2.bin bootflash:
copy ftp://cisco:pass@10.3.29.239/s72033-adventerprisek9_wan-mz.122-33.SXI2.bin disk0:
dir sup-bootflash:
dir bootflash:
dir disk0:

Continue reading “Upgrading a Cisco 6500” →

Troubleshooting a Cisco 6500 crash

I was asked recently to share some knowledge about the support of the Cisco 6500 switches as the information available on the DOC-CD could be fairly overwhelming.

As it happens a clients Cisco-6509 switch fell over yesterday. I was called out to address the issue of the Cisco-6509 that decided it was tired of life by rebooting itself. I’ll go through some of the steps I did to find the root cause. Obviously note the steps listed here will not find the cause of every possible issue with a 6500 switch, but can be used as a guideline.

Usually the first thing I would do is to see the reason for the reboot with a “sh version”. Look at the highlighted lines.

ndcbbnpendc0103#sh ver
Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Mon 18-Sep-06 23:32 by tinhuang
Image text-base: 0x40101040, data-base: 0x42D90000

ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
BOOTLDR: s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF6, RELEASE SOFTWARE (fc1)

ndcbbnpendc0103 uptime is 3 hours, 23 minutes
Time since ndcbbnpendc0103 switched to active is 3 hours, 22 minutes
System returned to ROM by s/w reset at 00:14:27 PDT Wed Sep 20 2006 (SP by bus error at PC 0x402DC89C, address 0x0)
System restarted at 09:13:44 ZA Wed Mar 10 2010
System image file is "disk0:s72033-adventerprisek9_wan-mz.122-18.SXF6.bin"

Obviously it is clear that the switch did a software reset caused by ‘bus error at PC 0x402DC89C, address 0x0‘.

Continue reading “Troubleshooting a Cisco 6500 crash” →

IOS upgrade tip

So it is sunday morning, the change window just kicked in, you copied the new IOS image to the router, used the ‘boot system’ command as per my previous post, you save your config and reload. All looking good for an early night, but when the router reloads you get a bunch of errors during bootup along the lines of:

% Invalid input detected at '^' marker.
% Incomplete command.

Oh no, you didn’t do you homework, did you check for command differences between the IOS versions? Did you test the current config on the new IOS in a lab prior to the upgrade (yes not always possible), do you have a config backup?

If you don’t have a full config backup you have BIGGER problems. If you remove the first ‘boot system’ command to boot of the working IOS, and write your config, usually all commands that gave errors during that boot-up, will now be LOST since you saved the config and overwrote the startup-config.

So what now?

Appose to freaking out and start dancing like a banshee doing some tribal dance, do the following. ‘Rename’ the NEWLY installed IOS image in Flash, the image specified in the first ‘boot system’ command to something else, and ‘Reload’ WITHOUT SAVING the config. When the router reboots it will attempt to locate the first specified boot system image, but because you renamed it, it can not be loaded. The router will then attempt to boot off the second specified boot system image, the old working IOS image. And happiness is restored.

Boot System command

Whenever doing IOS upgrades on production devices, it is always best to have a fail-back plan. With most platforms, the best way to archieve this is with the ‘boot system’ command and altough not a new command, still good to cover.

The ‘boot system’ command is non-exclusive, and if multiple entries exist, the commands are tried top down. If the first is unsuccessful, the second will be tried. If the first is successful it is loaded. This is very handy, since it provides IOS-image redundancy, and a fail-back option.

What do I mean by IOS-image redundancy?
If you are using a bigger platform that has 2x Compact Flash Slots, by loading the same IOS image on both disks you can have image redundancy, incase one of the Compact Flash Cards bombs out, with the following config:

boot system flash disk0:s72033-adventerprisek9_wan-mz.122-33.SXI2.bin boot system flash disk1:s72033-adventerprisek9_wan-mz.122-33.SXI2.bin

The same redundancy could be achieved by using a remote FTP/TFTP location as the second command.
.

This command however becomes most useful when doing IOS upgrades. Provided you have enough space to store a second IOS image, load the NEW IOS Image on that same disk, then by using the ‘boot system’ command, specify the path to the NEW IOS Image first and then the path to the current IOS Image:

boot system flash disk0:s72033-adventerprisek9_wan-mz.122-33.SXI2.bin boot system flash disk0:s72033-adventerprisek9_wan-mz.122-18.SXF6.bin

Now looking at the example above, when the router reboots, the router will try to load 122-33.SXI2 first, if unsuccessful, then it will load the current (122-18.SXF6).