Showing posts with label disaster. Show all posts

Thursday, August 7, 2014

Cisco – Modify / Fix Config via SNMP (SSH example)

Fixing SSH access on cisco via SNMP

Sometimes you may encounter a situation, when your SSH is not properly configured. In this situation you might be lucky enough to have SNMP RW community string configured. In this situation you can fix literally everything.

Download the Config via SNMP

You may download current device’s config to tftp server, edit necessary lines and upload it back. You may upload it to either running config, startup config or a flash file.

To download running config:

snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.2.111 i 1
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.3.111 i 4
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.4.111 i 1
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.1.252
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.6.111 s config_dsw1.txt
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.14.111 i 1

where:

san-fran - is the community key of your cisco router.
192.168.1.23 - is the ip address of your Cisco device.
192.168.1.252 - is the ip address of your tftp server.
config_dsw1 - is the name where the running configuration
will be saved.
111 - random number

Commands Explained

1.3.6.1.4.1.9.9.96.1.1.1.1.2.111 i 1
ccCopyProtocol: The protocol file transfer protocol:

1 - tftp
2 - ftp
3 - rcp
4 - scp
5 - sftp

1.3.6.1.4.1.9.9.96.1.1.1.1.3.111 i 4
ccCopySourceFileType: Specifies the type of file to copy from:

1 - networkFile
2 - iosFile
3 - startupConfig
4 - runningConfig
5 - terminal
6 - fabricStartupConfig

1.3.6.1.4.1.9.9.96.1.1.1.1.4.111 i 1
ccCopyDestFileType: specifies the type of file to copy to:

1 - networkFile
2 - iosFile
3 - startupConfig
4 - runningConfig
5 - terminal
6 - fabricStartupConfig

1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.1.252
ccCopyServerAddress: The IP address of the TFTP server

1.3.6.1.4.1.9.9.96.1.1.1.1.6.111 s config_dsw1.txt
ccCopyFileName: The file name (including the path) of the file.

1.3.6.1.4.1.9.9.96.1.1.1.1.14.336 i 1
ccCopyEntryRowStatus: The status of this table entry. Once is set to active, the associated entry cannot be modified until
the request completes (‘successful’ or ‘failed’) The object can be:

1 - active
2 - notInService
3 - notReady
4 - createAndGo
5 - createAndWait
6 - destroy

When enter this command, the router will send the running configuration to the ftp-server!

Change the Config

If you encountered situation with SSH with no generated certificate, You config might look like this:

line vty 0 4
length 0
transport input ssh
line vty 5 15
transport input ssh
exit

You should fix it to:

line vty 0 4
length 0
transport input telnet
line vty 5 15
transport input telnet
exit

Some commands can be cancelled with “no ” statment before the command. Some, as in above case, not.

Upload the Config via SNMP

Upload it back by the following commands. Be careful! If you upload to startup-config, IOS will not merge the uploaded config and the startup one, it will replace it instead. Do not upload partial sets of commands!. To be on a safe side always I recommend to never upload partial configs. Only necessary lines should be added/cancelled/corrected and the whole config should be uploaded.

snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.2.222 i 1
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.3.222 i 1
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.4.222 i 4
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.5.222 a 192.168.1.252
snmpset -c san-fran -v 2c 192.168.1.23 1.3.6.1.4.1.9.9.96.1.1.1.1.6.222 s config_dsw1.txt

where:

san-fran - is the community key of your cisco router.
192.168.1.23 - is the ip address of your Cisco device.
192.168.1.252 - is the ip address of your tftp server.
config_dsw1.txt - is the name where the running configuration will be saved.
222 - random number

Commands Explained

1.3.6.1.4.1.9.9.96.1.1.1.1.2.222 i 1
ccCopyProtocol: The protocol file transfer protocol:

1 - tftp
2 - ftp
3 - rcp
4 - scp
5 – sftp

1.3.6.1.4.1.9.9.96.1.1.1.1.3.222 i 1
ccCopySourceFileType: Specifies the type of file to copy from:

1 - networkFile
2 - iosFile
3 - startupConfig
4 - runningConfig
5 - terminal
6 - fabricStartupConfig

1.3.6.1.4.1.9.9.96.1.1.1.1.4.222 i 4
ccCopyDestFileType: specifies the type of file to copy to:

1 - networkFile
2 - iosFile
3 - startupConfig
4 - runningConfig
5 - terminal
6 - fabricStartupConfig

1.3.6.1.4.1.9.9.96.1.1.1.1.5.222 a 192.168.1.252
ccCopyServerAddress: The IP address of the TFTP server to copy the configuration file from.

1.3.6.1.4.1.9.9.96.1.1.1.1.6.222 s config_dsw1.txt
ccCopyFileName: The file name (including the path) of the file.

1.3.6.1.4.1.9.9.96.1.1.1.1.14.222 i 1
ccCopyEntryRowStatus: The status of this table entry.
Once is set to active, the associated entry cannot be modified until the request completes (‘successful’ or ‘failed’) The object can be:

1 - active
2 - notInService
3 - notReady
4 - createAndGo
5 - createAndWait
6 - destroy

Based On:

Related Links

Thursday, August 22, 2013

Cisco IOS Resilient Image and Configuration

Last week, we looked at Recovering a Router with the Password Recovery Service Disabled. Today we're going to examine a related Cisco IOS security feature, dubbed resilient configuration. This feature enables critical router files, namely the IOS image and configuration, to persist despite destructive events such as deletion of the startup configuration or a format of the Flash filesystem. The feature does not require any external services; all persistent files are stored locally on the router.

Enabling Resilient Configuration

First, a quick review of how Cisco ISR (x800 series) routers work. The binary IOS image used to boot the router is stored on the Flash filesystem, which is a type of memory very similar to that found inside a USB thumbdrive. The startup configuration file is stored on a separate filesystem, NVRAM. The contents of both filesystems can be viewed with the dir command.

Router# dir flash:
Directory of flash:/

1 -rw- 23587052 Jan 9 2010 17:16:58 +00:00 c181x-advipservicesk9-mz.124-24.T.bin
2 -rw- 600 Sep 26 2010 07:28:12 +00:00 vlan.dat

128237568 bytes total (104644608 bytes free)
Router# dir nvram:
Directory of nvram:/

189 -rw-        1396                      startup-config
190 ----          24                      private-config
191 -rw-        1396                      underlying-config
    1 -rw-           0                      ifIndex-table
    2 -rw-         593                      IOS-Self-Sig#3401.cer
    3 ----          32                      persistent-data
    4 -rw-        2945                      cwmp_inventory
   21 -rw-         581                      IOS-Self-Sig#1.cer

196600 bytes total (130616 bytes free)

The resilient image and configuration features are enabled with one command each.

Router(config)# secure boot-image
Router(config)#
%IOS_RESILIENCE-5-IMAGE_RESIL_ACTIVE: Successfully secured running image

Router(config)# secure boot-config
Router(config)#
%IOS_RESILIENCE-5-CONFIG_RESIL_ACTIVE: Successfully secured config archive [flash:.runcfg-20101017-020040.ar]

The combination of the secured IOS image and configuration file is referred to as the bootset. We can verify the secure configuration with the command show secure bootset.

Router# show secure bootset
IOS resilience router id FHK110913UQ

IOS image resilience version 12.4 activated at 02:00:30 UTC Sun Oct 17 2010
Secure archive flash:c181x-advipservicesk9-mz.124-24.T.bin type is image (elf) []
file size is 23587052 bytes, run size is 23752654 bytes
Runnable image, entry point 0x80012000, run from ram

IOS configuration resilience version 12.4 activated at 02:00:41 UTC Sun Oct 17 2010
Secure archive flash:.runcfg-20101017-020040.ar type is config
configuration archive size 1544 bytes

At this point, we notice that our IOS image file on Flash is now hidden.

Router# dir flash:
Directory of flash:/

2 -rw- 600 Sep 26 2010 07:28:12 +00:00 vlan.dat

128237568 bytes total (104636416 bytes free)

Restoring an Archived Configuration

Now suppose that the router's startup configuration file is erased (accidentally or otherwise) and the router is reloaded. Naturally, it boots with a default configuration. The resilient configuration feature will even appear to be disabled.

Router# erase startup-config
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete

Router# show startup-config
startup-config is not present
Router# reload

System configuration has been modified. Save? [yes/no]: n
Proceed with reload? [confirm]
...
Router> enable
Router# show secure bootset
%IOS image and configuration resilience is not active

To restore our original configuration, we simply have to extract it from the secure archive and save it to Flash. Next, we can replace the current running configuration with the archived config using the configure replace command.

Router(config)# secure boot-config restore flash:archived-config
ios resilience:configuration successfully restored as flash:archived-config
Router(config)# ^C

Router# configure replace flash:archived-config
This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ? [no]: y
Total number of passes: 1
Rollback Done

Router#

Don't forget to save the running configuration once the restoration is complete (copy run start).

Be aware that the resilient configuration file is not automatically updated along with the startup configuration. To update it, you must first delete the existing resilient configuration and issue the secure boot-config command again.

Router(config)# no secure boot-config
%IOS_RESILIENCE-5-CONFIG_RESIL_INACTIVE: Disabled secure config archival [removed
flash:.runcfg-20101017-020040.ar]

Router(config)# secure boot-config
%IOS_RESILIENCE-5-CONFIG_RESIL_ACTIVE: Successfully secured config archive
[flash:.runcfg-20101017-024745.ar]
Finally, note that the secure bootset features can only be disabled from the console line.

Router(config)# no secure boot-config
%You must be logged on the console to apply this command

In fact, attempting to disable either part of the secure bootset generates a handy syslog message to alert administrators:

%IOS_RESILIENCE-5-NON_CONSOLE_ACCESS: Non console configuration request denied for command "no secure boot-config "

What About the IOS Image?

It turns out that the secure boot image feature works pretty well too. Here we can see that it persists even when the Flash filesystem appears to have been formatted.

Router# format flash:
Format operation may take a while. Continue? [confirm]
Format operation will destroy all data in "flash:". Continue? [confirm]
Writing Monlib sectors...
Monlib write complete

Format: All system sectors written. OK...

Format: Total sectors in formatted partition: 250848
Format: Total bytes in formatted partition: 128434176
Format: Operation completed successfully.

Format of flash: complete
Router# dir
Directory of flash:/

No files in directory

128237568 bytes total (104640512 bytes free)
Router# reload
Proceed with reload? [confirm]

*Oct 17 02:37:37.127: %SYS-5-RELOAD: Reload requested by console. Reload Reason
: Reload Command.
System Bootstrap, Version 12.3(8r)YH8, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.
C1800 platform with 131072 Kbytes of main memory with parity disabled

Upgrade ROMMON initialized
program load complete, entry point: 0x80012000, size: 0xc0c0

Initializing ATA monitor library.......
program load complete, entry point: 0x80012000, size: 0xc0c0

Initializing ATA monitor library.......

program load complete, entry point: 0x80012000, size: 0x167e724
Self decompressing the image : #################################################
################################################################################
################################################################ [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 12.4(24)T,
RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 26-Feb-09 03:22 by prod_rel_team
...
Router> enable
Password:
Router# dir
Directory of flash:/

No files in directory

128237568 bytes total (104640512 bytes free)
Router# show version
Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 12.4(24)T,
RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 26-Feb-09 03:22 by prod_rel_team
...

Taken From: http://packetlife.net/blog/2010/oct/18/ios-resilient-configuration/

Saturday, April 11, 2009

Recovering Data From Disks With Bad Sectors

Hack and / - When Disaster Strikes: Hard Drive Crashes

All is not necessarily lost when your hard drive starts the click of death. Learn how to create a rescue image of a failing drive while it still has some life left in it.

The following is the beginning of a series of columns on Linux disasters and how to recover from them, inspired in part by a Halloween Linux Journal Live episode titled “Horror Stories”. You can watch the original episode at www.linuxjournal.com/video/linux-journal-live-horror-stories.

Nothing teaches you about Linux like a good disaster. Whether it's a hard drive crash, a wayward rm -rf command or fdisk mistakes, there are any number of ways your normal day as a Linux user can turn into a nightmare. Now, with that nightmare comes great opportunity: I've learned more about how Linux works by accidentally breaking it and then having to fix it again, than I ever have learned when everything was running smoothly. Believe me when I say that the following series of articles on system recovery is hard-earned knowledge.

Treated well, computer equipment is pretty reliable. Although I've experienced failures in just about every major computer part over the years, the fact is, I've had more computers outlast their usefulness than not. That being said, there's one computer component you can almost count on to fail at some point—the hard drive. You can blame it on the fast-moving parts, the vibration and heat inside a computer system or even a mistake on a forklift at the factory, but when your hard drive fails prematurely, no five-year warranty is going to make you feel better about all that lost data you forgot to back up.

The most important thing you can do to protect yourself from a hard drive crash (or really most Linux disasters) is back up your data. Back up your data! Not even a good RAID system can protect you from all hard drive failures (plus RAID doesn't protect you if you delete a file accidentally), so if the data is important, be sure to back it up. Testing your backups is just as important as backing up in the first place. You have not truly backed up anything if you haven't tested restoring the backup. The methods I list below for recovering data from a crashed hard drive are much more time consuming than restoring from a backup, so if at all possible, back up your data.

Now that I'm done with my lecture, let's assume that for some reason, one of your hard drives crashed and you did not have a backup. All is not necessarily lost. There are many different kinds of hard drive failure. Now, in a true hard drive crash, the head of the hard drive actually will crash into the platter as it spins at high speed. I've seen platters after a head crash that are translucent in sections as the head scraped off all of the magnetic coating. If this has happened to you, no command I list here will help you. Your only recourse will be one of the forensics firms out there that specialize in hard drive recovery. When most people say their hard drive has crashed, they are talking about a less extreme failure. Often, what has happened is that the hard drive has developed a number of bad blocks—so many that you cannot mount the filesystem—or in other cases, there is some different failure that results in I/O errors when you try to read from the hard drive. In many of these circumstances, you can recover at least some, if not most, of the data. I've been able to recover data from drives that sounded horrible and other people had completely written off, and it took only a few commands and a little patience.

Create a Recovery Image

Hard drive recovery works on the assumption that not all of the data on the drive is bad. Generally speaking, if you have bad blocks on a hard drive, they often are clustered together. The rest of the data on the drive could be fine if you could only access it. When hard drives start to die, they often do it in phases, so you want to recover as much data as quickly as possible. If a hard drive has I/O errors, you sometimes can damage the data further if you run filesystem checks or other repairs on the device itself. Instead, what you want to do is create a complete image of the drive, stored on good media, and then work with that image.

A number of imaging tools are available for Linux—from the classic dd program to advanced GUI tools—but the problem with most of them is that they are designed to image healthy drives. The problem with unhealthy drives is that when you attempt to read from a bad block, you will get an I/O error, and most standard imaging tools will fail in some way when they get an error. Although you can tell dd to ignore errors, it happily will skip to the next block and write nothing for the block it can't read, so you can end up with an image that's smaller than your drive. When you image an unhealthy drive, you want a tool designed for the job. For Linux, that tool is ddrescue.

ddrescue or dd_rescue

To make things a little confusing, there are two similar tools with almost identical names. dd_rescue (with an underscore) is an older rescue tool that still does the job, but it works in a fairly basic manner. It starts at the beginning of the drive, and when it encounters errors, it retries a number of times and then moves to the next block. Eventually (usually after a few days), it reaches the end of the drive. Often bad blocks are clustered together, and in the case when all of the bad blocks are near the beginning of the drive, you could waste a lot of time trying to read them instead of recovering all of the good blocks.

The ddrescue tool (no underscore) is part of the GNU Project and takes the basic algorithm of dd_rescue further. ddrescue tries to recover all of the good data from the device first and then divides and conquers the remaining bad blocks until it has tried to recover the entire drive. Another added feature of ddrescue is that it optionally can maintain a log file of what it already has recovered, so you can stop the program and then resume later right where you left off. This is useful when you believe ddrescue has recovered the bulk of the good data. You can stop the program and make a copy of the mostly complete image, so you can attempt to repair it, and then start ddrescue again to complete the image.

Prepare to Image

The first thing you will need when creating an image of your failed drive is another drive of equal or greater size to store the image. If you plan to use the second drive as a replacement, you probably will want to image directly from one device to the next. However, if you just want to mount the image and recover particular files, or want to store the image on an already-formatted partition or want to recover from another computer, you likely will create the image as a file. If you do want to image to a file, your job will be simpler if you image one partition from the drive at a time. That way, it will be easier to mount and fsck the image later.

The ddrescue program is available as a package (ddrescue in Debian and Ubuntu), or you can download and install it from the project page. Note that if you are trying to recover the main disk of a system, you clearly will need to recover either using a second system or find a rescue disk that has ddrescue or can install it live (Knoppix fits the bill, for instance).

Run ddrescue

Once ddrescue is installed, it is relatively simple to run. The first argument is the device you want to image. The second argument is the device or file to which you want to image. The optional third argument is the path to a log file ddrescue can maintain so that it can resume. For our example, let's say I have a failing hard drive at /dev/sda and have mounted a large partition to store the image at /mnt/recovery/. I would run the following command to rescue the first partition on /dev/sda:

$ sudo ddrescue /dev/sda1 /mnt/recovery/sda1_image.img
/mnt/recovery/logfile
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:        0 B,  errsize:   0 B,  errors:       0
Current status
rescued:  349372 kB,  errsize:   0 B,  current rate: 19398 kB/s
  ipos:  349372 kB,   errors:   0,    average rate: 16162 kB/s
  opos:  349372 kB

Note that you need to run ddrescue with root privileges. Also notice that I specified /dev/sda1 as the source device, as I wanted to image to a file. If I were going to output to another hard drive device (like /dev/sdb), I would have specified /dev/sda instead. If there were more than one partition on this drive that I wanted to recover, I would repeat this command for each partition and save each as its own image.

As you can see, a great thing about ddrescue is that it gives you constantly updating output, so you can gauge your progress as you rescue the partition. In fact, in some circumstances, I prefer using ddrescue over dd for regular imaging as well, just for the progress output. Having constant progress output additionally is useful when considering how long it can take to rescue a failing drive. In some circumstances, it even can take a few days, depending on the size of the drive, so it's good to know how far along you are.

Repair the Image Filesystem

Once you have a complete image of your drive or partition, the next step is to repair the filesystem. Presumably, there were bad blocks and areas that ddrescue could not recover, so the goal here is to attempt to repair enough of the filesystem so you at least can mount it. Now, if you had imaged to another hard drive, you would run the fsck against individual partitions on the drive. In my case, I created an image file, so I can run fsck directly against the file:

$ sudo fsck -y /mnt/recovery/sda1_image.img

I'm assuming I will encounter errors on the filesystem, so I added the -y option, which will make fsck go ahead and attempt to repair all of the errors without prompting me.

Mount the Image

Once the fsck has completed, I can attempt to mount the filesystem and recover my important files. If you imaged to a complete hard drive and want to try to boot from it, after you fsck each partition, you would try to mount them individually and see whether you can read from them, and then swap the drive into your original computer and try to boot from it. In my example here, I just want to try to recover some important files from this image, so I would mount the image file loopback:

$ sudo mount -o loop /mnt/recovery/sda1_image.img /mnt/image

Now I can browse through /mnt/image and hope that my important files weren't among the corrupted blocks.

Method of Last Resort

Unfortunately in some cases, a hard drive has far too many errors for fsck to correct. In these situations, you might not even be able to mount the filesystem at all. If this happens, you aren't necessarily completely out of luck. Depending on what type of files you want to recover, you may be able to pull the information you need directly from the image. If, for instance, you have a critical term paper or other document you need to retrieve from the machine, simply run the strings command on the image and output to a second file:

$ sudo strings /mnt/recovery/sda1_image.img >
/mnt/recovery/sda1_strings.txt

The sda1_strings.txt file will contain all of the text from the image (which might turn out to be a lot of data) from man page entries to config files to output within program binaries. It's a lot of data to sift through, but if you know a keyword in your term paper, you can open up this text file in less, and then press the / key and type your keyword in to see whether it can be found. Alternatively, you can grep through the strings file for your keyword and the surrounding lines. For instance, if you were writing a term paper on dolphins, you could run:

$ sudo grep -C 1000 dolphin /mnt/recovery/sda1_strings.txt >
/mnt/recovery/dolphin_paper.txt

This would not only pull out any lines containing the word dolphin, it also would pull out the surrounding 1,000 lines. Then, you can just browse through the dolphin_paper.txt file and remove lines that aren't part of your paper. You might need to tweak the -C argument in grep so that it grabs even more lines.

In conclusion, when your hard drive starts to make funny noises and won't mount, it isn't necessarily the end of the world. Although ddrescue is no replacement for a good, tested backup, it still can save the day when disaster strikes your hard drive. Also note that ddrescue will work on just about any device, so you can use it to attempt recovery on those scratched CD-ROM discs too.

Kyle Rankin is a Senior Systems Administrator in the San Francisco Bay Area and the author of a number of books, including Knoppix Hacks and Ubuntu Hacks for O'Reilly Media. He is currently the president of the North Bay Linux Users' Group.

Taken From: Linux Journal, Issue 179, March 2009 - Hack and / - When Disaster Strikes: Hard Drive Crashes