Restoring Dirvish Disk-to-Disk Backups: Part III

Keith Lofstrom

Dirvish is a disk-to-disk backup program, a Perl wrapper around the rsync file system network copy program. Using Unix-style hard links, dirvish and rsync allow successive backup images to share identical data files and occupy the same disk space, greatly reducing backup time and backup disk usage.

The first article in this series about dirvish (in the January Sys Admin) described how I inherited this open source program and where I plan to take it. The second article in February described how I set up dirvish for my Linux systems. These articles are available on the Sys Admin Web site:

http://www.samag.com/documents/s=9464/sam0501b/
http://www.samag.com/documents/s=9494/sam0503e/

Restores are unpredictable; you never know when you will need to restore a file or a whole drive, you won't have time for it, and you will have to complete the task very rapidly. This is a lousy time to learn a new set of tools, and the stress of the restore can lead to unrecoverable mistakes. So, while you are designing your backup system, it is a good idea to think about the restores you will someday need to do. You should create some restore scripts like the examples below to help you to deal with disasters.

Dirvish Single File Restores

You will rarely need to restore an entire disk drive; most of the time, you will only be restoring a single file or group of files. This may be because of user mistakes, program flaws, or enemy action.

At the time of writing, dirvish lacks a Web page-based automatic utility for user-controlled restoration of files. We hope to add this feature soon, after solving some difficult security and verification issues.

It would be easy to give users read access to the backup vaults on the backup server; dirvish stores backup information with the same UID/GID as the backup client machine. If your UID/GID scheme is identical across all your machines, and you trust your users to be very careful, they could log into the backup server, retrieve their files with sftp or rsync, and not bother you.

But in real life, the UID/GIDs are not always identical, and you should not trust users to access the backup server unsupervised. If you are as paranoid as I am, the backup server offers no external access to the local network, and the backup disk is left unmounted when it is not being used for backups. Restore programs can be corrupted and may have security holes. It is difficult to design an automated restore process that can verify that a request for a backed up file is legitimate. If you delete a user's account for security violations, for example, that user should not be able to use the file restore process to reverse the account deletion. Human judgment is still needed.

If a legitimate user loses or corrupts a file, instruction and guidance may be needed to prevent a recurrence. Improper use of the restore process can make things worse. A file deletion may stem from an improperly configured application, or hostile action by unauthorized outsiders. There are many reasons for an experienced systems administrator to look around before restoring a file.

For now, we rely on the judgment of the systems administrator to prevent unauthorized access to the backups, to verify that the correct files are being restored, to make sure that the restored files do not overwrite the user's live files, and to keep snoopers from looking at either the existing or the restored versions of the files.

Consider a typical sequence of events. User fgump has maintained files in directory foo on his laptop Jane for some weeks. The dirvish backups on server Mom contain numerous copies of this directory and its files, including the backup copy made early this morning, March 5, 2005.

fgump makes a mistake while editing file foo/fum.c, wiping out half the file. There are still valuable edits in the remainder of fum.c; we want to save as much of that work as we can. With a copy of fum.c from this morning's backup, fgump can merge the versions and resume work.

Your first task as systems administrator is to check the damage. Connect to Jane with ssh, look for the file, and check to make sure there is not already a usable copy of the damaged file in a dot file or in /tmp. See where you can put a restore directory if necessary.

Now log into the backup server Mom through the console port or KVM switch (No external ssh, right?). You probably already rotated this morning's backup drive into the fire-resistant safe, so you will have to unmount tomorrow's drive and put this morning's drive back. Mount the morning backup partition.

You can find all the versions of the file fum.c in the jane-home vault with:

dirvish-locate jane-home foo/fum\.c$

2 matches in 110 images
/home/fgump/src/foo/fum.c
    Feb 24 15:48 2005-0305-0300, 2005-0303-0300, 2005-0301-0300
                 2005-0227-0300, 2004-0225-0300

 /home/fgump/junk/foo/fum.c
    Jan  4 11:15 2005-0105-0300

/backup/dirvish/jane/jane-home/2005-0305-0300/tree/fgump/src/foo/fum.c
/backup/dirvish/jane/jane-home/2005-0303-0300/tree/fgump/src/foo/fum.c
... etc .

#!/bin/bash
srctree=/backup/dirvish/jane/jane-home/2005-0305-0300/tree
srcfile=/fgump/src/foo/fum.c
des=jane:/home/fgump/src/foo/fum.c
rsync -vvrltHpgoDx --backup --numeric-ids $srctee$srcfile $dst

To restore directory foo and its contents, you would do this:

#!/bin/bash
src=/backup/dirvish/jane/jane-home/2005-0305-0300/tree/fgump/src/foo
des=jane:/home/fgump/src
rsync -vvrltHpgoDx --backup --numeric-ids $src $dst

#!/bin/bash
src=/backup/dirvish/jane/jane-home/2005-0305-0300/tree/fgump/src/foo
des=jane:/home/fgump/src/2005-0305-0300

   rsync -vvrltHpgoDx --backup-dir --numeric-ids $src $dst

File and directory restore can be useful for other tasks. The dirvish.org Web site includes a wiki, allowing dirvish users and developers to change the content with just a Web browser. Occasionally, the wiki is attacked by a wiki spammer, and a long list of gambling and pornographic HTML links are added to a page so that search engine Web crawlers will find the links and increase their ranking for searches.

Users of the wiki find and correct this damage quickly, but the links remain in the version tree for the crawlers to find. It is easy to use rsync to restore appropriate parts of the wiki file tree to their pre-spam state, erasing the vandalism of the wiki-spammers. Using rsync to restore large chunks of the wiki is faster than laboriously editing individual pages and cvs trees.

These restore tasks can be simplified by Perl scripts that are aware of where the backups are stored by dirvish. I plan to add a "dirvish-file-restore" program to the dirvish toolkit, allowing a systems administrator to type something like:

dirvish-file-restore --image 2005-0305-0300 jane:/home/fgump/src/foo/foo.c

Full disk restore requires one more script, per client, which we will write by hand. It is conceivable to build an automated restore process that exactly duplicates the client drive, but in most circumstances you don't want to do that. The replacement drive may be a different size; you may want to merge partitions or change their size. This is easy with a disk-to-disk, image-to-image restore.

If you are restoring after the system has been compromised by script kiddies or other enemy action, you may want to restore a hybrid image that combines yesterday's user data, last week's system files, plus security updates from the Web. With all the data available in multiple images on one disk (after verifying that the backup images themselves have not been attacked!), you can modify the backup scripts to pick and choose from the images, then build a single composite image from which to boot.

While it would be convenient to automate the restore process, you should be very careful about making restores too easy. For example, some tools like systemimager can be used to restore a client drive using a BOOTP-style process. However, if implemented carelessly, this technique can allow an untrusted machine on the network to get a copy of the image of a different machine. The only trusted machine on the network is the backup server, and any full-disk-restore process should be initiated and controlled by a conscious decision of the administrators of both the backup and the client machines.

In the February article on dirvish, I showed how to build a backup drive with a full distro in the first 5-GB partition. So, now we can boot from that drive, access the backups on the same drive, download security upgrades from the Internet, etc. This saves precious minutes if the server is compromised or damaged. The following example assumes we are restoring the main hard drive for the backup server Mom; you can extrapolate to other clients.

The restore process for Mom, using the script below, should be tested with a clean new restore target drive, as large or larger than the original. This test drive will be mounted in a swap tray and inserted into the /dev/sda USB2 cage, while the bootable backup drive goes into the boot cage. Boot Mom from the backup drive.

The backup partition is /dev/hda3. Mount that as /restore. As part of the restore, we saved the output of sfdisk with the backups. If you are building a new main drive that matches the old one, you do not need to do much editing to the sfdisk.out file, but let's look at the file anyway. Here is a copy of Mom's sfdisk.out file in /backup/dirvish/mom/mom-root/2005-0315-0300/sfdisk.out:

# partition table of /dev/hda
unit: sectors
 
/dev/hda1 : start=       63, size=  2048193, Id=83, bootable
/dev/hda2 : start=  2048256, size=103652640, Id= 5
/dev/hda3 : start=        0, size=        0, Id= 0
/dev/hda4 : start=        0, size=        0, Id= 0
/dev/hda5 : start=  2048319, size= 20480481, Id=83  
/dev/hda6 : start= 22528863, size= 20480481, Id=83
/dev/hda7 : start= 43009407, size= 58594977, Id=83
/dev/hda8 : start=101604447, size=  4096449, Id=82

The sfdisk.out file can be modified (very carefully) by hand to change the sizes or number of partitions. Or, you can just print out the sfdisk.out and df.out files and use a tool like fdisk to build the new disk partitions. However, if you are restoring to a similar-sized disk and want to use same-sized partitions, you can just feed the sfdisk.out file right back into sfdisk, as shown in the restore script below.

Here is an example restore script, to duplicate the main disk for the backup server Mom:

#!/bin/bash
# /usr/local/sbin/recover.sda

# there can be more than one backup image directory
BDIR1=2005-0312-0300
BDIR2=2005-0315-0300

S=/backup/dirvish/mom
T=/new
DISK=/dev/sda
SFD=$S/sfdisk.out
 
MKDIR='/bin/mkdir '
MKFS='/sbin/mkfs.ext3 '
MOUNT='/bin/mount '
ECHO='/bin/echo '
COPY='/usr/bin/cp -a'
E='tree/'

# If you have not partitioned the target disk already,
# and want to use the same disk structure as before,
# uncomment the line below:

#/bin/cat $SFD | sfdisk --force $DISK


# set up partitions, THIS IS VERY DISK DEPENDENT
$MKFS  ${DISK}1
$MKFS  ${DISK}5
$MKFS  ${DISK}6
$MKFS  ${DISK}7

# make the swap partition
/sbin/mkswap     ${DISK}8

#mount point for new disk

$MKDIR       $T 

# mount and fill partitions, THIS IS VERY DISK DEPENDENT

$ECHO        "now copying root"
$MOUNT       ${DISK}5                 $T
$COPY        $S/mom-root/$BDIR1/$E    $T

$ECHO        "now copying boot"
$MOUNT       ${DISK}1                 $T/boot
$COPY        $S/mom-boot/$BDIR1/$E    $T/boot

$ECHO        "now copying usr"
$MOUNT       ${DISK}6                 $T/usr
$COPY        $S/mom-usr/$BDIR1/$E     $T/usr

$ECHO        "now copying home"
$MOUNT       ${DISK}7                 $T/home
$COPY        $S/mom-home/$BDIR2/$E    $T/home

# make the proc mount point

$MKDIR                                $T/proc

# install the grub boot loader 

/sbin/grub --batch --device-map=/dev/null << EOF
device (hd1) ${DISK}
root (hd1,0)
setup (hd1)
quit
EOF

exit 0

Make sure the target drive doesn't have anything important on it, and make sure that you are not targeting your backup drive instead! Although sfdisk is supposed to write to the disk specified by the command line, for safety's sake you may want to work from a version of the sfdisk.out file that has been explicitly edited to target /dev/sda rather than the original /dev/hda. Symbolic links can also be used to match backup and restore devices.

The script will restore disks at about 30 GB per hour; afterwards, we will have a disk in /dev/sda that is ready to be moved back into the main drive slot and booted. A modified version of the recover script listed above can be used for all open source filesystem formats.

Being able to restore a whole drive to the previous night's state by running a pre-designed shell script is empowering. You will find more courage to perform experiments with your systems, upgrade distributions, etc. You can use the backup restore script to build duplicates of client boot drives to use on experimental machines, trying out new hardware and software, testing security, or thrashing suspect hardware to look for weaknesses, all without interrupting the normal use of client systems. A well-tested restore process has many other uses besides disaster recovery.

Expires, Backup Disk Editing, and File Security

As mentioned in the February issue, dirvish includes a dirvish-expire command that eliminates old images from the backup vault. For example, if the dirvish master.conf file contains this expire rule:

expire-rule:
#       MIN     HR      DOM     MON     DOW     STRFTIME_FMT
        *       *       *       *       *       +2 months
        *       *       *       *       1       +6 months
        *       *       1-7     *       1       +never

Note that the expire rules are stored by dirvish with the image at backup time; if you change the expire rules after an image is made, the image will be expired on the original schedule.

How helpful is expire? Each evening's expire eliminates an image, which eliminates a set of directory information and perhaps some files that are uniquely in that image. On my systems, backup data grows at about 0.1% per day, on average, and about half of that represents growth of the original file systems in addition to the directory and file space used by each additional backup image. Expiring a single daily image increases available backup disk space by perhaps 0.05% -- plainly not worth the effort.

On a hypothetical larger and more active network, backing up 10 TB each night with 0.5% unique image growth, recovering the additional 50 GB used per image may be worthwhile -- that's about $25 worth of disk space per day at current prices. On the other hand, the extra server activity and extra disk wear involved may lower the value of aggressive expires. In any case, expires should probably be run on weekends, expiring many images at once when system loads are otherwise light.

You may want to edit your backup disk for other reasons. I am an engineering consultant, and my contracts often specify the removal of sensitive client data from my systems at the end of a contract. After sending CD archives of the data to the client, I remove this data from my dirvish archives. I can explore the backups with dirvish-locate then build a shell script to do the job:

#!/bin/bash
hdparm -zb 1 /dev/backup
mount /dev/backup /backup
# files removed here
rm -rf /backup/dirvish/jane/jane-home/200*/tree/fgump/xyzcorp
rm -rf /backup/dirvish/jane/jane-home/200*/tree/fgump/Maildir/xyzmail
rm -rf /backup/dirvish/dick/dick-home/200*/tree/cnerd/data/xyzcorp
# ... more files and directories ...
#
umount /dev/backup
hdparm -b 0 /dev/backup

For the above technique to work, you must keep track of where client data is stored during course of the project. You must store client-related "keeper" data, such as contract documents and billing data, elsewhere. This requires disciplined design of data storage per project. After client data is removed, there will still be a record of backed up file names in the daily backup log, stored along with the tree. It is difficult and unnecessary to edit these backup logs; although dirvish-locate will still indicate that the files are there, the sensitive data itself will be gone.

In the future, dirvish may be extended to perform an additional level of system security for client systems. If the rsync file transfer process has not been compromised (big if!), the daily images on the backup server may be compared to previous versions, and differences on critical files can raise security flags. Forensic analysis on previous images may help determine the history of a system exploit, and the permanent record of past log files that dirvish permits will also be helpful. While such additional safeguards do not eliminate the need for client tools such as tripwire, they do add an additional layer of protection.

Conclusion

Dirvish and rsync produce backups that are far more usable than tapes and other media for single file or whole disk restore. In the future, more dirvish tools will be developed to simplify these tasks, but current tools, and the modification of an array of simple shell scripts, provide systems administrators with quick responses to file corruption and hardware failure.

Dirvish is an open source work in progress, and your contributions of code, bug fixes, and documentation will help us produce more powerful versions in the future. Please join us at http://www.dirvish.org.

Keith Lofstrom (http://www.keithl.com) owns an integrated circuit design consultancy in Beaverton, Oregon. His specialty is mixed-signal and statistical design for deep submicron processes, as well as design for testability using the IEEE 1149.x standards. Keith has been using some flavor of Unix since 1980, and although he admits to a brief flirtation with DOS and Windows, he has seen the error of his ways. He is currently managing the dirvish backup program until a better leader comes along.