Database Migrations the VxVM Way

Rainer Raab

The duties of a Unix systems administrator range from the simple and mundane, to the complex and mission-critical. We are often called upon to reset a password, remove an unwanted file, restore a backup, apply security patches, etc. Though not very challenging, these tasks are important and must be performed with vigor. Less frequently, we are enlisted to perform a challenging task, offering us a chance to learn something new and prove our worth. Such tasks include operating system upgrades, building Web server farms, allocating new disk arrays, and database migrations, to name a few. Database migrations, which are the most complex and mission critical, offer the most reward and the opportunity to add new skills to one's oeuvre.

Migration Methods

The term database migration can mean a variety of things. It can refer to the movement from one database to another where the data is moved between the databases, such as moving from an Informix IDS version 7.31 to a new IDS version 9.40 database using Informix dbexport/dbimport utilities. Or, it can refer to the movement of the database to an entirely new platform, such as moving from Solaris 2.6 on SPARC to Linux 2.6 or Windows Server 2003 on Intel x86. It can refer to the movement of the database from one server to another, such as moving from a Sun Enterprise E450 to a Sun Fire 4800 system. Or, it can simply mean moving the database from one disk array to another.

The operative word here is "move". An in-place upgrade of a database, from one version to another, is not considered a database migration. For example, upgrading an Informix IDS version 7.31 to IDS version 9.30 would not be considered a database migration.

Database migrations are initiated for a variety of reasons. Sometimes they are done for increased performance. If database loads or nightly database refreshes are taking too long, then a new server with more or faster CPUs may help. Sometimes they are done for data reorganization. Perhaps disk hot spots are leading to poor performance. Sometimes migrations are done as part of server consolidation projects, where entire departments are asked to move their databases to a single server. Often, it's just a simple matter of economics. The original environment may have become too costly due to high maintenance costs, or the new environment may offer lower software licensing costs.

A number of database migration methods can be employed. The DBA can unload the data to a transportable file format and recreate the database from scratch on the destination system. If the new environment supports the database's data files, the files can be archived and copied to the target system using a slew of Unix utilities (e.g., tar, cpio, pax, etc.). If the database data files are stored on raw devices, the Unix dd command can be used to pipe the data to the target system. If the raw devices are managed by a logical volume manager (LVM), such as Veritas Volume Manager (VxVM), then the data may be mirrored to new devices on a new array and then physically moved to the target system. I'll demonstrate this last method, using Veritas Volume Manager, to quickly and reliably migrate a database.

VxVM Migration Prerequisites

Veritas Volume Manager, available for most Unix platforms, has become the de facto LVM in many shops because of its advanced features and standardized command set across platforms. To provide a brief overview, VxVM allows the creation of volumes, which are logical devices that appear to the operating system as a type of hard disk or disk partition (i.e., a virtual disk). Volumes can be constructed from one disk to many disks supporting various RAID levels (RAID-0, RAID-1, RAID-5, etc.) as well as simple disk concatenation. The advantages of volumes include increased storage capacity beyond single disk, various degrees of data protection, increased read/write performance, and ease of storage management to name a few.

Successful database migrations with VxVM require careful planning and preparation, but the reward is well worth the effort. Before the migration can begin, the DBA must determine the feasibility of the migration, since not all migrations can be performed with Veritas Volume Manager. There are several prerequisites for performing a migration with VxVM.

First, the database should to be built from raw device volumes. If it isn't, forget about using VxVM for the migration. Instead, use any of the supplied database utilities or one of the aforementioned Unix archive/copy utilities. Second, does the target database support the original database's data files? If the migration is to a minor version upgrade of the database on the same platform, then this is most likely the case. However, if the migration is to a new major version of the database, then the DBA may need to consult with the database vendor first. In any event, if the new version of the database doesn't directly support the database files, VxVM may still be used for the migration and an in-place upgrade on the migrated database can be performed. Unfortunately, a VxVM database migration to a new platform, such as from Solaris to Windows, or to a new database platform, such as from Informix to Oracle, is probably not possible.

If you've satisfied the VxVM database migration prerequisites, then a database migration the VxVM way might just be for you.

Setting the Stage

So, you've outgrown your current server and decided to purchase a new, more powerful replacement. Your current server is hosting a multi-terabyte database on an old disk array and you need additional disk space for growth, so you've decided to purchase a new disk array as well. By purchasing both a new server and new storage, you've set the stage to performing a database migration. By performing the migration to a new server, you can continue to host the database on the old server while copying/moving the data to a new, more powerful server with more storage and without much downtime.

You've also consulted with the DBA, and he has satisfied all of the VxVM migration prerequisites and has a game plan. The DBA is going to stick with the same major version of the database software, but with a minor upgrade. You've decided that the new server will be running a new version of the operating system, and the DBA has confirmed that the database software is supported. The plan is to copy the database to the new server and bring it online with the same name, allowing for two copies of the database to exist at once. This will make migrating to the new database easier and transparent to the users.

Mirroring the Volumes

With some careful planning, you've attached the new disk array to the old server and configured the storage for use by the operating system. Because the new disk array has more capacity than the old array, disk space will not be an issue. To copy the data from the old array to the new, you must add a new disk (LUN) to each of the original VxVM disk groups, from the new array. Because the new LUNs are much larger, you should initialize the LUNs, soon to be christened VM disks by VxVM, with a large VxVM private region using the vxdisksetup command:

vxdisksetup -i <new_device> privlen=8192
example: vxdisksetup -i c3t8d0 privlen=8192

After initializing the disks, add the disks to the VxVM disk groups with the vxdg command:

vxdg -g <old_dg> adddisk <new_vm_disk>=<new_device>
example: vxdg -g idsdg1 adddisk d2_lun1=c3t8d0

If the volumes to be mirrored are simple volumes, you can use the vxmirror command:

vxmirror -g <old_dg> <old_vm_disk> <new_vm_disk>
example: vxmirror -g idsdg1 s1_lun1 d2_lun1

Breaking the Mirrors

When all of the volumes have been successfully mirrored, the next step is to split or "break" them into two. It's a good idea to get the DBA involved to schedule a database outage and shut down the database before breaking the mirrors. You don't want changes to be made to the database after you have broken the mirrors. You could break the mirrors while the databases are online, but then you would have to keep track of the changes and apply them manually later.

Breaking the mirrors is a cumbersome process because you need to run the vplex command for each of the mirrored volumes:

vxplex -g <old_dg> dis <new_volume>
example: vxplex -g idsdg1 dis pdf11282004-02

function make_vols {
dg=idsdg1
metadata=/var/tmp/$dg.config

plexes=$(vxprint -g $dg -mte '"d2_lun1" in (sd_disk)'|grep pl_name|awk -F= '{print $2}')
for plex in $plexes; do
echo "Disassociating $plex from disk group $dg"
#vxplex -g $dg dis $plex
volume_tmp=$(echo $plex|sed -e 's/-0[0-9]*$//')
volume=$(echo $volume_tmp"_d2")
echo "Creating new volume $volume using plex $plex"
#vxmake -g $dg -U gen vol $volume plex=$plex
echo "Extracting volume $volume metadata and appending it to $metadata"
#vxprint -hmvpsQqr -g $dg $volume >> $metadata
echo " "
done
}

Note that function not only breaks the mirrors, but that it also creates new volumes (the volume names are appended by a "_d2" to avoid conflicting with the existing volumes) from the disassociated plexes on the new disk (it will become apparent later why we needed to create new volumes in the first place). Also, the script extracts all of the VxVM volume metadata to a flat file, which will be used later. Run the function for each of your disk groups, until all of the mirrors have been broken.

There is a caveat with extracting the metadata. I've noticed that the permissions on the volumes do not get preserved. I'll present a ksh function to correct this problem.

Deporting the Disk Groups

When all of the mirrors have been successfully split, the next step is to delete the newly created volumes. Don't be alarmed -- we'll restore the volumes later from the metadata we extracted earlier. This step is necessary, because we must create new disk groups in which to store our new volumes. We can then export the new disk groups to our new server, leaving the old disk groups untouched.

Use this ksh function to remove the volumes:

function remove_vols {
dg=idsdg1
metadata=/var/tmp/$dg.config

volumes=$(grep "^vol " $metadata|awk '{print $2}')
for volume in $volumes; do
        echo "Removing volume $volume from $dg"
#vxedit -g $dg -r rm $volume
        echo " "
done
}

vxdg -g <old_dg> rmdisk <new_vm_disk>
example: vxdg -g idsdg1 rmdisk d2_lun1

vxdg init <new_dg> <new_vm_disk>=<new_device>
example: vxdg init idsdg1_d2 d2_lun2=c3t8d0

vxmake -g <new_dg> -d <dg_meta_data>
example: vxmake -g idsdg1_d2 -d /var/tmp/idsdg1.config

After you've created the new disk groups with the restored volumes, all that's left is to deport the new disk groups to the new server:

vxdg -n <old_dg> deport <new_dg>
example: vxdg -n idsdg1 deport idsg1_d2

Importing the Disk Groups

Once all of the new disk groups have been successfully deported, disconnect the new array from the old server and attach it to the new server. You'll have to go through the motions of making the disk array visible on the new server. Don't fret about integrity of the data. It's safely stored on the VxVM disks. Don't worry about device renumbering either (e.g., the cxtxdx name changing on the new server), because VxVM tracks disks by the information stored in the private region on the disk and not by the operating system device name.

Once you feel confident that all of the LUNs on the disk array are accounted for and visible to the operating system, import the disk groups:

vxdg import <old_dg>
example: vxdg import idsdg1

Import all of the formerly deported disk groups, using the original name of the disk group on the old server (remember we deported the disk groups with the rename option). Once all of the disk groups have been imported, don't forget to rename to volumes back to their original names. If you used the make_vols function, it appended a "_d2" (or whatever value you chose) to the end of the new volume names). Use this ksh function to rename the volumes:

function rename_vols {
dg=idsdg1
volumes=$(vxprint -g $dg -mte v_kstate|grep "^vol "|awk '{print $2}')
for volume in $volumes; do
        new_volume=$(echo $volume|sed -e 's/_d2$//')
        echo "Renaming volume $volume to $new_volume in disk group $dg"
        #vxedit -g $dg rename $volume $new_volume
        echo " "
done
}

Before starting the volumes, make sure that the permissions on the volumes are correct. I've noticed that VxVM is not consistent in restoring the owner and group id from the metadata. This is critical for a database because the volumes must be owned by the database id (e.g., Informix or Oracle). Use this ksh function to correct the problem:

function fix_vols {
dg=idsdg1
volumes=$(vxprint -g $dg -mte v_kstate|grep "^vol "|awk '{print $2}')
for volume in $volumes; do
        echo "Changing ownership and mode on $volume in disk group $dg"
        #vxedit -g $dg set mode=0660 user=informix group=informix $volume
        echo " "
done
}

#vxrecover -g <old_dg> -Esb
example: vxrecover -g idsdg1 -Esb

For example, for the following Informix IDS root dbspace (rootdbs), the volume /ids_prod1/rootdbs is really a symbolic link to the raw volume:

# onstat -d|grep "/rootdbs"
2ac06928 1   1   0     1000000  997130    PO-   /ids_prod1/rootdbs

# ls -la /ids_prod1/rootdbs
lrwxrwxrwx   1 informix   informix   33 Jun 10  2003 \
  /ids_prod1/rootdbs -> /dev/vx/rdsk/idsdg1/rootdbs_prod1

If the database does use symbolic links, the links must be recreated exactly. VxVM preserves the device names and consistently stores the devices (volumes) in the /dev/vx directory (even across platforms). If you copied the symbolic links correctly from the old server to the new, the links will point to the right volumes.

Once the symbolic links have been verified as correct, the DBA can install the database software and copy over any configuration files needed from the old server to bring the database online. Once the database is online and the DBA is satisfied with the results, you can put another feather in your cap and call it a day.

Conclusion

It has come to my attention that Veritas Volume Manager for Unix (versions 3.2 and above), includes several new features that automatically perform some of the VxVM functions/commands I presented. Specifically, the vxdg command has been enhanced to allow you to move, join, and split disk groups. These enhancements will allow you to more easily perform a database migration, but they do require an extra license. It's probably worth it to purchase the license, but it doesn't hurt to know the sordid details anyway.

The database migration method I presented using Veritas Volume Manager is one that I have successfully used several times in the past. It may not be as efficient for migrating smaller databases, since there are many steps to perform, but it is well worth the effort if you have a very large database to migrate. The methods presented can be applied to other types of data migration, not just databases. Perhaps you will find some new uses and pass them along.

References

Rockwood, Ben. The Cuddletech Veritas Volume Manager Series: Advanced Veritas Theory. August 10, 2002. http://www.cuddletech.com/veritas/advx/index.html (March 28, 2004)

Veritas Software Corporation. Veritas Volume Manager 3.1.1 Administrator's Guide. February 2001.

Veritas Software Corporation. How to move a disk between disk groups: TechNote ID: 182857. October 9, 2002. http://seer.support.veritas.com/docs/182857.htm (March 28, 2004)

Rainer Raab is a Senior UNIX System Administrator consulting for Wells Fargo Bank. He received his MBA in Computer Information Systems from California State University at Hayward. When he's not tinkering around with Linux, he's spending time with his lovely wife and adorable newborn daughter. He can be reached at: hopfen@sbcglobal.net.