Cover V13, i12

Listing 1


Off-Host Backup Processing with Veritas FlashSnap

Borislav Stoichkov

Backup times and the resources associated with them are becoming more and more important in the evolving model of 24/7 application development and content management. Developers all over the world collaborate on the same projects and access the same resources that must be 100% available during the business hours of their respective time zones. This gives systems administrators very little room to completely satisfy their customers -- the developers.

Source code and content repositories contain hundreds of projects and millions of small files that require considerable amounts of time and system resources to back up. Also, data protection is a top priority that presents system and backup engineers with the question of how to effectively ensure data protection and availability in case of a disaster and, at the same time, minimize the duration and resource overhead of the process.

The Problem

My organization was faced with these very issues. One of our high-profile customers was using Interwoven's flagship product TeamSite installed on a Sun Solaris 8 server. Interwoven Teamsite's key features are content management and services, code and media versioning, collaboration and parallel development, branching of projects, transactional content deployment, etc. Developers all over the world were using the system's resources throughout the day for mission-critical tasks. As the number of projects and branches increased so did the number of files and the amount of data that needed to be backed up and protected.

Suddenly, the application was managing millions of small files and hundreds of branches and projects. Backup times were reaching 7-8 hours with the bottleneck caused by the sheer amount of files and data. Complaints were received that during the backup window the application as well as the system were becoming difficult to use and that system performance was becoming unacceptable. The customer requested a solution that would be as cheap as possible and would not require a change in their development and content management model.

From a storage perspective, the server had two internal mirrored drives for the operating system file systems under Veritas Volume Manager control. An external Fibre Channel array was attached to the machine presenting a single LUN on which the Interwoven Teamsite application was installed. The LUN had a 143-GB Veritas file system and was under Veritas Volume Manager control as well.

The idea for the solution was to take a snapshot of the application file system and use that snapshot for a backup to tape on another host. Thus, the backup window could be extended as much as needed without affecting the performance or usability of the server. File system snapshots, however, do not allow off-host processing. Given that Veritas Volume Manager was already installed and active on the machine, using its built-in volume snapshot features seemed natural. The only problems remaining were to break off the volume with the snapshot in a manner that was supported by the vendor and did not present any risks, and to minimize the time needed for the reverse operation -- syncing the snapped off mirror without mirroring a 143-GB file system from scratch, which is a long and tedious process.

Implementing FlashSnap

The resolutions to both problems are found in the Veritas FlashSnap product. FlashSnap is a license key-enabled option of the Veritas Foundation Suite solutions. The license enables the use of the FastResync and Dynamic Split and Join features of Veritas Volume Manager. With FastResync enabled on a volume, Volume Manager uses a map to keep track of which blocks are updated in the volume and in the snapshot. In time the data on the original volume changes, and the data on the snapshot volume becomes outdated.

The presence of a FastResync map ensures that in an operation where the snapshot is resynchronized with the primary volume only the modified blocks (dirty blocks) are applied. Full mirror synchronization is no longer necessary. The map is persistent across reboots because it is stored in a data change object (DCO) log volume associated with the original volume. Dynamic Split and Join allow for the volume snapshots to be placed into a separate disk group, which can be deported and imported on another host for off-host processing. The only requirement is for the disks to be visible to the designated host. At a later stage, the disk group can be re-imported on the original host and joined with the original disk group or, if necessary, with a different one.

For the implementation, additional storage was required on the storage array equal to the original amount of 143 GB. The added storage was configured into a new LUN. A new low-end server running Sun Solaris 8 (host2) was attached to the array as well. The newly added LUN (LUN1) was presented to both hosts, while the original LUN (LUN0) was only made visible on the original host (host1).

DCO Logging

Persistent FastResync requires a DCO log to be associated with the original volume. That option has been available only since Veritas Volume Manager 3.2 and disk group version 90, so the volume management software was upgraded to the latest version. The existing disk group was upgraded to the latest version as well. The FlashSnap license obtained from Veritas was installed on both hosts. For verification that the newly added license is functional, the following command can be issued:

# vxdctl license
All features are available:
A small problem arose from the fact that there was no room for a DCO log on LUN0 because all of its space was allocated for the application volume. Luckily, the file system on it was VXFS, and it was possible for the volume and the file system to be shrunk:

host1# vxresize -F vxfs -g DG1 Vol01 -20M
With that fixed, a DCO (data change object) log volume was associated with the original volume:

host1# vxprint -g DG1
dm DG101        c4t9d0s2     -    286676992 -       -        -       -

v  Vol01          fsgen          ENABLED      286636032   -   ACTIVE    - -
pl Vol01-01       Vol01          ENABLED      286656000   -   ACTIVE    - -
sd DG101-01       Vol01-01       ENABLED      286656000   0      -      - -
host1# vxassist -g DG1 addlog Vol01 logtype=dco dcologlen=1056 ndcolog=1 DG101
host1# vxprint -g DG1
dm DG101        c4t9d0s2     -    286676992 -       -        -       -

v  Vol01          fsgen          ENABLED  286636032   -       ACTIVE   -  -
pl Vol01-01       Vol01          ENABLED  286656000   -       ACTIVE   -  -
sd DG101-01       Vol01-01       ENABLED  286656000   0         -      -  -
dc Vol01_dco      Vol01            -         -        -         -      -  -
v  Vol01_dcl      gen            ENABLED  1056        -       ACTIVE   -  -
pl Vol01_dcl-01   Vol01_dcl      ENABLED  1056        -       ACTIVE   -  -
sd DG101-02       Vol01_dcl-01   ENABLED  1056        0         -      -  -
The length of the DCO log determines the level at which changes are tracked. A longer DCO log will trigger more in-depth tracking and will require less time for the snapshot to resynchronize. Increasing the log too much may cause performance overhead on the system. The default number of plexes in the mirrored DCO log volume is 2. It is recommended that the number of DCO log plexes configured equal the number of data plexes in the volume -- in our case, one.

The default size for a DCO plex is 133 blocks. A different number can be specified, but it must be a number from 33 up to 2112 blocks in multiples of 33. If the snapshot volumes are to be moved to a different disk group, the administrator must ensure that the disks containing the DCO plexes can accompany them.

Establishing a Snapshot Mirror

The next step is to enable persistent FastResync on the volume, so that sequential re-mirroring operations take considerably less than the establishment of a full mirror and are applied from the DCO log:

host1# vxvol -g DG1 set fastresync=on Vol01
host1# vxprint -g DG1 -m Vol01 | grep fastresync
The addition of LUN1 to the DG1 disk group as disk DG102 completes our preparation phase, so now we are ready to establish our snapshot:

host1# vxassist -g DG1 snapstart Vol01 alloc=DG102
This operation will establish a mirror of volume Vol01 and will add an additional DCO log object that will be in a DISABLED and DCOSNP state for use by the snapshot. The snapstart process takes a considerable amount of time, because it is a full mirror creation. The vxassist command will block until the snapshot mirror is complete. It can be placed in the background by using the -b argument to vxassist. During the snapstart phase, disk group DG1 will look like this:

v  Vol01         fsgen          ENABLED  286636032 -    ACTIVE   ATT1    -
pl Vol01-01      Vol01          ENABLED  286656000 -    ACTIVE    -      -
sd DG101-01      Vol01-01       ENABLED  286656000 0      -       -      -
pl Vol01-02      Vol01          ENABLED  286656000 -    SNAPATT  ATT     -
sd DG102-01      Vol01-02       ENABLED  286656000 0      -       -      -
dc Vol01_dco     Vol01             -        -      -      -       -      -
v  Vol01_dcl     gen            ENABLED  1056      -    ACTIVE    -      -
pl Vol01_dcl-01  Vol01_dcl      ENABLED  1056      -    ACTIVE    -      -
sd DG101-02      Vol01_dcl-01   ENABLED  1056      0      -       -      -
pl Vol01_dcl-02  Vol01_dcl      DISABLED 1056      -    DCOSNP    -      -
sd DG102-02      Vol01_dcl-02   ENABLED  1056      0      -       -      -
Once the mirror is established, the plex on disk DG102 will be in a SNAPDONE state ready to be separated from the original volume. If the snapshot is attempted before the snapshot plex is in a SNAPDONE state, the command will fail. If snapstart is placed in the background with the -b switch, the vxassist snapwait command will wait until the snapstart command is done and can be used in scripts to ensure that no other commands are issued before the completion of snapstart:

v  Vol01        fsgen           ENABLED  286636032 -    ACTIVE   -       -
pl Vol01-01     Vol01           ENABLED  286656000 -    ACTIVE   -       -
sd DG101-01     Vol01-01        ENABLED  286656000 0      -      -       -
pl Vol01-02     Vol01           ENABLED  286656000 -    SNAPDONE -       -
sd DG102-01     Vol01-02        ENABLED  286656000 0      -      -       -
dc Vol01_dco    Vol01              -        -      -      -      -       -
v  Vol01_dcl    gen             ENABLED  1056      -    ACTIVE   -       -
pl Vol01_dcl-01 Vol01_dcl       ENABLED  1056      -    ACTIVE   -       -
sd DG101-02     Vol01_dcl-01    ENABLED  1056      0      -      -       -
pl Vol01_dcl-02 Vol01_dcl       DISABLED 1056      -    DCOSNP   -       -
sd DG102-02     Vol01_dcl-02    ENABLED  1056      0      -      -       -
To execute the actual snapshot:

host1# vxassist -g DG1 snapshot Vol01 SNAP-Vol01
host1# vxprint -g DG1

v  SNAP-Vol01      fsgen          ENABLED  286636032 -     ACTIVE   -     -
pl Vol01-02        SNAP-Vol01     ENABLED  286656000 -     ACTIVE   -     -
sd Dg102-01        Vol01-02       ENABLED  286656000 0        -     -     -
dc SNAP-Vol01_dco  SNAP-Vol01        -        -      -        -     -     -
v  SNAP-Vol01_dcl  gen            ENABLED  1056      -     ACTIVE   -     -
pl Vol01_dcl-02    SNAP-Vol01_dcl ENABLED  1056      -     ACTIVE   -     -
sd DG102-02        Vol01_dcl-02   ENABLED  1056      0        -     -     -
sp Vol01_snp       SNAP-Vol01        -        -      -        -     -     -

v  Vol01           fsgen          ENABLED  286636032 -     ACTIVE   -     -
pl Vol01-01        Vol01          ENABLED  286656000 -     ACTIVE   -     -
sd DG101-01        Vol01-01       ENABLED  286656000 0        -     -     -
dc Vol01_dco       Vol01             -        -      -        -     -     -
v  Vol01_dcl       gen            ENABLED  1056      -     ACTIVE   -     -
pl Vol01_dcl-01    Vol01_dcl      ENABLED  1056      -     ACTIVE   -     -
sd Dg102-02        Vol01_dcl-01   ENABLED  1056      0        -     -     -
sp SNAP-Vol01_snp  Vol01             -        -      -        -     -     -
Now the disk group can be split so that the disk containing the snapshot volume is placed in a different group:

host1# vxdg split DG1 SNAPDG1 SNAP-Vol01
The new disk group SNAPDG1 containing SNAP-Vol01 and its DCO log volume can be deported and imported on the alternate host:

host1# vxdg deport SNAPDG1
host2# vxdg import SNAPDG1
Following the split, the snapshot volume is disabled. The following commands can be used to recover and start the volume:

host2# vxrecover -g SNAPDG1 -m SNAP-Vol01
host2# vxvol -g SNAPDG1 start SNAP-Vol01
A consistency check can be performed on the volume's file system, and it can be mounted for backup processing or any other type of data manipulation:

host2# fsck -F vxfs /dev/vx/rdsk/SNAPDG1/ SNAP-Vol01
host2# mount -F vxfs /dev/vx/dsk/SNAPDG1/ SNAP-Vol01 /data
Before the backup window kicks in, or in case the snapshot needs to be refreshed, the file system can be unmounted and the volume deported and imported again on the original host:

host2# umount /data
host2# vxvol -g SNAPDG1 stop SNAP-Vol01
host2# vxdg deport SNAPDG1
host1# vxdg import SNAPDG1
Now the disk(s) in disk group SNAPDG1 can be joined into disk group DG1:

host1# vxdg join SNAPDG1 DG1
host1# vxrecover -g SNAPDG1 -m Vol01
Once the snapshot volume is back into its original disk group, we can perform the snapback operation:

host1# vxassist -g DG1 snapback SNAP-Vol01
In some cases when there is data corruption on the original volume, the data on the snapshot volume can be used for the synchronization. This is achieved by using the resyncfromreplica argument to the vxassist -o option with snapback. The operation will not take long to execute at all. If performed within hours of the first snapshot, the process may take less than a minute depending on the amount of file system changes.

In our environment, a snapback process that is executed approximately 24 hours after the previous one takes no longer than 12 minutes. Effectively, we have decreased the time it takes to back up the application from hours to less than 15 minutes from the developers' and system users' point of view.

Automating the Process

The last challenge in this project was to automate the process so that it would occur transparently on a daily basis before the backup window with no manual intervention required, as well as ensure that anything that went wrong with the process would be caught in a timely manner and resolved before the actual backup to tape. Remote syslog logging and log file scraping had been implemented in the environment for a while, and this gave us the option to log all errors to a remote syslog server. The string used to log errors was submitted to the monitoring department and was added to the list of strings that triggered an alert with the control center. The alert automatically generated a trouble ticket and dispatched it to an administrator. The whole process needed to be synchronized on both servers.

After some debate, we chose a solution utilizing SSH with public key authentication. The password-less OpenSSH private and public keys were generated on host1, and the public key was imported into the authorized_keys file for root on host2. Root logins through SSH were allowed on host2, and logins via SSH using the public key generated on host1 were only allowed from that server. Another aspect to the solution was that the same shell script, vxfsnap, would be used on both sides with a switch instructing it to execute in local or remote mode.

The vxfsnap Script

The vxfsnap script (see Listing 1) accepts the following arguments: original disk group, original volume name, name for the snapshot volume, name for the snapshot disk group, hostname/IP of the host processing the snapshot, and mount point for the snapshot volume. It has four modes of operation:

  • deport -- Unmounts the file system and deports the snapshot disk group.
  • join -- Imports the snapshot disk group and joins it into the target disk group executing a snapback.
  • snap -- Performs the snapshot and deports the disk group.
  • import -- Imports the snapshot disk group and mounts the file system.

Another optional switch can be used to freeze the Interwoven Teamsite application for the duration of snapback improving the consistency of the data used for the backup. This shell script was designed with re-usability in mind so that it can be implemented with little or no effort in a similar solution. It can be executed on one of the hosts, and it can control the process from a central location.

This command:

host1# vxfsnap -r -h host2 -G SNAPDG1 -V SNAP-Vol01 -m /data -e deport
would unmount the /data file system on host2 and deport the SNAPDG disk group. This can be followed by:

host1# vxfsnap -g DG1 -v Vol01 -G SNAPDG1 -V SNAP-Vol1 -e join -f
to import the SNAPDG1 disk group on host1 and perform everything including a snapback up to executing a snapshot, freezing the Interwoven Teamsite backing store, and unfreezing it after snapback is complete:

host1# vxfsnap -g DG1 -v Vol01 -G SNAPDG1 -V SNAP-Vol01 -e snap
Snap mode will take the snapshot. The separated volume will be split off into a new disk group that remains deported ready for other interested hosts to import. Finally, we can make use of the data on host2:

host1# vxfsnap -r -h host2 -G SNAPDG1 -V SNAP-Vol01 -m /data -e import
The vxfsnap script utility also can be used in other scripts that can be executed as cron jobs shortly before the backup window:




$FLASHSNAP -r -h $RHOST -G $SDG -V $SVOL -m $MNT -e deport && <\\>
$FLASHSNAP -g $DG -v $VOL -G $SDG -V $SVOL -e join -f && <\\>
$FLASHSNAP -g $DG -v $VOL -G $SDG -V $SVOL -e snap && <\\>
$FLASHSNAP -r -h $RHOST -G $SDG -V $SVOL -m $MNT -e import
With the Veritas FlashSnap solution in place, the file system containing the Interwoven Teamsite application was added to the exclude list for the backup client software. Rebooting the server used for backup processing can potentially break the configuration, because the cron job requires that the file system be mounted on host2. This can be solved with a startup script that checks whether the designated disk group is recognized as local at boot time and mounts the volume under a specified mount point, or by adding the file system in the /etc/vfstab configuration file and expecting a failure if the disk group or volume are unavailable.


This solution achieved a little more than it was designed to. Effectively, the copy of the data used for backup to tape is available all the time in case a file, directory, or the whole volume becomes corrupted and needs to be restored. Recovering from any of the mentioned disasters is a simple process that takes minutes, requires no special backup infrastructure resources, and adds further to the value of the solution.

Veritas Flashsnap is a technology that can help both users and administrators in their quest to better utilize the resources of a system. It can be used in simple scenarios with machines directly attached to the storage media or in more complex configurations in Storage Area Networks as a host-controlled solution. It can also be used with a number of applications for point-in-time backup copies at the volume level that can be used for anything from off-host backup processing to disaster recovery.

Borislav Stoichkov has an MS degree in Computer Science with a focus on cryptography as well as certifications from Sun Microsystems and Red Hat. He has been engineering and implementing solutions and managing Linux and Solaris systems in large enterprise environments for the past 5 years. Interests include secure communication and data storage, high performance computing. Currently, he works as a Unix consultant in the Washington DC area and can be reached at: