 Off-Host
Backup Processing with Veritas FlashSnap
Borislav Stoichkov
Backup times and the resources associated with them are becoming
more and more important in the evolving model of 24/7 application
development and content management. Developers all over the world
collaborate on the same projects and access the same resources that
must be 100% available during the business hours of their respective
time zones. This gives systems administrators very little room to
completely satisfy their customers -- the developers.
Source code and content repositories contain hundreds of projects
and millions of small files that require considerable amounts of
time and system resources to back up. Also, data protection is a
top priority that presents system and backup engineers with the
question of how to effectively ensure data protection and availability
in case of a disaster and, at the same time, minimize the duration
and resource overhead of the process.
The Problem
My organization was faced with these very issues. One of our high-profile
customers was using Interwoven's flagship product TeamSite installed
on a Sun Solaris 8 server. Interwoven Teamsite's key features are
content management and services, code and media versioning, collaboration
and parallel development, branching of projects, transactional content
deployment, etc. Developers all over the world were using the system's
resources throughout the day for mission-critical tasks. As the
number of projects and branches increased so did the number of files
and the amount of data that needed to be backed up and protected.
Suddenly, the application was managing millions of small files
and hundreds of branches and projects. Backup times were reaching
7-8 hours with the bottleneck caused by the sheer amount of files
and data. Complaints were received that during the backup window
the application as well as the system were becoming difficult to
use and that system performance was becoming unacceptable. The customer
requested a solution that would be as cheap as possible and would
not require a change in their development and content management
model.
From a storage perspective, the server had two internal mirrored
drives for the operating system file systems under Veritas Volume
Manager control. An external Fibre Channel array was attached to
the machine presenting a single LUN on which the Interwoven Teamsite
application was installed. The LUN had a 143-GB Veritas file system
and was under Veritas Volume Manager control as well.
The idea for the solution was to take a snapshot of the application
file system and use that snapshot for a backup to tape on another
host. Thus, the backup window could be extended as much as needed
without affecting the performance or usability of the server. File
system snapshots, however, do not allow off-host processing. Given
that Veritas Volume Manager was already installed and active on
the machine, using its built-in volume snapshot features seemed
natural. The only problems remaining were to break off the volume
with the snapshot in a manner that was supported by the vendor and
did not present any risks, and to minimize the time needed for the
reverse operation -- syncing the snapped off mirror without mirroring
a 143-GB file system from scratch, which is a long and tedious process.
Implementing FlashSnap
The resolutions to both problems are found in the Veritas FlashSnap
product. FlashSnap is a license key-enabled option of the Veritas
Foundation Suite solutions. The license enables the use of the FastResync
and Dynamic Split and Join features of Veritas Volume Manager. With
FastResync enabled on a volume, Volume Manager uses a map to keep
track of which blocks are updated in the volume and in the snapshot.
In time the data on the original volume changes, and the data on
the snapshot volume becomes outdated.
The presence of a FastResync map ensures that in an operation
where the snapshot is resynchronized with the primary volume only
the modified blocks (dirty blocks) are applied. Full mirror synchronization
is no longer necessary. The map is persistent across reboots because
it is stored in a data change object (DCO) log volume associated
with the original volume. Dynamic Split and Join allow for the volume
snapshots to be placed into a separate disk group, which can be
deported and imported on another host for off-host processing. The
only requirement is for the disks to be visible to the designated
host. At a later stage, the disk group can be re-imported on the
original host and joined with the original disk group or, if necessary,
with a different one.
For the implementation, additional storage was required on the
storage array equal to the original amount of 143 GB. The added
storage was configured into a new LUN. A new low-end server running
Sun Solaris 8 (host2) was attached to the array as well. The newly
added LUN (LUN1) was presented to both hosts, while the original
LUN (LUN0) was only made visible on the original host (host1).
DCO Logging
Persistent FastResync requires a DCO log to be associated with
the original volume. That option has been available only since Veritas
Volume Manager 3.2 and disk group version 90, so the volume management
software was upgraded to the latest version. The existing disk group
was upgraded to the latest version as well. The FlashSnap license
obtained from Veritas was installed on both hosts. For verification
that the newly added license is functional, the following command
can be issued:
# vxdctl license
All features are available:
........
FastResync
DGSJ
A small problem arose from the fact that there was no room for a DCO
log on LUN0 because all of its space was allocated for the application
volume. Luckily, the file system on it was VXFS, and it was possible
for the volume and the file system to be shrunk:
host1# vxresize -F vxfs -g DG1 Vol01 -20M
With that fixed, a DCO (data change object) log volume was associated
with the original volume:
host1# vxprint -g DG1
.............
dm DG101 c4t9d0s2 - 286676992 - - - -
v Vol01 fsgen ENABLED 286636032 - ACTIVE - -
pl Vol01-01 Vol01 ENABLED 286656000 - ACTIVE - -
sd DG101-01 Vol01-01 ENABLED 286656000 0 - - -
host1# vxassist -g DG1 addlog Vol01 logtype=dco dcologlen=1056 ndcolog=1 DG101
host1# vxprint -g DG1
..............
dm DG101 c4t9d0s2 - 286676992 - - - -
v Vol01 fsgen ENABLED 286636032 - ACTIVE - -
pl Vol01-01 Vol01 ENABLED 286656000 - ACTIVE - -
sd DG101-01 Vol01-01 ENABLED 286656000 0 - - -
dc Vol01_dco Vol01 - - - - - -
v Vol01_dcl gen ENABLED 1056 - ACTIVE - -
pl Vol01_dcl-01 Vol01_dcl ENABLED 1056 - ACTIVE - -
sd DG101-02 Vol01_dcl-01 ENABLED 1056 0 - - -
The length of the DCO log determines the level at which changes are
tracked. A longer DCO log will trigger more in-depth tracking and
will require less time for the snapshot to resynchronize. Increasing
the log too much may cause performance overhead on the system. The
default number of plexes in the mirrored DCO log volume is 2. It is
recommended that the number of DCO log plexes configured equal the
number of data plexes in the volume -- in our case, one.
The default size for a DCO plex is 133 blocks. A different number
can be specified, but it must be a number from 33 up to 2112 blocks
in multiples of 33. If the snapshot volumes are to be moved to a
different disk group, the administrator must ensure that the disks
containing the DCO plexes can accompany them.
Establishing a Snapshot Mirror
The next step is to enable persistent FastResync on the volume,
so that sequential re-mirroring operations take considerably less
than the establishment of a full mirror and are applied from the
DCO log:
host1# vxvol -g DG1 set fastresync=on Vol01
host1# vxprint -g DG1 -m Vol01 | grep fastresync
fastresync=on
The addition of LUN1 to the DG1 disk group as disk DG102 completes
our preparation phase, so now we are ready to establish our snapshot:
host1# vxassist -g DG1 snapstart Vol01 alloc=DG102
This operation will establish a mirror of volume Vol01 and will add
an additional DCO log object that will be in a DISABLED and DCOSNP
state for use by the snapshot. The snapstart process takes a considerable
amount of time, because it is a full mirror creation. The vxassist
command will block until the snapshot mirror is complete. It can be
placed in the background by using the -b argument to vxassist.
During the snapstart phase, disk group DG1 will look like this:
v Vol01 fsgen ENABLED 286636032 - ACTIVE ATT1 -
pl Vol01-01 Vol01 ENABLED 286656000 - ACTIVE - -
sd DG101-01 Vol01-01 ENABLED 286656000 0 - - -
pl Vol01-02 Vol01 ENABLED 286656000 - SNAPATT ATT -
sd DG102-01 Vol01-02 ENABLED 286656000 0 - - -
dc Vol01_dco Vol01 - - - - - -
v Vol01_dcl gen ENABLED 1056 - ACTIVE - -
pl Vol01_dcl-01 Vol01_dcl ENABLED 1056 - ACTIVE - -
sd DG101-02 Vol01_dcl-01 ENABLED 1056 0 - - -
pl Vol01_dcl-02 Vol01_dcl DISABLED 1056 - DCOSNP - -
sd DG102-02 Vol01_dcl-02 ENABLED 1056 0 - - -
Once the mirror is established, the plex on disk DG102 will be in
a SNAPDONE state ready to be separated from the original volume. If
the snapshot is attempted before the snapshot plex is in a SNAPDONE
state, the command will fail. If snapstart is placed in the background
with the -b switch, the vxassist snapwait command will
wait until the snapstart command is done and can be used in
scripts to ensure that no other commands are issued before the completion
of snapstart:
v Vol01 fsgen ENABLED 286636032 - ACTIVE - -
pl Vol01-01 Vol01 ENABLED 286656000 - ACTIVE - -
sd DG101-01 Vol01-01 ENABLED 286656000 0 - - -
pl Vol01-02 Vol01 ENABLED 286656000 - SNAPDONE - -
sd DG102-01 Vol01-02 ENABLED 286656000 0 - - -
dc Vol01_dco Vol01 - - - - - -
v Vol01_dcl gen ENABLED 1056 - ACTIVE - -
pl Vol01_dcl-01 Vol01_dcl ENABLED 1056 - ACTIVE - -
sd DG101-02 Vol01_dcl-01 ENABLED 1056 0 - - -
pl Vol01_dcl-02 Vol01_dcl DISABLED 1056 - DCOSNP - -
sd DG102-02 Vol01_dcl-02 ENABLED 1056 0 - - -
To execute the actual snapshot:
host1# vxassist -g DG1 snapshot Vol01 SNAP-Vol01
host1# vxprint -g DG1
v SNAP-Vol01 fsgen ENABLED 286636032 - ACTIVE - -
pl Vol01-02 SNAP-Vol01 ENABLED 286656000 - ACTIVE - -
sd Dg102-01 Vol01-02 ENABLED 286656000 0 - - -
dc SNAP-Vol01_dco SNAP-Vol01 - - - - - -
v SNAP-Vol01_dcl gen ENABLED 1056 - ACTIVE - -
pl Vol01_dcl-02 SNAP-Vol01_dcl ENABLED 1056 - ACTIVE - -
sd DG102-02 Vol01_dcl-02 ENABLED 1056 0 - - -
sp Vol01_snp SNAP-Vol01 - - - - - -
v Vol01 fsgen ENABLED 286636032 - ACTIVE - -
pl Vol01-01 Vol01 ENABLED 286656000 - ACTIVE - -
sd DG101-01 Vol01-01 ENABLED 286656000 0 - - -
dc Vol01_dco Vol01 - - - - - -
v Vol01_dcl gen ENABLED 1056 - ACTIVE - -
pl Vol01_dcl-01 Vol01_dcl ENABLED 1056 - ACTIVE - -
sd Dg102-02 Vol01_dcl-01 ENABLED 1056 0 - - -
sp SNAP-Vol01_snp Vol01 - - - - - -
Now the disk group can be split so that the disk containing the snapshot
volume is placed in a different group:
host1# vxdg split DG1 SNAPDG1 SNAP-Vol01
The new disk group SNAPDG1 containing SNAP-Vol01 and its DCO log volume
can be deported and imported on the alternate host:
host1# vxdg deport SNAPDG1
host2# vxdg import SNAPDG1
Following the split, the snapshot volume is disabled. The following
commands can be used to recover and start the volume:
host2# vxrecover -g SNAPDG1 -m SNAP-Vol01
host2# vxvol -g SNAPDG1 start SNAP-Vol01
A consistency check can be performed on the volume's file system,
and it can be mounted for backup processing or any other type of data
manipulation:
host2# fsck -F vxfs /dev/vx/rdsk/SNAPDG1/ SNAP-Vol01
host2# mount -F vxfs /dev/vx/dsk/SNAPDG1/ SNAP-Vol01 /data
Before the backup window kicks in, or in case the snapshot needs to
be refreshed, the file system can be unmounted and the volume deported
and imported again on the original host:
host2# umount /data
host2# vxvol -g SNAPDG1 stop SNAP-Vol01
host2# vxdg deport SNAPDG1
host1# vxdg import SNAPDG1
Now the disk(s) in disk group SNAPDG1 can be joined into disk group
DG1:
host1# vxdg join SNAPDG1 DG1
host1# vxrecover -g SNAPDG1 -m Vol01
Once the snapshot volume is back into its original disk group, we
can perform the snapback operation:
host1# vxassist -g DG1 snapback SNAP-Vol01
In some cases when there is data corruption on the original volume,
the data on the snapshot volume can be used for the synchronization.
This is achieved by using the resyncfromreplica argument to
the vxassist -o option with snapback. The operation will not
take long to execute at all. If performed within hours of the first
snapshot, the process may take less than a minute depending on the
amount of file system changes.
In our environment, a snapback process that is executed approximately
24 hours after the previous one takes no longer than 12 minutes.
Effectively, we have decreased the time it takes to back up the
application from hours to less than 15 minutes from the developers'
and system users' point of view.
Automating the Process
The last challenge in this project was to automate the process
so that it would occur transparently on a daily basis before the
backup window with no manual intervention required, as well as ensure
that anything that went wrong with the process would be caught in
a timely manner and resolved before the actual backup to tape. Remote
syslog logging and log file scraping had been implemented in the
environment for a while, and this gave us the option to log all
errors to a remote syslog server. The string used to log errors
was submitted to the monitoring department and was added to the
list of strings that triggered an alert with the control center.
The alert automatically generated a trouble ticket and dispatched
it to an administrator. The whole process needed to be synchronized
on both servers.
After some debate, we chose a solution utilizing SSH with public
key authentication. The password-less OpenSSH private and public
keys were generated on host1, and the public key was imported into
the authorized_keys file for root on host2. Root logins through
SSH were allowed on host2, and logins via SSH using the public key
generated on host1 were only allowed from that server. Another aspect
to the solution was that the same shell script, vxfsnap, would be
used on both sides with a switch instructing it to execute in local
or remote mode.
The vxfsnap Script
The vxfsnap script (see Listing 1) accepts the following arguments:
original disk group, original volume name, name for the snapshot
volume, name for the snapshot disk group, hostname/IP of the host
processing the snapshot, and mount point for the snapshot volume.
It has four modes of operation:
- deport -- Unmounts the file system and deports the snapshot
disk group.
- join -- Imports the snapshot disk group and joins it
into the target disk group executing a snapback.
- snap -- Performs the snapshot and deports the disk group.
- import -- Imports the snapshot disk group and mounts
the file system.
Another optional switch can be used to freeze the Interwoven Teamsite
application for the duration of snapback improving the consistency
of the data used for the backup. This shell script was designed
with re-usability in mind so that it can be implemented with little
or no effort in a similar solution. It can be executed on one of
the hosts, and it can control the process from a central location.
This command:
host1# vxfsnap -r -h host2 -G SNAPDG1 -V SNAP-Vol01 -m /data -e deport
would unmount the /data file system on host2 and deport the SNAPDG
disk group. This can be followed by:
host1# vxfsnap -g DG1 -v Vol01 -G SNAPDG1 -V SNAP-Vol1 -e join -f
to import the SNAPDG1 disk group on host1 and perform everything including
a snapback up to executing a snapshot, freezing the Interwoven Teamsite
backing store, and unfreezing it after snapback is complete:
host1# vxfsnap -g DG1 -v Vol01 -G SNAPDG1 -V SNAP-Vol01 -e snap
Snap mode will take the snapshot. The separated volume will be split
off into a new disk group that remains deported ready for other interested
hosts to import. Finally, we can make use of the data on host2:
host1# vxfsnap -r -h host2 -G SNAPDG1 -V SNAP-Vol01 -m /data -e import
The vxfsnap script utility also can be used in other scripts that
can be executed as cron jobs shortly before the backup window:
#!/bin/bash
DG=DG1
VOL=Vol01
SDG=SNAPDG1
SVOL=SNAP-Vol01
MNT="/backup"
RHOST="host2"
FLASHSNAP="/usr/local/vxfsnap/vxfsnap"
$FLASHSNAP -r -h $RHOST -G $SDG -V $SVOL -m $MNT -e deport && <\\>
$FLASHSNAP -g $DG -v $VOL -G $SDG -V $SVOL -e join -f && <\\>
$FLASHSNAP -g $DG -v $VOL -G $SDG -V $SVOL -e snap && <\\>
$FLASHSNAP -r -h $RHOST -G $SDG -V $SVOL -m $MNT -e import
With the Veritas FlashSnap solution in place, the file system containing
the Interwoven Teamsite application was added to the exclude list
for the backup client software. Rebooting the server used for backup
processing can potentially break the configuration, because the cron
job requires that the file system be mounted on host2. This can be
solved with a startup script that checks whether the designated disk
group is recognized as local at boot time and mounts the volume under
a specified mount point, or by adding the file system in the /etc/vfstab
configuration file and expecting a failure if the disk group or volume
are unavailable.
Conclusions
This solution achieved a little more than it was designed to.
Effectively, the copy of the data used for backup to tape is available
all the time in case a file, directory, or the whole volume becomes
corrupted and needs to be restored. Recovering from any of the mentioned
disasters is a simple process that takes minutes, requires no special
backup infrastructure resources, and adds further to the value of
the solution.
Veritas Flashsnap is a technology that can help both users and
administrators in their quest to better utilize the resources of
a system. It can be used in simple scenarios with machines directly
attached to the storage media or in more complex configurations
in Storage Area Networks as a host-controlled solution. It can also
be used with a number of applications for point-in-time backup copies
at the volume level that can be used for anything from off-host
backup processing to disaster recovery.
Borislav Stoichkov has an MS degree in Computer Science with
a focus on cryptography as well as certifications from Sun Microsystems
and Red Hat. He has been engineering and implementing solutions
and managing Linux and Solaris systems in large enterprise environments
for the past 5 years. Interests include secure communication and
data storage, high performance computing. Currently, he works as
a Unix consultant in the Washington DC area and can be reached at:
borislav.stoichkov@meanstream.org. | 
