Quick Links
Have you ever had the need to bulk restore your Citrix-Xen VMs for a disaster recovery (DR) solution, or just to test that your backups are working? HTG explains how to use Xen-Phoenix, a free bash script, for VM restoring.
Photo Credit: Ryan McCurdy via Compfight cc
Like we've mentioned in "How To Backup Citrix Xen VMs for Free with Xen-pocalypse", one of the nice things in Citrix Xen is that many of its features are free of charge. With that said, if you want the "Automated VM protection and recovery" feature, you would have to start paying for the "Advance" license. Again, it is not that we at HTG are dismissing the value of a true backup solution, but if you're on a tight budget, have already gotten the VM images out of the hypervisor and are in need of a way to automate your "test restore"/"DR refresh" procedures, you might find Xen-Phoenix to be a perfectly reasonable solution before you make the budget commitment.
Overview
The "use case": You have a couple of VMs that require a restore. The importing in "Xen Center" using right clicks works ok, but you want this process to happen automatically and on a schedule. This Bash script uses the "XE" command to perform its duties. XE is the Xen command line interface (CLI), automatic equivalent for issuing the "right clicks" in the "Xen Center". We will be calling the script from Cron which will supply the "scheduling" part. In its simplest form, the restore flow is:
- Delete all* previously existing VMs on the server
- Import VMs from files in the backup location.
- Verify that all VMs are operable by activating them one by one and looking for the guest tools heartbeat.
- Turn VMs off when a heartbeat has been discovered, or a timeout has been reached.
*The delete all VMs behavior can be disabled completely and supports exceptions (see below).
Lets get cracking :)
Obtain the script
Xen-phoenix can be freely obtained from github, using the regular git methods. With that said, if you're not versed in git yet, you can grab the zip file with this link. As the script needs to run on one of your Xen servers, you should extract it there so that execution permissions are preserved.
wget https://github.com/aviadra/Xen-phoenix/archive/master.zip
unzip master
While the above would work, you are advised to use the GIT method so that you may benefit from future updates.
The exports location
We need to configure where to take the VM exports from.
While randomly poking around in Citrix Xen, I have found that the Storage Repositories (SRs) are available for use under "/var/run/sr-mount/%UUID%" where UUID is the unique identifier of the SR, which can be obtained from the GUI.
This means we can use the regular "Next -> Next -> Finish" wizard to create the "mount" to the desired backup location, and then have the script use that path (as apposed to messing with mounting from the command line which is beyond the scope of this guide).
To Create a new "mount", right click the server name and select New SR.
In this example, we will point Xen to a Windows share, so choose "Windows File Sharing (CIFS)":
Complete the Next -> Next -> Finish.
Obtain the SR's UUID
To obtain an SR's UUID, simply click on its name in the Xen Center and go to the "General" tab.
In order to copy the UUID, simply right click it and choose "copy".
With this information at hand, you are ready to edit the settings file.
Configure the Settings file
The Xen-phoenix project comes bundled with a "settings" file template. This template should be edited to reflect your setup and passed as the first argument to the script.
The settings file designates the following:
- The location of the source exports -- If you have followed the guide to this point, you only have to replace the %UUID% with the SR's as it was obtained from above.
- The location of SendEmail -- If you have opted to enable Email, you need to input where you have extracted the perl executable here.
- Verifier -- This controls the post restore verification procedure. This is enabled by default, as a successful "file level restore" doesn't necessary mean a functional VM.
- Server_prep -- This controls the pre-restore deletion of all VMs on the DR server. This is enabled by default because it is assumed that the DR server is a dedicated server for this purpose. If you need a VM that's running on this server to not be deleted, configure it to be excluded. If this behavior is not suited for your situation, simply disable it completely.
- Email details -- Again, if you have enabled email, you need to define details such as: To, From, Servername/IP & etc'.
- Debugging -- The default is to have debugging turned off with the value "0″ (zero). You should not need to turn this on, but if you do, more information is noted in the troubleshooting segment.
Execution
This script takes the settings file as its first argument and any other arguments as "Chevrons" to search for (separated by spaces). Chevrons, are "strings", which support regular expressions which represent at least part of the desired file name of the VM export.
That is, in its simplest form, an invocation of Xen-phoenix would look like:
./Xen-phoenix.sh settings.cfg DevTools
Where in the case above, we are inside the directory that holds the script & the settings file and the "Chevron" the script will look for is "DevTools". This will cause all the files in the exports directory with the string "DevTools" in their name to be imported.
A more complex example can look like:
./Xen-phoenix.sh settings.cfg [Aa][Dd] [Bb][iI] [Dd]ev[Tt]ools
Where in the example above, files that contain the word "devtools" spelled with a capital "D" and "T" or non-capital ones, as well as "AD" & "BI" with or without capitalization, will be imported.
Scheduling
As noted above, we will use Cron to schedule the execution. Before we go into the configuration, it is highly recommended that you configure the already installed SSMTP package on your Xen server. While this is an optional step, doing so will give you a "backwash" collector. Having such a "backwash collector" may alert you to things the script isn't able to.
Enter into cron's editing mode by issuing:
crontab -e
If you have followed the instructions above and you want to add a scheduled backup for Sunday at 01:31 (1:31AM), input the below:
31 01 * * sun /root/Xen-phoenix-master/Xen-phoenix.sh /root/Xen-phoenix-master/settings.cfg [Aa][Dd] DB [Bb][iI] [Dd]ev[Tt]ools
The above is correct assuming your script and settings file are both under "/root/Xen-phoenix-master/".
Obtain SendEmail (optional)
We've written about the SendEmail perl program in the past, so there is no need to reiterate here. Suffice it to say, it works the same way on Linux as it does on Windows.
While enabling email is optional, it is highly recommended because then the script will be able to:
- Inform you when it started & finished running.
- Alert you of any errors it was able to detect and handle.
Download it to the Xen server and extract.
wget http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz
tar xvzhf sendEmail-v1.56.tar.gz
Note the location where you extracted it to. You will need it for the settings file.
Defining the "Delete Exception" Tag (optional)
Xen-Phoenix takes a page from its ancestor (Xen-Pocalypse) and gives you the ability to granularly exclude a VM from the system-wide deletion using a control TAG. In order to do this, you need to define a new "Custom Field".
To do this, open the server's or even a VM's properties. In the navigation pane, select "Custom Fields".
If this is the first time you define a "custom field" (as in the example above), you will not have the "Phoenix_keeper" field to input data into, so you need to create it. To do this, click on "Edit Custom Fields" in the dialog box that pops up, then click on "Add..."
Create a "Text" type field with the name "Phoenix_keeper".
Note: The name of the custom field has been "hard coded" into the script, so you MUST not deviate from the spelling above, unless you change the relevant code as well.
Once the field has been created, you should see:
Close the window. You should now have the "Phoenix_keeper" field to fill, as in the picture below.
Now, all you have to do is populate this field with a remark (any remark will do), which will cause Xen-phoenix to skip it when deleting.
Troubleshooting
While I've put a lot of effort into making the script as easy to use and as foolproof as possible, "the world is a bigger lab". The information below may help you ascertain what is the source of your troubles.
Logging
All "logging" is collected by the Xen host running the script in the syslog mechanism. This, of course, can be viewed with:
less +F /var/log/messages
You are looking for the key word "Xen-Phoenix".
Note: Citrix has set a two (2) day retention policy for syslog of its servers. You may want to keep that in mind for postmortems.
Debugging
As noted in the settings file segment, there is a directive to enable debugging. Enabling debugging will cause the script to output verbose logging to the console and castrate it from sending emails and actually performing the imports, unless the relevant flags are also set. The possible flags are noted in the settings file template, and they enable you to granularly define what you want to debug.
It is my hope that you haven't needed any debugging and you are reaping the fruits of my labor :)