[virt-tools-list] pre/post backup script for KVM guests

Wed Oct 21 08:56:58 UTC 2009

Hello Daniel.

I have one question - does BackupPC backup sparse files correctly ? What i mean is , using rsync/rdiff etc. results in sparse files being backed up as non-sparse and taking up a lot of space. I was searching for a solution to this until i found storeBackup :

http://www.nongnu.org/storebackup/

It does backup the image files as sparse, and has the option of splitting the backup into several smaller files ( with configurable size ), then checking their md5sums and backing up only the changed parts + hard-linking the ones that stayed the same from the previous backups. This way i get the following :

Directory where the image files are  :

root at blackwing:/srv/storage/virt# du -h
39G	.

Directory where the backups are made ( has 4 backups at the moment ):

root at blackwing:/srv/backup/virtual# du -h
3.3G	./2009.10.15_23.46.45/testwing-sda.qcow2
543M	./2009.10.15_23.46.45/newframe-sda.img
4.0K	./2009.10.15_23.46.45/.storeBackupLinks
590M	./2009.10.15_23.46.45/ubuntu-9.04-server-amd64.iso
2.3G	./2009.10.15_23.46.45/bluewing-sda.qcow2
651M	./2009.10.15_23.46.45/ubuntu-9.10-beta-server-amd64.iso
455M	./2009.10.15_23.46.45/purplewing-sda.img
579M	./2009.10.15_23.46.45/ubuntu-8.04.3-server-amd64.iso
755M	./2009.10.15_23.46.45/whitewing-sda.qcow2
3.5G	./2009.10.15_23.46.45/yellowwing-sda.qcow2
1.2G	./2009.10.15_23.46.45/redwing-sda.img
14G	./2009.10.15_23.46.45
5.3M	./2009.10.15_21.17.45/testwing-sda.qcow2
7.9M	./2009.10.15_21.17.45/newframe-sda.img
4.0K	./2009.10.15_21.17.45/.storeBackupLinks
20K	./2009.10.15_21.17.45/ubuntu-9.04-server-amd64.iso
44M	./2009.10.15_21.17.45/bluewing-sda.qcow2
24K	./2009.10.15_21.17.45/ubuntu-9.10-beta-server-amd64.iso
9.4M	./2009.10.15_21.17.45/purplewing-sda.img
20K	./2009.10.15_21.17.45/ubuntu-8.04.3-server-amd64.iso
3.7M	./2009.10.15_21.17.45/whitewing-sda.qcow2
58M	./2009.10.15_21.17.45/yellowwing-sda.qcow2
2.7M	./2009.10.15_21.17.45/redwing-sda.img
131M	./2009.10.15_21.17.45
474M	./2009.10.20_07.16.51/testwing-sda.qcow2
86M	./2009.10.20_07.16.51/newframe-sda.img
4.0K	./2009.10.20_07.16.51/.storeBackupLinks
28K	./2009.10.20_07.16.51/ubuntu-9.04-server-amd64.iso
230M	./2009.10.20_07.16.51/bluewing-sda.qcow2
32K	./2009.10.20_07.16.51/ubuntu-9.10-beta-server-amd64.iso
20M	./2009.10.20_07.16.51/purplewing-sda.img
28K	./2009.10.20_07.16.51/ubuntu-8.04.3-server-amd64.iso
42M	./2009.10.20_07.16.51/whitewing-sda.qcow2
321M	./2009.10.20_07.16.51/yellowwing-sda.qcow2
241M	./2009.10.20_07.16.51/redwing-sda.img
1.4G	./2009.10.20_07.16.51
13M	./2009.10.17_18.21.07/testwing-sda.qcow2
41M	./2009.10.17_18.21.07/newframe-sda.img
4.0K	./2009.10.17_18.21.07/.storeBackupLinks
28K	./2009.10.17_18.21.07/ubuntu-9.04-server-amd64.iso
112M	./2009.10.17_18.21.07/bluewing-sda.qcow2
32K	./2009.10.17_18.21.07/ubuntu-9.10-beta-server-amd64.iso
13M	./2009.10.17_18.21.07/purplewing-sda.img
28K	./2009.10.17_18.21.07/ubuntu-8.04.3-server-amd64.iso
28M	./2009.10.17_18.21.07/whitewing-sda.qcow2
132M	./2009.10.17_18.21.07/yellowwing-sda.qcow2
34M	./2009.10.17_18.21.07/redwing-sda.img
372M	./2009.10.17_18.21.07
16G	.

I am working on a python script to automate the whole process ( something among the lines of what your script does ) i.e. :

1. Taking a lvm snapshot
2. mounting the snapshot
3. backing up the kvm images ( selectively  - storebackup supports conditions - i.e. i wouldn't want to backup the iso files as i did above )
4. unmounting and destroying the lvm snapshot.

I could scrap the above for your script, but the disk usage of backups is critical for me.

----- Original Message -----
From: "Daniel Berteaud" <daniel at firewall-services.com>
To: virt-tools-list at redhat.com, backuppc-users at lists.sourceforge.net
Cc: tech at firewall-services.com
Sent: Monday, October 19, 2009 4:02:58 PM
Subject: [virt-tools-list] pre/post backup script for KVM guests

Hi everyone.

I've written a script to backup virtual machines managed by libvirt
(only tested with KVM guests, but should works for Xen too, maybe others
as well).

It's called virt-backup.pl

I've written it for integration with BackupPC:
- take a dump of a VM in using the pre-backup facility
- backup the dumps using BackupPC
- cleanup the dumps in the post-backup phase

This script can be used outside of BackupPC as it's quite generic.

There are two main mode for this script:
--pre: take the backup of the VM. This is called --pre because I use it
as a pre-backup script for BackupPC
--post: cleanup the dumps

Here're some functionalities:

- no configuration file needed, everything can be passed as command line
arg
- can take snapshots of virtual disks if they are on LVM Logical volume
(the default is to try LVM in any case, if it's not possible, then, just
dump the block device/file)
- Supports backup of running VM with minimal downtime (if each virtual
disks can be snapshoted, just resume the VM immediately resulting in
just a few seconds of downtime, then, dump the snapshots. If snapshots
are not available, the guest is suspended during the dump)
- can save the state of running vm (equivalent of virsh save/virsh
restore). This is optional because it's still not very reliable, and
sometimes the restoration fails, leaving a crashed qemu process running
and eating CPU cycles.
- Can compress on-the-fly the virtual disks dumps
(gzip,bzip2,pbzip2,lzop,xz)
- Support virtual disks exclusions (if you want to backup the system
disk of a VM, but not the data one for example)
- Can work on installations where virtual disks are stored on one hosts,
and guests runs on another on (NFS, iscsi etc... in any case, the script
must be run on the host which holds the virtual disks)
- Can backup as many guests as you want in one run (they'll be dumped
sequentially)
- Backups are run with low priority (nice and ionice), so it should'nt
slow down too much your system.

Here are the dependencies for this script to work
- Sys::Virt perl module
- XML::Simple per module
- Getopt::Long perl module
- lvm2
- gzip (optional)
- bzip2 (optional)
- pbzip2 (optional)
- lzop (optional)
- xz (optional)

You can run this script without argument to see the help. Edit it if you
want more informations (there are some examples on how to use it at the
beginning of the script).

The script can be found here:

http://repo.firewall-services.com/misc/virt-backup.pl

Regards, Daniel

-- 
Daniel Berteaud
FIREWALL-SERVICES SARL.
Société de Services en Logiciels Libres
Technopôle Montesquieu
33650 MARTILLAC
Tel : 05 56 64 15 32
Fax : 05 56 64 15 32
Mail: daniel at firewall-services.com
Web : http://www.firewall-services.com

_______________________________________________
virt-tools-list mailing list
virt-tools-list at redhat.com
https://www.redhat.com/mailman/listinfo/virt-tools-list