Difference between revisions of "Full system backup with rsync"

From ArchWiki
Jump to: navigation, search
m (With a list: lowered the character size for the title)
(Add error handling to script. Force correct usage (give destination parameter) before execution. Currently the script will run anyway and start dumping in the root of your installation.)
(48 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
[[Category:System recovery]]
 
[[Category:System recovery]]
 
[[cs:Full System Backup with rsync]]
 
[[cs:Full System Backup with rsync]]
This [[rsync]] script allows creating a full backup copy across filesystems. It is setup so that the copy includes intact booting capabilities, optionally excluding selected files.
+
{{Article summary start}}
 +
{{Article summary text|Instructions on backing up the root tree, creating a bootable copy of your system, or for transferring your system to a new drive or partition.}}
 +
{{Article summary heading|Related}}
 +
{{Article summary wiki|Backup Programs}}
 +
{{Article summary wiki|rsync}}
 +
{{Article summary end}}
  
The approach has benefits over omitting system files by just copying personal data; if the system becomes corrupted in the main partition, overcoming the problem means booting into the backup as opposed to identifying and reinstalling affected programs.
+
This article is about using [[rsync]] to transfer a copy of your "/" tree, excluding a few select folders. This approach is considered to be better than [[disk cloning]] with {{ic|dd}} since it allows for a different size, partition table and filesystem to be used, and better than copying with {{ic|cp -a}} as well, because it allows greater control over file permissions, attributes, Access Control Lists (ACLs) and extended attributes. [http://www.bestbits.at/acl/about.html]
  
Instructions were converted from [http://bbs.archlinux.org/viewtopic.php?id=83071 this forum post].
+
Either method will work even while the system is running. Since it's going to take a while, you may freely browse the web during this time. Worst case scenario you won't get the same opened tabs when you restore the backup (or boot from it) because they weren't saved. Not a big deal.
  
==With a single command==
+
== With a single command ==
 
+
This operation can be done while the system is running. Since it's going to take a while, you may freely browse the web during this time. Worst case scenario you won't get the same opened tabs when you restore the backup (or boot from it) because they weren't saved. Not a big deal.
+
  
 
As root, run:
 
As root, run:
  
{{Warning|If you plan on backing up your system somewhere other than /mnt or /media, don't forget to add it to the list, to avoid a copying loop.}}
+
{{Note|If you plan on backing up your system somewhere other than {{ic|/mnt}} or {{ic|/media}}, don't forget to add it to the list, to avoid an infinite loop.}}
  
  # rsync -avx /* /media/Backup/backup --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,~/.gvfs}
+
  # rsync -aAXv /* /path/to/backup/folder --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs}
  
{{Note|The contents of {{ic|/dev}}, {{ic|/proc}}, {{ic|/sys}}, {{ic|/tmp}}, {{ic|/run}} were excluded because they are populated at boot (while the folders themselves are ''not'' created), {{ic|lost+found}} is filesystem-specific, and {{ic|~/.gvfs}} should be added too, so that it won't complain at the end that "some files/attrs were not transferred". The {{ic|-avx}} parameters tell rsync to keep permissions, be verbose and stay on the same partition.}}
+
For information on why these folders were excluded, read the next section.
  
==With a list==
+
== Using a script ==
For this method, two files are needed: the backup script and a file stating which files to include/exclude from the backup source.
+
  
===Backup script===
+
Same as in the above method, the system files are transferred in archive mode, ensuring that symbolic links, devices, permissions and ownerships, among other file attributes are preserved, while excluding files that match the patterns from the {{ic|--exclude}} string. On top of that, it shows at the end how much time it took, and it also writes a blank file stating when the backup was created. To learn more about what this script does, read {{ic|man rsync}} and {{ic|man date}}.
The script is very simple; it rsyncs in archive mode, ensuring that symbolic links, devices, permissions and ownerships, among other file attributes are preserved, while excluding files that match the patterns from the include/exclude list.
+
  
Save it as {{ic|rbackup.sh}} and make it executable:
+
{{Note|Again, if you plan on backing up your system somewhere other than {{ic|/mnt}} or {{ic|/media}}, don't forget to add it to the list, to avoid an infinite loop.}}
{{hc|rbackup.sh|2=
+
 
 +
{{hc|$ cd ~/Scripts
 +
$ nano backup.sh|<nowiki>
 
#!/bin/sh
 
#!/bin/sh
# rsync backup script
 
  
sudo sh -c "
+
if [ $# -lt 1 ]; then
     rsync -av --delete-excluded --exclude-from=backup.lst / $1;
+
     echo "No destination defined. Usage: $0 destination" >&2
     date > $1/BACKUP
+
     exit 1
"
+
elif [ $# -gt 1 ]; then
}}
+
    echo "Too many arguments. Usage: $0 destination" >&2
 +
    exit 1
 +
fi
  
;Backup source; {{ic|/}}
+
START=$(date +%s)
:In this case it's performing a backup on the whole root.
+
rsync -aAXv /* $1 --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,/var/lib/pacman/sync/*}
 +
FINISH=$(date +%s)
 +
echo "total time: $(( ($FINISH-$START) / 60 )) minutes, $(( ($FINISH-$START) % 60 )) seconds"
  
;Backup destination; {{ic|$}}1
+
touch $1/"Backup from $(date '+%A, %d %B %Y, %T')"</nowiki>}}
:Passed as an argument to the script; e.g. {{ic|/media/backup}}
+
  
;Include/exclude list; {{ic|<nowiki>--exclude-from=backup.lst</nowiki>}}
+
$ chmod +x backup.sh
:This example uses {{ic|backup.lst}}.
+
  
===Include/exclude list===
+
{{Note|The contents of {{ic|/dev}}, {{ic|/proc}}, {{ic|/sys}}, {{ic|/tmp}}, {{ic|/run}} were excluded because they are populated at boot (while the folders themselves are ''not'' created), {{ic|/lost+found}} is filesystem-specific, and {{ic|/home/*/.gvfs}} should be added too, so that it won't complain at the end that "some files/attrs were not transferred". For Arch Linux, {{ic|/var/lib/pacman/sync/*}} can also be excluded. This can save a lot of time on every backup since the directory contains many small files that tend to change quite often. These are description files for every package from the repositories and can be re-generated with {{ic|pacman -Syu}}. Additionally, you may also want to skip {{ic|/home/*/.thumbnails/*}}, {{ic|/home/*/.mozilla/firefox/*.default/Cache/*}} and {{ic|/home/*/.cache/chromium/*}}.}}
As deciding which files should populate this list can be difficult, here's a typical backup example that excludes common files that do not need to be backed up, such as the vast majority of {{ic|/dev}}. Note that specifying every desired file or directory in {{ic|Include}} is not needed; this section only acts as a filter for statements in {{ic|Exclude}}. This file is in the traditional include/exclude rsync format.
+
  
Save the following as {{ic|backup.lst}}:
+
Backing up is easy.
{{hc|backup.lst|
+
# Include
+
+ /dev/console
+
+ /dev/initctl
+
+ /dev/null
+
+ /dev/zero
+
  
# Exclude
+
While the system is running, open up a terminal and run (as root):
- /dev/*
+
- /run/*
+
- /proc/*
+
- /sys/*
+
- /tmp/*
+
- lost+found/
+
- /media/*
+
- /mnt/*
+
}}
+
  
;Exclude: Content in system directories; {{ic|/dev}}, {{ic|/proc}}, {{ic|/sys}} and {{ic|/tmp}} are excluded because they are created by the system at runtime, while the directories themselves need to be preserved since they are ''not'' regenerated at boot. Lastly, all {{ic|lost+found}} instances are skipped since they are partition-specific. For Archlinux {{ic|/var/lib/pacman/sync/*}} can also be excluded. This can save a lot of time on every backup since the directory contains many small files that tend to change quite often. These are description files for every package from the repositories. These files can be regenerated with {{ic|pacman -Syu}}.
+
# ~/Scripts/backup.sh /some/destination
  
{{Warning|don't forget to also exclude the mounted directory where you'll put the backup to avoid an infinite loop (in this example {{ic|'''/media/backup/'''}}).}}
+
You can also replace both {{ic|$1}} instances from the script with the actual destination path, move it to one of the folders from {{ic|echo $PATH}}, and then simply run (as root):
  
;Include: Even though {{ic|/dev}} is excluded, 4 files that are not dynamically created by [[udev]] need to be preserved. These are {{ic|console}}, {{ic|initctl}}, {{ic|null}} and {{ic|zero}}.
+
# backup.sh
  
==Backing up==
+
== Boot requirements ==
Substitute {{ic|/media/'''backup'''}} as appropriate, and mount the destination device:
+
# mount /dev/sdb1 /media/backup
+
{{tip|if the ability to boot the backup isn't important, omit the previous step and simply backup to an arbitrary directory.}}
+
  
Run the backup script (note that the trailing "{{ic|/}}" character is necessary):
+
Having a bootable backup can be useful in case the filesystem becomes corrupt or if an update breaks the system. The backup can also be used as a test bed for updates, with the [testing] repo enabled, etc. If you transferred the system to a different partition or drive and you want to boot it, the process is as simple as updating the backup's {{ic|/etc/fstab}} and your bootloader's configuration file.
# ./rbackup.sh /media/backup/
+
  
==Boot setup==
+
=== Update the fstab ===
After the sync is finished, the backup destination's {{ic|/etc/fstab}} has to be modified, a boot loader needs to be installed on the backup destination, and configuration in the destination's {{ic|/boot/grub/menu.lst}} requires to reflect the new location.
+
  
===Modify fstab===
+
Without rebooting, edit the backup's [[fstab]] to reflect the changes:
Edit the backup destination's fstab:
+
{{hc|# nano /path/to/backup/etc/fstab|2=
{{hc|$ nano /media/backup/etc/fstab|
+
tmpfs        /tmp          tmpfs    nodev,nosuid            0   0
none        /dev/pts      devpts    defaults      0 0
+
none        /dev/shm      tmpfs    defaults      0 0
+
  
<i>/dev/sda1    /boot        ext4     defaults     0 1
+
<font color=#888888><i>/dev/sda1    /boot        ext2     defaults                 0   2
/dev/sda5    /var         ext4     defaults     0 1
+
/dev/sda5    none         swap     defaults                 0  0
/dev/sda6    /usr          ext4      defaults     0 1
+
/dev/sda6    /             ext4      defaults                 0   1
/dev/sda7   /            ext4      defaults      0 1
+
/dev/sda7    /home        ext4      defaults                 0   2</i></font>}}
/dev/sda8   /home        ext4      defaults     0 1
+
/dev/sda9    swap          swap      defaults      0 0</i>
+
}}
+
  
Because rsync has performed a recursive copy of the ''entire'' root filesystem, all of the {{ic|sda}} mounpoints are problematic and will cause the backup boot to fail. In this example, all of the offending entries are replaced with a single one:
+
Because rsync has performed a recursive copy of the ''entire'' root filesystem, all of the {{ic|sda}} mountpoints are problematic and booting the backup will fail. In this example, all of the offending entries are replaced with a single one:
{{hc|$ nano /media/backup/etc/fstab|
+
none        /dev/pts      devpts    defaults      0 0
+
none        /dev/shm      tmpfs    defaults      0 0
+
  
/dev/'''sdb1'''    /            '''ext4'''      defaults      0 1
+
{{hc|# nano /path/to/backup/etc/fstab|
}}
+
tmpfs        /tmp          tmpfs    nodev,nosuid             0   0
  
As before, remember to use the proper device name and filesystem type.
+
/dev/'''sdb1'''    /            ext4      defaults                0  1}}
  
===Install bootloader===
+
Remember to use the proper device name and filesystem type.
While these instructions assume [[GRUB]] is being employed, they could easily be adapted to other bootloaders, such as [[LILO]].
+
  
Open the GRUB console:
+
=== Update the bootloader's configuration file ===
# grub
+
  
Direct the install towards the destination device:
+
This section assumes that you backed up the system to another drive or partition, that your current bootloader is working fine, and that you want to boot from the backup as well.
root (hd'''1,0''')
+
setup (hd'''1''')
+
  
;root; {{ic|hd 1,0}}
+
For [[Syslinux]], all you need to do is duplicate the current entry, except pointing to a different drive or partition:
:This should point to where the GRUB files are located--in this case, "{{ic|hd 1}}" means the second storage device ({{ic|/dev/sdb}}) and "{{ic|0}}" is the first partition ({{ic|/dev/sdb''1''}}).
+
  
;setup; {{ic|hd 1}}
+
{{Tip|Instead of editing {{ic|syslinux.cfg}}, you can also temporarily edit the menu during boot. When the menu shows up, press the {{Keypress|Tab}} key and change the relevant entries. Partitions are counted from one, drives are counted from zero.}}
:The command specifies where the actual boot loader is to be installed. In this example it is installed to the [[MBR]] of the second storage device.
+
 
+
===Configure bootloader===
+
The problem here is that even though the boot loader installs correctly, its menu entries are for the main system's partitions, not the backup system's.
+
 
+
It's possible to fix this by creating a custom {{ic|/boot/grub/menu.lst}} for the backup destination. In order to do this, modify {{ic|rbackup.sh}} so that it copies a custom {{ic|menu.lst}}:
+
{{hc|rbackup.sh|2=
+
#!/bin/sh
+
# rsync backup script
+
  
sudo sh -c "
+
# nano /boot/syslinux/syslinux.cfg
    rsync -av --delete-excluded --exclude-from=backup.lst / $1;
+
    '''cp ~/custom.menu.lst $1/boot/grub/menu.lst;'''
+
    date > $1/BACKUP
+
"
+
}}
+
  
{{tip|instead of replacing {{ic|menu.lst}} with a custom version solely for the backup, add a new GRUB entry pointing to the backup device or simply edit GRUB's menu during boot time.}}
+
For [[GRUB]], it's recommended that you automatically re-generate the {{ic|grub.cfg}} file:
  
== External links ==
+
# pacman -S os-prober
A sample implementation can be found here:
+
# grub-mkconfig -o /boot/grub/grub.cfg
* https://bitbucket.org/wooptoo/bin/raw/tip/bak
+
* https://bitbucket.org/wooptoo/bin/raw/tip/bak_list
+

Revision as of 12:14, 21 November 2012

Template:Article summary start Template:Article summary text Template:Article summary heading Template:Article summary wiki Template:Article summary wiki Template:Article summary end

This article is about using rsync to transfer a copy of your "/" tree, excluding a few select folders. This approach is considered to be better than disk cloning with dd since it allows for a different size, partition table and filesystem to be used, and better than copying with cp -a as well, because it allows greater control over file permissions, attributes, Access Control Lists (ACLs) and extended attributes. [1]

Either method will work even while the system is running. Since it's going to take a while, you may freely browse the web during this time. Worst case scenario you won't get the same opened tabs when you restore the backup (or boot from it) because they weren't saved. Not a big deal.

With a single command

As root, run:

Note: If you plan on backing up your system somewhere other than /mnt or /media, don't forget to add it to the list, to avoid an infinite loop.
# rsync -aAXv /* /path/to/backup/folder --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs}

For information on why these folders were excluded, read the next section.

Using a script

Same as in the above method, the system files are transferred in archive mode, ensuring that symbolic links, devices, permissions and ownerships, among other file attributes are preserved, while excluding files that match the patterns from the --exclude string. On top of that, it shows at the end how much time it took, and it also writes a blank file stating when the backup was created. To learn more about what this script does, read man rsync and man date.

Note: Again, if you plan on backing up your system somewhere other than /mnt or /media, don't forget to add it to the list, to avoid an infinite loop.
$ cd ~/Scripts
$ nano backup.sh
#!/bin/sh

if [ $# -lt 1 ]; then 
    echo "No destination defined. Usage: $0 destination" >&2
    exit 1
elif [ $# -gt 1 ]; then
    echo "Too many arguments. Usage: $0 destination" >&2
    exit 1
fi

START=$(date +%s)
rsync -aAXv /* $1 --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,/var/lib/pacman/sync/*}
FINISH=$(date +%s)
echo "total time: $(( ($FINISH-$START) / 60 )) minutes, $(( ($FINISH-$START) % 60 )) seconds"

touch $1/"Backup from $(date '+%A, %d %B %Y, %T')"
$ chmod +x backup.sh
Note: The contents of /dev, /proc, /sys, /tmp, /run were excluded because they are populated at boot (while the folders themselves are not created), /lost+found is filesystem-specific, and /home/*/.gvfs should be added too, so that it won't complain at the end that "some files/attrs were not transferred". For Arch Linux, /var/lib/pacman/sync/* can also be excluded. This can save a lot of time on every backup since the directory contains many small files that tend to change quite often. These are description files for every package from the repositories and can be re-generated with pacman -Syu. Additionally, you may also want to skip /home/*/.thumbnails/*, /home/*/.mozilla/firefox/*.default/Cache/* and /home/*/.cache/chromium/*.

Backing up is easy.

While the system is running, open up a terminal and run (as root):

# ~/Scripts/backup.sh /some/destination

You can also replace both $1 instances from the script with the actual destination path, move it to one of the folders from echo $PATH, and then simply run (as root):

# backup.sh

Boot requirements

Having a bootable backup can be useful in case the filesystem becomes corrupt or if an update breaks the system. The backup can also be used as a test bed for updates, with the [testing] repo enabled, etc. If you transferred the system to a different partition or drive and you want to boot it, the process is as simple as updating the backup's /etc/fstab and your bootloader's configuration file.

Update the fstab

Without rebooting, edit the backup's fstab to reflect the changes:

# nano /path/to/backup/etc/fstab
tmpfs        /tmp          tmpfs     nodev,nosuid             0   0

/dev/sda1    /boot         ext2      defaults                 0   2
/dev/sda5    none          swap      defaults                 0   0
/dev/sda6    /             ext4      defaults                 0   1
/dev/sda7    /home         ext4      defaults                 0   2

Because rsync has performed a recursive copy of the entire root filesystem, all of the sda mountpoints are problematic and booting the backup will fail. In this example, all of the offending entries are replaced with a single one:

# nano /path/to/backup/etc/fstab
tmpfs        /tmp          tmpfs     nodev,nosuid             0   0

/dev/sdb1    /             ext4      defaults                 0   1

Remember to use the proper device name and filesystem type.

Update the bootloader's configuration file

This section assumes that you backed up the system to another drive or partition, that your current bootloader is working fine, and that you want to boot from the backup as well.

For Syslinux, all you need to do is duplicate the current entry, except pointing to a different drive or partition:

Tip: Instead of editing syslinux.cfg, you can also temporarily edit the menu during boot. When the menu shows up, press the Template:Keypress key and change the relevant entries. Partitions are counted from one, drives are counted from zero.
# nano /boot/syslinux/syslinux.cfg

For GRUB, it's recommended that you automatically re-generate the grub.cfg file:

# pacman -S os-prober
# grub-mkconfig -o /boot/grub/grub.cfg