Difference between revisions of "Full system backup with rsync"

From ArchWiki
Jump to: navigation, search
m (Using a script)
(With a single command: there is no such thing as "Linux compatible file system")
 
(127 intermediate revisions by 42 users not shown)
Line 1: Line 1:
 
[[Category:System recovery]]
 
[[Category:System recovery]]
[[cs:Full System Backup with rsync]]
+
[[cs:Full system backup with rsync]]
{{Article summary start}}
+
[[es:Full system backup with rsync]]
{{Article summary text|Instructions on backing up the root tree, creating a bootable copy of your system, or for transferring your system to a new drive or partition.}}
+
[[ja:Rsync によるフルシステムバックアップ]]
{{Article summary heading|Related}}
+
[[zh-tw:Full system backup with rsync]]
{{Article summary wiki|Backup Programs}}
+
{{Related articles start}}
{{Article summary wiki|rsync}}
+
{{Related|Synchronization and backup programs}}
{{Article summary end}}
+
{{Related|rsync}}
 +
{{Related articles end}}
 +
This article is about using [[rsync]] to transfer a copy of your "/" tree, excluding a few select folders. This approach is considered to be better than [[disk cloning]] with {{ic|dd}} since it allows for a different size, partition table and filesystem to be used, and better than copying with {{ic|cp -a}} as well, because it allows greater control over file permissions, attributes, [[Access Control Lists]] and [[extended attributes]].
  
This article is about using [[rsync]] to transfer a copy of your "/" tree, excluding a few select folders. This approach is considered to be better than [[disk cloning]] with {{ic|dd}} since it allows for a different size, partition table and filesystem to be used, and better than copying with {{ic|cp -a}} as well, because it allows greater control over file permissions, attributes, Access Control Lists (ACLs) and extended attributes. [http://www.bestbits.at/acl/about.html]
+
Either method will work even while the system is running, but files changed during the transfer may or may not be transferred, which could cause undefined behaviour of some programs using the transferred files.
  
Either method will work even while the system is running. Since it's going to take a while, you may freely browse the web during this time. Worst case scenario you won't get the same opened tabs when you restore the backup (or boot from it) because they weren't saved. Not a big deal.
+
This approach will also work well for migrating an existing installation to a new harddrive or [[SSD]].
  
 
== With a single command ==
 
== With a single command ==
  
As root, run:
+
This command depends on brace expansion available in both the [https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html bash] and [http://zsh.sourceforge.net/Doc/Release/Expansion.html#Brace-Expansion zsh] shells. When using a different [[shell]], {{ic|--exclude}} patterns should be repeated manually.
  
{{Note|If you plan on backing up your system somewhere other than {{ic|/mnt}} or {{ic|/media}}, don't forget to add it to the list, to avoid an infinite loop.}}
+
# rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / ''/path/to/backup/folder''
  
# rsync -aAXv /* /path/to/backup/folder --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs}
+
Using the {{ic|-aAX}} set of options, the files are transferred in archive mode, ensuring that symbolic links, devices, permissions and ownerships, modification times, [[ACL]]s and extended attributes are preserved (assuming that the target [[file system]] supports the feature).
  
For information on why these folders were excluded, read the next section.
+
The {{ic|--exclude}} option will cause files that match the given patterns to be excluded. The contents of {{ic|/dev}}, {{ic|/proc}}, {{ic|/sys}}, {{ic|/tmp}} and {{ic|/run}} were excluded because they are populated at boot (while the folders themselves are ''not'' created), {{ic|/lost+found}} is filesystem-specific. Quoting the exclude patterns will avoid expansion by [[shell]], which is necessary e.g. when backing up over [[SSH]]. Ending the excluded paths with {{ic|*}} will still ensure that the directories themselves are created if not already existing.
  
== Using a script ==
+
{{Note|
 +
* If you plan on backing up your system somewhere other than {{ic|/mnt}} or {{ic|/media}}, do not forget to add it to the list of exclude patterns to avoid an infinite loop.
 +
* If there are any bind mounts in the system, they should be excluded as well, so that the bind mounted contents is copied only once.
 +
* If you use a [[swap file]], make sure to exclude it as well.
 +
* Consider also if you want to backup the {{ic|/home/}} folder. If it contains your data, it might be considerably larger than the system. Otherwise consider excluding unimportant subdirectories such as  {{ic|/home/*/.thumbnails/*}}, {{ic|/home/*/.cache/mozilla/*}}, {{ic|/home/*/.cache/chromium/*}}, {{ic|/home/*/.local/share/Trash/*}}, depending on software installed on the system. If [[GVFS]] is installed, {{ic|/home/*/.gvfs}} must be excluded to prevent rsync errors.
 +
}}
  
Same as in the above method, the system files are transferred in archive mode, ensuring that symbolic links, devices, permissions and ownerships, among other file attributes are preserved, while excluding files that match the patterns from the {{ic|--exclude}} string. On top of that, it shows at the end how much time it took, and it also writes a blank file stating when the backup was created. To learn more about what this script does, read {{ic|man rsync}} and {{ic|man date}}.
+
You may want to include additional [[rsync]] options, such as the following (see {{ic|man rsync}} for the full list):
  
{{Note|Again, if you plan on backing up your system somewhere other than {{ic|/mnt}} or {{ic|/media}}, don't forget to add it to the list, to avoid an infinite loop.}}
+
* If you are a heavy user of hardlinks, you might consider adding the {{ic|-H}} option, which is turned off by default as memory expensive, but nowadays it should be no problem on most modern machines. There are a lot of hard links below the {{ic|/usr/}} folder, which save disk space.
 +
* You may want to add rsync's {{ic|--delete}} option if you are running this multiple times to the same backup folder. In this case make sure that the source path does not end with {{ic|/*}}, or this option will only have effect on the files inside the subdirectories of the source directory, but no effect on the files residing directly inside the source directory.
 +
* If you use any sparse files, such as virtual disks, [[Docker]] images and similar, you should add the {{ic|-S}} option.
 +
* The {{ic|--numeric-ids}} option will disable mapping of user and group names, numeric group and user IDs will be transfered instead. This is useful when backing up over [[SSH]] or when using a live system to backup different system disk.
 +
* Choosing {{ic|1=--info=progress2}} option instead of {{ic|-v}} will show overal progress info and transfer speed instead of huge list of files.
  
{{hc|$ cd ~/Scripts
+
If you wish to restore the backup use the same rsync command that was executed, but with the source and destination reversed.
$ nano backup.sh|<nowiki>
+
#!/bin/sh
+
 
+
if [ $# -lt 1 ]; then
+
    echo "No destination defined. Usage: $0 destination" >&2
+
    exit 1
+
elif [ $# -gt 1 ]; then
+
    echo "Too many arguments. Usage: $0 destination" >&2
+
    exit 1
+
fi
+
 
+
START=$(date +%s)
+
rsync -aAXv /* $1 --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,/var/lib/pacman/sync/*}
+
FINISH=$(date +%s)
+
echo "total time: $(( ($FINISH-$START) / 60 )) minutes, $(( ($FINISH-$START) % 60 )) seconds"
+
 
+
touch $1/"Backup from $(date '+%A, %d %B %Y, %T')"</nowiki>}}
+
 
+
$ chmod +x backup.sh
+
 
+
{{Note|The contents of {{ic|/dev}}, {{ic|/proc}}, {{ic|/sys}}, {{ic|/tmp}}, {{ic|/run}} were excluded because they are populated at boot (while the folders themselves are ''not'' created), {{ic|/lost+found}} is filesystem-specific, and {{ic|/home/*/.gvfs}} should be added too, so that it won't complain at the end that "some files/attrs were not transferred". For Arch Linux, {{ic|/var/lib/pacman/sync/*}} can also be excluded. This can save a lot of time on every backup since the directory contains many small files that tend to change quite often. These are description files for every package from the repositories and can be re-generated with {{ic|pacman -Syu}}. Additionally, you may also want to skip {{ic|/home/*/.thumbnails/*}}, {{ic|/home/*/.mozilla/firefox/*.default/Cache/*}} and {{ic|/home/*/.cache/chromium/*}}.}}
+
 
+
Backing up is easy.
+
 
+
While the system is running, open up a terminal and run (as root):
+
 
+
# /home/user/Scripts/backup.sh /some/destination
+
(replace user with username since you created the directory as user in the user's home directory)
+
 
+
You can also replace both {{ic|$1}} instances from the script with the actual destination path, move it to one of the folders from {{ic|echo $PATH}}, and then simply run (as root):
+
 
+
# backup.sh
+
  
 
== Boot requirements ==
 
== Boot requirements ==
Line 69: Line 48:
  
 
Without rebooting, edit the backup's [[fstab]] to reflect the changes:
 
Without rebooting, edit the backup's [[fstab]] to reflect the changes:
{{hc|# nano /path/to/backup/etc/fstab|2=
+
{{hc|/path/to/backup/etc/fstab|2=
 
tmpfs        /tmp          tmpfs    nodev,nosuid            0  0
 
tmpfs        /tmp          tmpfs    nodev,nosuid            0  0
  
Line 79: Line 58:
 
Because rsync has performed a recursive copy of the ''entire'' root filesystem, all of the {{ic|sda}} mountpoints are problematic and booting the backup will fail. In this example, all of the offending entries are replaced with a single one:
 
Because rsync has performed a recursive copy of the ''entire'' root filesystem, all of the {{ic|sda}} mountpoints are problematic and booting the backup will fail. In this example, all of the offending entries are replaced with a single one:
  
{{hc|# nano /path/to/backup/etc/fstab|
+
{{hc|/path/to/backup/etc/fstab|
 
tmpfs        /tmp          tmpfs    nodev,nosuid            0  0
 
tmpfs        /tmp          tmpfs    nodev,nosuid            0  0
  
Line 90: Line 69:
 
This section assumes that you backed up the system to another drive or partition, that your current bootloader is working fine, and that you want to boot from the backup as well.
 
This section assumes that you backed up the system to another drive or partition, that your current bootloader is working fine, and that you want to boot from the backup as well.
  
For [[Syslinux]], all you need to do is duplicate the current entry, except pointing to a different drive or partition:
+
For [[Syslinux]], all you need to do is duplicate the current entry, except pointing to a different drive or partition.
 +
 
 +
{{Tip|Instead of editing {{ic|syslinux.cfg}}, you can also temporarily edit the menu during boot. When the menu shows up, press the {{ic|Tab}} key and change the relevant entries. Partitions are counted from one, drives are counted from zero.}}
 +
 
 +
For [[GRUB]], it is recommended that you automatically [[GRUB#Generate_the_main_configuration_file|re-generate the main configuration file]].
 +
 
 +
Also verify the new menu entry in {{ic|/boot/grub/grub.cfg}}. Make sure the UUID is matching the new partition, otherwise it could still boot the old system. Find the UUID of a partition as follows:
 +
 
 +
# lsblk -no NAME,UUID /dev/sdb3
 +
 
 +
where you substitute the desired partition for /dev/sdb3. To list the UUIDs of partitions grub thinks it can boot, use grep:
 +
 
 +
# grep UUID= /boot/grub/grub.cfg
 +
 
 +
{{Accuracy|How is mkinitcpio related to the UUID problem?}}
 +
 
 +
If the one you found from lsblk is not found by grep, then grub-mkconfig did not work. Most likely, you will have to [[Change root]] into the duplicate file system and then use [[mkinitcpio]]. For example, if you had used rsync to duplicate root on /dev/sdb3 then change root and use mkinitcpio as follows:
 +
 
 +
# mkdir /mnt/arch
 +
# mount /dev/sdb3 /mnt/arch
 +
# cd /mnt/arch
 +
# mount -t proc proc proc/
 +
# mount --rbind /sys sys/
 +
# mount --rbind /dev dev/
 +
# chroot /mnt/arch /bin/bash
 +
# mkinitcpio -p linux
 +
 
 +
After exiting, verify the new UUID is included.
 +
 
 +
== First boot ==
 +
 
 +
Reboot the computer and select the right entry in the bootloader. This will load the system for the first time. All peripherals should be detected and the empty folders in {{ic|/}} will be populated.
 +
 
 +
Now you can re-edit {{ic|/etc/fstab}} to add the previously removed partitions and mount points.
  
{{Tip|Instead of editing {{ic|syslinux.cfg}}, you can also temporarily edit the menu during boot. When the menu shows up, press the {{Keypress|Tab}} key and change the relevant entries. Partitions are counted from one, drives are counted from zero.}}
+
If you transferred the data from HDD to SSD (solid state drive), do not forget to activate TRIM. Also consider using HDD and tmpfs mount points to reduce SSD wearing - see [[Maximizing performance#Relocate files to tmpfs]] and [[Solid State Drives#Tips for minimizing disk reads/writes]].
  
# nano /boot/syslinux/syslinux.cfg
+
{{Note|You may have to reboot again in order to get all services and daemons working correctly. Personally, pulseaudio would not initialise because of a module loading error. I restarted the dbus.service to make it work.}}
  
For [[GRUB]], it's recommended that you automatically re-generate the {{ic|grub.cfg}} file:
+
== See also ==
  
# pacman -S os-prober
+
* [http://blog.pointsoftware.ch/index.php/howto-local-and-remote-snapshot-backup-using-rsync-with-hard-links/ Howto – local and remote snapshot backup using rsync with hard links] Includes file deduplication with hard-links, MD5 integrity signature, 'chattr' protection, filter rules, disk quota, retention policy with exponential distribution (backups rotation while saving more recent backups than older)
# grub-mkconfig -o /boot/grub/grub.cfg
+

Latest revision as of 07:50, 4 June 2016

This article is about using rsync to transfer a copy of your "/" tree, excluding a few select folders. This approach is considered to be better than disk cloning with dd since it allows for a different size, partition table and filesystem to be used, and better than copying with cp -a as well, because it allows greater control over file permissions, attributes, Access Control Lists and extended attributes.

Either method will work even while the system is running, but files changed during the transfer may or may not be transferred, which could cause undefined behaviour of some programs using the transferred files.

This approach will also work well for migrating an existing installation to a new harddrive or SSD.

With a single command

This command depends on brace expansion available in both the bash and zsh shells. When using a different shell, --exclude patterns should be repeated manually.

# rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup/folder

Using the -aAX set of options, the files are transferred in archive mode, ensuring that symbolic links, devices, permissions and ownerships, modification times, ACLs and extended attributes are preserved (assuming that the target file system supports the feature).

The --exclude option will cause files that match the given patterns to be excluded. The contents of /dev, /proc, /sys, /tmp and /run were excluded because they are populated at boot (while the folders themselves are not created), /lost+found is filesystem-specific. Quoting the exclude patterns will avoid expansion by shell, which is necessary e.g. when backing up over SSH. Ending the excluded paths with * will still ensure that the directories themselves are created if not already existing.

Note:
  • If you plan on backing up your system somewhere other than /mnt or /media, do not forget to add it to the list of exclude patterns to avoid an infinite loop.
  • If there are any bind mounts in the system, they should be excluded as well, so that the bind mounted contents is copied only once.
  • If you use a swap file, make sure to exclude it as well.
  • Consider also if you want to backup the /home/ folder. If it contains your data, it might be considerably larger than the system. Otherwise consider excluding unimportant subdirectories such as /home/*/.thumbnails/*, /home/*/.cache/mozilla/*, /home/*/.cache/chromium/*, /home/*/.local/share/Trash/*, depending on software installed on the system. If GVFS is installed, /home/*/.gvfs must be excluded to prevent rsync errors.

You may want to include additional rsync options, such as the following (see man rsync for the full list):

  • If you are a heavy user of hardlinks, you might consider adding the -H option, which is turned off by default as memory expensive, but nowadays it should be no problem on most modern machines. There are a lot of hard links below the /usr/ folder, which save disk space.
  • You may want to add rsync's --delete option if you are running this multiple times to the same backup folder. In this case make sure that the source path does not end with /*, or this option will only have effect on the files inside the subdirectories of the source directory, but no effect on the files residing directly inside the source directory.
  • If you use any sparse files, such as virtual disks, Docker images and similar, you should add the -S option.
  • The --numeric-ids option will disable mapping of user and group names, numeric group and user IDs will be transfered instead. This is useful when backing up over SSH or when using a live system to backup different system disk.
  • Choosing --info=progress2 option instead of -v will show overal progress info and transfer speed instead of huge list of files.

If you wish to restore the backup use the same rsync command that was executed, but with the source and destination reversed.

Boot requirements

Having a bootable backup can be useful in case the filesystem becomes corrupt or if an update breaks the system. The backup can also be used as a test bed for updates, with the [testing] repo enabled, etc. If you transferred the system to a different partition or drive and you want to boot it, the process is as simple as updating the backup's /etc/fstab and your bootloader's configuration file.

Update the fstab

Without rebooting, edit the backup's fstab to reflect the changes:

/path/to/backup/etc/fstab
tmpfs        /tmp          tmpfs     nodev,nosuid             0   0

/dev/sda1    /boot         ext2      defaults                 0   2
/dev/sda5    none          swap      defaults                 0   0
/dev/sda6    /             ext4      defaults                 0   1
/dev/sda7    /home         ext4      defaults                 0   2

Because rsync has performed a recursive copy of the entire root filesystem, all of the sda mountpoints are problematic and booting the backup will fail. In this example, all of the offending entries are replaced with a single one:

/path/to/backup/etc/fstab
tmpfs        /tmp          tmpfs     nodev,nosuid             0   0

/dev/sdb1    /             ext4      defaults                 0   1

Remember to use the proper device name and filesystem type.

Update the bootloader's configuration file

This section assumes that you backed up the system to another drive or partition, that your current bootloader is working fine, and that you want to boot from the backup as well.

For Syslinux, all you need to do is duplicate the current entry, except pointing to a different drive or partition.

Tip: Instead of editing syslinux.cfg, you can also temporarily edit the menu during boot. When the menu shows up, press the Tab key and change the relevant entries. Partitions are counted from one, drives are counted from zero.

For GRUB, it is recommended that you automatically re-generate the main configuration file.

Also verify the new menu entry in /boot/grub/grub.cfg. Make sure the UUID is matching the new partition, otherwise it could still boot the old system. Find the UUID of a partition as follows:

# lsblk -no NAME,UUID /dev/sdb3

where you substitute the desired partition for /dev/sdb3. To list the UUIDs of partitions grub thinks it can boot, use grep:

# grep UUID= /boot/grub/grub.cfg

Tango-inaccurate.pngThe factual accuracy of this article or section is disputed.Tango-inaccurate.png

Reason: How is mkinitcpio related to the UUID problem? (Discuss in Talk:Full system backup with rsync#)

If the one you found from lsblk is not found by grep, then grub-mkconfig did not work. Most likely, you will have to Change root into the duplicate file system and then use mkinitcpio. For example, if you had used rsync to duplicate root on /dev/sdb3 then change root and use mkinitcpio as follows:

# mkdir /mnt/arch
# mount /dev/sdb3 /mnt/arch
# cd /mnt/arch
# mount -t proc proc proc/
# mount --rbind /sys sys/
# mount --rbind /dev dev/
# chroot /mnt/arch /bin/bash
# mkinitcpio -p linux

After exiting, verify the new UUID is included.

First boot

Reboot the computer and select the right entry in the bootloader. This will load the system for the first time. All peripherals should be detected and the empty folders in / will be populated.

Now you can re-edit /etc/fstab to add the previously removed partitions and mount points.

If you transferred the data from HDD to SSD (solid state drive), do not forget to activate TRIM. Also consider using HDD and tmpfs mount points to reduce SSD wearing - see Maximizing performance#Relocate files to tmpfs and Solid State Drives#Tips for minimizing disk reads/writes.

Note: You may have to reboot again in order to get all services and daemons working correctly. Personally, pulseaudio would not initialise because of a module loading error. I restarted the dbus.service to make it work.

See also