Difference between revisions of "Rsync"

From ArchWiki
Jump to: navigation, search
(Configuration)
(wikify some external links, use https for archlinux.org)
(40 intermediate revisions by 21 users not shown)
Line 1: Line 1:
{{stub}}
+
{{Lowercase_title}}
= Introduction =
+
[[Category:Data compression and archiving]]
The rsync protocol can be used to make quick and easy backups, as it only transfers files that have changed, which means backups are much quicker, than if you just copied all your files over every time.
+
[[Category:Networking]]
This tutorial aims to show you how to make a scheduled backup, using rsync, typically to a removable media.
+
[[Category:System recovery]]
 +
[[zh-CN:Rsync]]
 +
{{Article summary start}}
 +
{{Article summary text|Instructions on using rsync.}}
 +
{{Article summary heading|Related}}
 +
{{Article summary wiki|Full System Backup with rsync}}
 +
{{Article summary wiki|Backup Programs}}
 +
{{Article summary end}}
 +
 
 +
[http://samba.anu.edu.au/rsync/ rsync] is an open source utility that provides fast incremental file transfer.
 +
 
 +
== Installation ==
 +
 
 +
Install the {{Pkg|rsync}} package using [[pacman]]:
  
= Prerequisites =
 
Obviously, to use rsync, you need rsync:
 
 
  # pacman -S rsync
 
  # pacman -S rsync
  
= Configuration =
+
== Usage ==
We need to make a script that tells rsync to backup your data.
+
 
 +
For more examples, search the [https://bbs.archlinux.org/viewforum.php?id=27 Community Contributions] and [https://bbs.archlinux.org/viewforum.php?id=33 General Programming] forums.
 +
 
 +
=== As a cp alternative ===
 +
 
 +
rsync can be used as an advanced alternative for the {{ic|cp}} command, especially for copying larger files:
 +
 
 +
$ rsync -P source destination
 +
 
 +
The {{Ic|-P}} option is the same as {{Ic|--partial --progress}}, which keeps partially transferred files and shows a progress bar during transfer.
 +
 
 +
You may want to use the {{Ic|-r --recursive}} option to recurse into directories, or the {{Ic|-R}} option for using relative path names (recreating entire folder hierarchy on the destination folder).
 +
 
 +
=== As a backup utility ===
 +
 
 +
The rsync protocol can easily be used for backups, only transferring files that have changed since the last backup. This section describes a very simple scheduled backup script using rsync, typically used for copying to removable media. For a more thorough example and '''additional options required to preserve some system files''', see [[Full System Backup with rsync]].
 +
 
 +
==== Automated backup ====
 +
 
 +
For the sake of this example, the script is created in the {{ic|/etc/cron.daily}} directory, and will be run on a daily basis if a cron [[daemon]] is installed and properly configured. Configuring and using [[cron]] is outside the scope of this article.
 +
 
 +
First, create a script containing the appropriate command options:
 +
 
 +
{{hc|/etc/cron.daily/backup|
 +
#!/bin/bash
 +
rsync -a --delete /folder/to/backup /location/to/backup &> /dev/null}}
 +
 
 +
; {{Ic|-a}} : indicates that files should be archived, meaning that most of their characteristics are preserved (but '''not''' ACLs, hard links or extended attributes such as capabilities)
 +
; {{Ic|--delete}} : means files deleted on the source are to be deleted on the backup aswell
 +
 
 +
Here, {{ic|/folder/to/backup}} should be changed to what needs to be backed-up ({{ic|/home}}, for example) and {{ic|/location/to/backup}} is where the backup should be saved ({{ic|/media/disk}}, for instance).
 +
 
 +
Finally, the script must be executable:
 +
 
 +
# chmod +x /etc/cron.daily/rsync.backup
 +
 
 +
==== Automated backup with SSH ====
 +
 
 +
If backing-up to a remote host using [[SSH]], use this script instead:
 +
 
 +
{{hc|/etc/cron.daily/backup|
 +
#!/bin/bash
 +
rsync -a --delete -e ssh /folder/to/backup remoteuser@remotehost:/location/to/backup &> /dev/null}}
 +
 
 +
; {{Ic|-e ssh}} : tells rsync to use SSH
 +
; {{Ic|remoteuser}} : is the user on the host {{Ic|remotehost}}
 +
; {{Ic|-a}} : groups all these options {{Ic|-rlptgoD}} (recursive, links, perms, times, group, owner, devices)
 +
 
 +
==== Automated backup with NetworkManager ====
 +
 
 +
This script starts a backup when you plugin your wire.
 +
 
 +
First, create a script containing the appropriate command options:
 +
 
 +
{{hc|/etc/NetworkManager/dispatcher.d/backup|2=
 +
#!/bin/bash
 +
 
 +
if [ x"$2" = "xup" ] ; then
 +
  rsync --force --ignore-errors -a --delete --bwlimit=2000 --files-from=files.rsync /folder/to/backup /location/to/backup
 +
fi}}
 +
 
 +
; {{Ic|-a}} : group all this options {{Ic|-rlptgoD}} recursive, links, perms, times, group, owner, devices
 +
; {{Ic|--files-from}} : read the relative path of ''/folder/to/backup'' from this file
 +
; {{Ic|--bwlimit}} : limit I/O bandwidth; KBytes per second
 +
 
 +
==== Differential backup on a week ====
 +
 
 +
This is a useful option of rsync, creating a full backup and a differential backup for each day of a week.
 +
 
 +
First, create a script containing the appropriate command options:
 +
 
 +
{{hc|/etc/cron.daily/backup|2=
 +
#!/bin/bash
 +
 
 +
DAY=$(date +%A)
 +
 
 +
if [ -e /location/to/backup/incr/$DAY ] ; then
 +
  rm -fr /location/to/backup/incr/$DAY
 +
fi
 +
 
 +
rsync -a --delete --inplace --backup --backup-dir=/location/to/backup/incr/$DAY /folder/to/backup/ /location/to/backup/full/ &> /dev/null}}
 +
 
 +
; {{Ic|--inplace}} : implies {{Ic|--partial}} update destination files in-place
 +
 
 +
==== Snapshot backup ====
  
Open a text editor, for example:
+
The same idea can be used to maintain a tree of snapshots of your files. In other words, a directory with date-ordered copies of the files. The copies are made using hardlinks, which means that only files that did change will occupy space. Generally speaking, this is the idea behind Apple's TimeMachine.
# nano /etc/cron.daily/rsync.backup
+
In this location, it will be run daily, without any further configuration.
+
  
You need to paste this script into the text editor:
+
This script implements a simple version of it:
  
#!/bin/bash
+
{{hc|/usr/local/bin/rsnapshot.sh|<nowiki>
rsync -ar --delete /folder_to_backup/ /location_to_backup/ &> /dev/null
+
#!/bin/bash
  
-a means files are archived, -r means files are copied recursively, and --delete means files deleted on the source are deleted on the backup
+
## my own rsync-based snapshot-style backup procedure
 +
## (cc) marcio rps AT gmail.com
  
Here, "folder_to_backup" needs to be changed to what you want to backup (/home/ , for example) and "location_to_backup" is where you want to back it up to (/media/disk , for example).
+
# config vars
  
If you want to backup to a remote host using [[SSH]], use this script instead:
+
SRC="/home/username/files/" #dont forget trailing slash!
 +
SNAP="/snapshots/username"
 +
OPTS="-rltgoi --delay-updates --delete --chmod=a-w"
 +
MINCHANGES=20
  
#!/bin/bash
+
# run this process with real low priority
rsync -ar --delete -e ssh /folder_to_backup/ remoteuser@remotehost:/location_to_backup/ &> /dev/null
+
  
-e ssh tells rsync to use ssh, remoteuser is the user on the host remotehost
+
ionice -c 3 -p $$
 +
renice +12  -p $$
  
We now need to make it executable:
+
# sync
#chmod +x /etc/cron.daily/rsync.backup
+
  
And that's it. It'll now run daily, as long as your cron daemon is configured correctly (which is out of the scope of this document).
+
rsync $OPTS $SRC $SNAP/latest >> $SNAP/rsync.log
  
To check if rsync actually has preformed a backup you can use a log, here's a simple example:
+
# check if enough has changed and if so
#echo  beginning backup `date` >> ~/RsyncBackupLog
+
# make a hardlinked copy named as the date
Put this line in your backup script and it will write "beginning backup 02-01-2009" in the text file RsyncBackupLog now you can see when the last backup is made.
+
  
You can also log the output of rsync:
+
COUNT=$( wc -l $SNAP/rsync.log|cut -d" " -f1 )
#rsync -ar --delete /folder_to_backup/ /location_to_backup/ >> ~/Rsync.log
+
if [ $COUNT -gt $MINCHANGES ] ; then
The log can be find in your home directory and it's called Rsync.log
+
  DATETAG=$(date +%Y-%m-%d)
 +
  if [ ! -e $SNAP/$DATETAG ] ; then
 +
      cp -al $SNAP/latest $SNAP/$DATETAG
 +
      mv $SNAP/rsync.log $SNAP/$DATETAG
 +
  fi
 +
fi
 +
</nowiki>}}
  
When you are making a backup to a server you can build in a check if the server is online:
+
To make things really, really simple this script can be run out of {{ic|/etc/rc.local}}.
#if ping -c 1 -w 5 10.0.0.102 > /dev/null ;
+
then
+
mount the share
+
rsync -ar --delete -e ssh /folder_to_backup/ remoteuser@remotehost:/location_to_backup/ &> /dev/null
+
else
+
echo echo  backup failed `date` >> ~/RsyncBackupLog
+
fi
+
+
For more examples of really great bash scripting you should just google for 'rsync backup script' or go to the 'General Programming Forum' of the Arch forum
+

Revision as of 14:37, 3 December 2012

Template:Article summary start Template:Article summary text Template:Article summary heading Template:Article summary wiki Template:Article summary wiki Template:Article summary end

rsync is an open source utility that provides fast incremental file transfer.

Installation

Install the rsync package using pacman:

# pacman -S rsync

Usage

For more examples, search the Community Contributions and General Programming forums.

As a cp alternative

rsync can be used as an advanced alternative for the cp command, especially for copying larger files:

$ rsync -P source destination

The -P option is the same as --partial --progress, which keeps partially transferred files and shows a progress bar during transfer.

You may want to use the -r --recursive option to recurse into directories, or the -R option for using relative path names (recreating entire folder hierarchy on the destination folder).

As a backup utility

The rsync protocol can easily be used for backups, only transferring files that have changed since the last backup. This section describes a very simple scheduled backup script using rsync, typically used for copying to removable media. For a more thorough example and additional options required to preserve some system files, see Full System Backup with rsync.

Automated backup

For the sake of this example, the script is created in the /etc/cron.daily directory, and will be run on a daily basis if a cron daemon is installed and properly configured. Configuring and using cron is outside the scope of this article.

First, create a script containing the appropriate command options:

/etc/cron.daily/backup
#!/bin/bash
rsync -a --delete /folder/to/backup /location/to/backup &> /dev/null
-a 
indicates that files should be archived, meaning that most of their characteristics are preserved (but not ACLs, hard links or extended attributes such as capabilities)
--delete 
means files deleted on the source are to be deleted on the backup aswell

Here, /folder/to/backup should be changed to what needs to be backed-up (/home, for example) and /location/to/backup is where the backup should be saved (/media/disk, for instance).

Finally, the script must be executable:

# chmod +x /etc/cron.daily/rsync.backup

Automated backup with SSH

If backing-up to a remote host using SSH, use this script instead:

/etc/cron.daily/backup
#!/bin/bash
rsync -a --delete -e ssh /folder/to/backup remoteuser@remotehost:/location/to/backup &> /dev/null
-e ssh 
tells rsync to use SSH
remoteuser 
is the user on the host remotehost
-a 
groups all these options -rlptgoD (recursive, links, perms, times, group, owner, devices)

Automated backup with NetworkManager

This script starts a backup when you plugin your wire.

First, create a script containing the appropriate command options:

/etc/NetworkManager/dispatcher.d/backup
#!/bin/bash

if [ x"$2" = "xup" ] ; then
  rsync --force --ignore-errors -a --delete --bwlimit=2000 --files-from=files.rsync /folder/to/backup /location/to/backup
fi
-a 
group all this options -rlptgoD recursive, links, perms, times, group, owner, devices
--files-from 
read the relative path of /folder/to/backup from this file
--bwlimit 
limit I/O bandwidth; KBytes per second

Differential backup on a week

This is a useful option of rsync, creating a full backup and a differential backup for each day of a week.

First, create a script containing the appropriate command options:

/etc/cron.daily/backup
#!/bin/bash

DAY=$(date +%A)

if [ -e /location/to/backup/incr/$DAY ] ; then
  rm -fr /location/to/backup/incr/$DAY
fi

rsync -a --delete --inplace --backup --backup-dir=/location/to/backup/incr/$DAY /folder/to/backup/ /location/to/backup/full/ &> /dev/null
--inplace 
implies --partial update destination files in-place

Snapshot backup

The same idea can be used to maintain a tree of snapshots of your files. In other words, a directory with date-ordered copies of the files. The copies are made using hardlinks, which means that only files that did change will occupy space. Generally speaking, this is the idea behind Apple's TimeMachine.

This script implements a simple version of it:

/usr/local/bin/rsnapshot.sh
#!/bin/bash

## my own rsync-based snapshot-style backup procedure
## (cc) marcio rps AT gmail.com

# config vars

SRC="/home/username/files/" #dont forget trailing slash!
SNAP="/snapshots/username"
OPTS="-rltgoi --delay-updates --delete --chmod=a-w"
MINCHANGES=20

# run this process with real low priority

ionice -c 3 -p $$
renice +12  -p $$

# sync

rsync $OPTS $SRC $SNAP/latest >> $SNAP/rsync.log

# check if enough has changed and if so
# make a hardlinked copy named as the date

COUNT=$( wc -l $SNAP/rsync.log|cut -d" " -f1 )
if [ $COUNT -gt $MINCHANGES ] ; then
   DATETAG=$(date +%Y-%m-%d)
   if [ ! -e $SNAP/$DATETAG ] ; then
      cp -al $SNAP/latest $SNAP/$DATETAG
      mv $SNAP/rsync.log $SNAP/$DATETAG
   fi
fi

To make things really, really simple this script can be run out of /etc/rc.local.