https://wiki.archlinux.org/api.php?action=feedcontributions&user=Rpbritton&feedformat=atomArchWiki - User contributions [en]2024-03-28T15:09:05ZUser contributionsMediaWiki 1.41.0https://wiki.archlinux.org/index.php?title=Rsync&diff=529831Rsync2018-07-15T18:25:06Z<p>Rpbritton: Changed the misspelled /location/to/backup to /location/of/backup to match the example command above it.</p>
<hr />
<div>{{Lowercase title}}<br />
[[Category:Data compression and archiving]]<br />
[[Category:System recovery]]<br />
[[es:Rsync]]<br />
[[ja:Rsync]]<br />
[[zh-hans:Rsync]]<br />
{{Related articles start}}<br />
{{Related|System backup}}<br />
{{Related|Synchronization and backup programs}}<br />
{{Related articles end}}<br />
[https://rsync.samba.org/ rsync] is an open source utility that provides fast incremental file transfer.<br />
<br />
== Installation ==<br />
<br />
[[Install]] the {{Pkg|rsync}} package.<br />
<br />
''rsync'' must be installed on both the source and the destination machine.<br />
<br />
=== Front-ends ===<br />
<br />
* {{App|Grsync|GTK+ front-end.|http://www.opbyte.it/grsync/|{{Pkg|grsync}}}}<br />
* {{App|gutback|rsync wrapper written in Shell.|https://github.com/gutenye/gutbackup|{{Aur|gutbackup}}}}<br />
* {{App|JotaSync|Java Swing GUI for rsync with integrated scheduler.|https://trixon.se/projects/jotasync/|{{Aur|jotasync}}}}<br />
* {{App|luckyBackup|Qt front-end written in C++.|http://luckybackup.sourceforge.net/index.html|{{Aur|luckybackup}}}}<br />
<br />
== As a cp alternative ==<br />
<br />
rsync can be used as an advanced alternative for the {{ic|cp}} command, especially for copying larger files:<br />
<br />
$ rsync -P source destination<br />
<br />
The {{ic|-P}} option is the same as {{Ic|--partial --progress}}, which keeps partially transferred files and shows a progress bar during transfer.<br />
<br />
You may want to use the {{ic|-r}}/{{ic|--recursive}} option to recurse into directories.<br />
<br />
Files can be copied locally as with cp, but the motivating purpose of rsync is to copy files remotely, i.e. between two different hosts. Remote locations can be specified with a host-colon syntax:<br />
<br />
$ rsync source host:destination<br />
<br />
or<br />
<br />
$ rsync host:source destination<br />
<br />
Network file transfers use the [[SSH]] protocol by default and {{ic|host}} can be a real hostname or a predefined profile/alias from {{ic|.ssh/config}}.<br />
<br />
{{Accuracy|By default, rsync does not compare contents of files. Rsync uses a "quick check" that (by default) checks if each file’s size and time of last modification match between the sender and receiver.}}<br />
<br />
Whether transferring files locally or remotely, rsync first creates an index of block checksums of each source file. This index is used to find any identical blocks of data which might exist in the destination. Such blocks are used in-place, rather than being copied from the source. This can greatly accelerate the synchronization of large files with small changes. For more information, see [https://rsync.samba.org/documentation.html official documentation], [https://rsync.samba.org/how-rsync-works.html how rsync works].<br />
<br />
=== Trailing slash caveat ===<br />
<br />
Arch by default uses GNU cp (part of GNU {{Pkg|coreutils}}). However, rsync follows the convention of BSD cp, which gives special treatment to source directories with a trailing slash "/". Although<br />
<br />
$ rsync -r source destination<br />
<br />
creates a directory "destination/source" with the contents of "source", the command<br />
<br />
$ rsync -r source/ destination<br />
<br />
copies all of the files in "source/" directly into "destination", with no intervening subdirectory - just as if you had invoked it as<br />
<br />
$ rsync -r source/. destination<br />
<br />
This behavior is different from that of GNU cp, which treats "source" and "source/" identically (but not "source/."). Also, some shells automatically append the trailing slash when tab-completing directory names. Because of these factors, there can be a tendency among new or occasional rsync users to forget about rsync's different behavior, and inadvertently create a mess or even overwrite important files by leaving the trailing slash on the command line.<br />
<br />
Thus it can be prudent to use a wrapper script to automatically remove trailing slashes before invoking rsync:<br />
<br />
#!/bin/zsh<br />
new_args=();<br />
for i in "$@"; do<br />
case $i in /) i=/;; */) i=${i%/};; esac<br />
new_args+=$i;<br />
done<br />
exec rsync "${(@)new_args}"<br />
<br />
This script can be put somewhere in the path, and aliased to rsync in the shell init file.<br />
<br />
== As a backup utility ==<br />
<br />
The rsync protocol can easily be used for backups, only transferring files that have changed since the last backup. This section describes a very simple scheduled backup script using rsync, typically used for copying to removable media.<br />
<br />
=== Automated backup ===<br />
<br />
For the sake of this example, the script is created in the {{ic|/etc/cron.daily}} directory, and will be run on a daily basis if a cron [[daemon]] is installed and properly configured. Configuring and using [[cron]] is outside the scope of this article.<br />
<br />
First, create a script containing the appropriate command options:<br />
<br />
{{hc|/etc/cron.daily/backup|<br />
#!/bin/bash<br />
rsync -a --delete --quiet /folder/to/backup /location/of/backup}}<br />
<br />
; {{ic|-a}} : indicates that files should be archived, meaning that most of their characteristics are preserved (but '''not''' ACLs, hard links or extended attributes such as capabilities)<br />
; {{ic|--delete}} : means files deleted on the source are to be deleted on the backup as well<br />
<br />
Here, {{ic|/folder/to/backup}} should be changed to what needs to be backed-up ({{ic|/home}}, for example) and {{ic|/location/of/backup}} is where the backup should be saved ({{ic|/media/disk}}, for instance).<br />
<br />
Finally, the script must be executable:<br />
<br />
# chmod +x /etc/cron.daily/backup<br />
<br />
=== Automated backup with SSH ===<br />
<br />
If backing-up to a remote host using [[SSH]], use this script instead:<br />
<br />
{{hc|/etc/cron.daily/backup|<br />
#!/bin/bash<br />
rsync -a --delete --quiet -e ssh /folder/to/backup remoteuser@remotehost:/location/of/backup<br />
}}<br />
<br />
; {{ic|-e ssh}} : tells rsync to use SSH<br />
; {{ic|remoteuser}} : is the user on the host {{ic|remotehost}}<br />
; {{ic|-a}} : groups all these options {{ic|-rlptgoD}} (recursive, links, perms, times, group, owner, devices)<br />
<br />
=== Automated backup with NetworkManager ===<br />
<br />
This script starts a backup when network connection is established.<br />
<br />
First, create a script containing the appropriate command options:<br />
<br />
{{hc|/etc/NetworkManager/dispatcher.d/backup|2=<br />
#!/bin/bash<br />
<br />
if [ x"$2" = "xup" ] ; then<br />
rsync --force --ignore-errors -a --delete --bwlimit=2000 --files-from=files.rsync /folder/to/backup /location/to/backup<br />
fi<br />
}}<br />
<br />
; {{ic|-a}} : group all this options {{ic|-rlptgoD}} recursive, links, perms, times, group, owner, devices<br />
; {{ic|--files-from}} : read the relative path of ''/folder/to/backup'' from this file<br />
; {{ic|--bwlimit}} : limit I/O bandwidth; KBytes per second<br />
<br />
Also, the script must have write permission for owner (root, of course) only (see [[NetworkManager#Network services with NetworkManager dispatcher]] for details).<br />
<br />
=== Automated backup with systemd and inotify ===<br />
<br />
{{Note|<br />
* Due to the limitations of inotify and systemd (see [http://www.quora.com/Linux-Kernel/Inotify-monitoring-of-directories-is-not-recursive-Is-there-any-specific-reason-for-this-design-in-Linux-kernel this question and answer]), recursive filesystem monitoring is not possible. Although you can watch a directory and its contents, it will not recurse into subdirectories and watch the contents of them; you must explicitly specify every directory to watch, even if that directory is a child of an already watched directory.<br />
* This setup is based on a [[systemd/User]] instance.<br />
}}<br />
<br />
Instead of running time interval backups with time based schedules, such as those implemented in [[cron]], it is possible to run a backup every time one of the files you are backing up changes. {{ic|systemd.path}} units use {{ic|inotify}} to monitor the filesystem, and can be used in conjunction with {{ic|systemd.service}} files to start any process (in this case your [[rsync]] backup) based on a filesystem event.<br />
<br />
First, create the {{ic|systemd.path}} file that will monitor the files you are backing up:<br />
<br />
{{hc|~/.config/systemd/user/backup.path|<nowiki><br />
[Unit]<br />
Description=Checks if paths that are currently being backed up have changed<br />
<br />
[Path]<br />
PathChanged=%h/documents<br />
PathChanged=%h/music<br />
<br />
[Install]<br />
WantedBy=default.target<br />
</nowiki>}}<br />
<br />
Then create a {{ic|systemd.service}} file that will be activated when it detects a change. By default a service file of the same name as the path unit (in this case {{ic|backup.path}}) will be activated, except with the {{ic|.service}} extension instead of {{ic|.path}} (in this case {{ic|backup.service}}).<br />
<br />
{{Note|If you need to run multiple rsync commands, use {{ic|1=Type=oneshot}}. This allows you to specify multiple {{ic|1=ExecStart=}} parameters, one for each [[rsync]] command, that will be executed. Alternatively, you can simply write a script to perform all of your backups, just like [[cron]] scripts.}}<br />
<br />
{{hc|~/.config/systemd/user/backup.service|<nowiki><br />
[Unit]<br />
Description=Backs up files<br />
<br />
[Service]<br />
ExecStart=/usr/bin/rsync %h/./documents %h/./music -CERrltm --delete ubuntu:<br />
</nowiki>}}<br />
<br />
Now all you have to do is [[start]]/enable {{ic|backup.path}} like a normal systemd service and it will start monitoring file changes and automatically starting {{ic|backup.service}}.<br />
<br />
=== Differential backup on a week ===<br />
<br />
This is a useful option of rsync, resulting in a full backup (on each run) and keeping a differential backup copy of changed files only in a separate directory for each day of a week.<br />
<br />
First, create a script containing the appropriate command options:<br />
<br />
{{hc|/etc/cron.daily/backup|2=<br />
#!/bin/bash<br />
<br />
DAY=$(date +%A)<br />
<br />
if [ -e /location/to/backup/incr/$DAY ] ; then<br />
rm -fr /location/to/backup/incr/$DAY<br />
fi<br />
<br />
rsync -a --delete --quiet --inplace --backup --backup-dir=/location/to/backup/incr/$DAY /folder/to/backup/ /location/to/backup/full/<br />
}}<br />
<br />
; {{ic|--inplace}} : implies {{ic|--partial}} update destination files in-place<br />
<br />
=== Snapshot backup ===<br />
<br />
The same idea can be used to maintain a tree of snapshots of your files. In other words, a directory with date-ordered copies of the files. The copies are made using hardlinks, which means that only files that did change will occupy space. Generally speaking, this is the idea behind Apple's TimeMachine.<br />
<br />
This basic script is easy to implement and creates quick incremental snapshots using the {{ic|--link-dest}} option to hardlink unchanged files: <br />
<br />
{{hc|/usr/local/bin/snapbackup.sh|<nowiki><br />
#!/bin/bash<br />
<br />
# Basic snapshot-style rsync backup script <br />
<br />
# Config<br />
OPT="-aPh"<br />
LINK="--link-dest=/snapshots/username/last/" <br />
SRC="/home/username/files/"<br />
SNAP="/snapshots/username/"<br />
LAST="/snapshots/username/last"<br />
date=`date "+%Y-%b-%d:_%T"`<br />
<br />
# Run rsync to create snapshot<br />
rsync $OPT $LINK $SRC ${SNAP}$date<br />
<br />
# Remove symlink to previous snapshot<br />
rm -f $LAST<br />
<br />
# Create new symlink to latest snapshot for the next backup to hardlink<br />
ln -s ${SNAP}$date $LAST<br />
</nowiki>}}<br />
<br />
There must be a symlink to a full backup already in existence as a target for {{ic|--link-dest}}. If the most recent snapshot is deleted, the symlink will need to be recreated to point to the most recent snapshot. If {{ic|--link-dest}} does not find a working symlink, rsync will proceed to copy all source files instead of only the changes. <br />
<br />
A more sophisticated version keeps an up-to-date full backup {{ic|$SNAP/latest}} and in case a certain number of files has changed since the last full backup, it creates a snapshot {{ic|$SNAP/$DATETAG}} of the current full-backup utilizing {{ic|cp -al}} to hardlink unchanged files:<br />
<br />
{{hc|/usr/local/bin/rsnapshot.sh|<nowiki><br />
#!/bin/bash<br />
<br />
## my own rsync-based snapshot-style backup procedure<br />
## (cc) marcio rps AT gmail.com<br />
<br />
# config vars<br />
<br />
SRC="/home/username/files/" #dont forget trailing slash!<br />
SNAP="/snapshots/username"<br />
OPTS="-rltgoi --delay-updates --delete --chmod=a-w"<br />
MINCHANGES=20<br />
<br />
# run this process with real low priority<br />
<br />
ionice -c 3 -p $$<br />
renice +12 -p $$<br />
<br />
# sync<br />
<br />
rsync $OPTS $SRC $SNAP/latest >> $SNAP/rsync.log<br />
<br />
# check if enough has changed and if so<br />
# make a hardlinked copy named as the date<br />
<br />
COUNT=$( wc -l $SNAP/rsync.log|cut -d" " -f1 )<br />
if [ $COUNT -gt $MINCHANGES ] ; then<br />
DATETAG=$(date +%Y-%m-%d)<br />
if [ ! -e $SNAP/$DATETAG ] ; then<br />
cp -al $SNAP/latest $SNAP/$DATETAG<br />
chmod u+w $SNAP/$DATETAG<br />
mv $SNAP/rsync.log $SNAP/$DATETAG<br />
chmod u-w $SNAP/$DATETAG<br />
fi<br />
fi<br />
</nowiki>}}<br />
<br />
To make things really, really simple this script can be run from a [[systemd/Timers]] unit.<br />
<br />
=== Full system backup ===<br />
<br />
This section is about using ''rsync'' to transfer a copy of the entire {{ic|/}} tree, excluding a few selected folders. This approach is considered to be better than [[disk cloning]] with {{ic|dd}} since it allows for a different size, partition table and filesystem to be used, and better than copying with {{ic|cp -a}} as well, because it allows greater control over file permissions, attributes, [[Access Control Lists]] and [[extended attributes]].<br />
<br />
''rsync'' will work even while the system is running, but files changed during the transfer may or may not be transferred, which can cause undefined behavior of some programs using the transferred files.<br />
<br />
This approach works well for migrating an existing installation to a new hard drive or [[SSD]].<br />
<br />
Run the following command as root to make sure that rsync can access all system files and preserve the ownership:<br />
<br />
# rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / ''/path/to/backup/folder''<br />
<br />
By using the {{ic|-aAX}} set of options, the files are transferred in archive mode which ensures that symbolic links, devices, permissions, ownerships, modification times, [[ACL]]s, and extended attributes are preserved, assuming that the target [[file system]] supports the feature.<br />
<br />
The {{ic|--exclude}} option causes files that match the given patterns to be excluded. The contents of {{ic|/dev}}, {{ic|/proc}}, {{ic|/sys}}, {{ic|/tmp}}, and {{ic|/run}} are excluded in the above command, because they are populated at boot, although the folders themselves are ''not'' created. {{ic|/lost+found}} is filesystem-specific. The command above depends on brace expansion available in both the [https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html bash] and [http://zsh.sourceforge.net/Doc/Release/Expansion.html#Brace-Expansion zsh] shells. When using a different [[shell]], {{ic|--exclude}} patterns should be repeated manually. Quoting the exclude patterns will avoid expansion by the [[shell]], which is necessary, for example, when backing up over [[SSH]]. Ending the excluded paths with {{ic|*}} ensures that the directories themselves are created if they do not already exist.<br />
<br />
{{Note|<br />
* If you plan on backing up your system somewhere other than {{ic|/mnt}} or {{ic|/media}}, do not forget to add it to the list of exclude patterns to avoid an infinite loop.<br />
* If there are any bind mounts in the system, they should be excluded as well so that the bind mounted contents is copied only once.<br />
* If you use a [[swap file]], make sure to exclude it as well.<br />
* Consider if you want to backup the {{ic|/home/}} folder. If it contains your data it might be considerably larger than the system. Otherwise consider excluding unimportant subdirectories such as {{ic|/home/*/.thumbnails/*}}, {{ic|/home/*/.cache/mozilla/*}}, {{ic|/home/*/.cache/chromium/*}}, and {{ic|/home/*/.local/share/Trash/*}}, depending on software installed on the system. If [[GVFS]] is installed, {{ic|/home/*/.gvfs}} must be excluded to prevent rsync errors.<br />
}}<br />
<br />
You may want to include additional [[rsync]] options, such as the following. See {{man|1|rsync}} for the full list.<br />
<br />
* If you use many hard links, consider adding the {{ic|-H}} option, which is turned off by default due to its memory expense; however, it should be no problem on most modern machines. Many hard links reside under the {{ic|/usr/}} directory.<br />
* You may want to add rsync's {{ic|--delete}} option if you are running this multiple times to the same backup folder. In this case make sure that the source path does not end with {{ic|/*}}, or this option will only have effect on the files inside the subdirectories of the source directory, but it will have no effect on the files residing directly inside the source directory.<br />
* If you use any sparse files, such as virtual disks, [[Docker]] images and similar, you should add the {{ic|-S}} option.<br />
* The {{ic|--numeric-ids}} option will disable mapping of user and group names; instead, numeric group and user IDs will be transfered. This is useful when backing up over [[SSH]] or when using a live system to backup different system disk.<br />
* Choosing {{ic|1=--info=progress2}} option instead of {{ic|-v}} will show the overall progress info and transfer speed instead of the list of files being transferred.<br />
<br />
=== Restore a backup ===<br />
<br />
If you wish to restore a backup, use the same rsync command that was executed but with the source and destination reversed.<br />
<br />
== File system cloning ==<br />
<br />
rsync provides a way to do a copy of all data in a file system while preserving as much information as possible, including the file system metadata. It is a procedure of data cloning on a file system level where source and destination file systems don't need to be of the same type. It can be used for backing up, file system migration or data recovery.<br />
<br />
rsync's ''archive'' mode comes close to being fit for the job, but it doesn't back up the special file system metadata such as access control lists, extended attributes or sparse file properties. For successful cloning at the file system level, some additional options need to be provided:<br />
<br />
rsync -qaHAXS SOURCE_DIR DESTINATION_DIR<br />
<br />
And their meaning is (from the manpage):<br />
<br />
-H, --hard-links preserve hard links<br />
-A, --acls preserve ACLs (implies -p)<br />
-X, --xattrs preserve extended attributes<br />
-S, --sparse handle sparse files efficiently<br />
<br />
Produced copy can be simply reread and checked (for example after a data recovery attempt) at the file system level with {{ic|diff}}'s recursive option:<br />
<br />
diff -r SOURCE_DIR DESTINATION_DIR<br />
<br />
It is possible to do a successful file system migration by using rsync as described in this article and updating the [[fstab]] and [[bootloader]] as described in [[Migrate installation to new hardware]]. This essentially provides a way to convert any root file system to another one.<br />
<br />
== rsync daemon ==<br />
<br />
''rsync'' can be run as daemon on a server listening on port {{ic|873}}.<br />
<br />
Edit the template {{ic|/etc/rsyncd.conf}}, configure a share and [[start]] the {{ic|rsyncd.service}}.<br />
<br />
{{Note|As of {{Pkg|rsync}}-3.1.2-5 the systemd unit {{ic|rsyncd.service}} included in the package adds security feature {{ic|1=ProtectSystem=full}} ({{ic|1=ProtectHome=on}} has been undone in {{Pkg|rsync}}-3.1.2-8) under the {{ic|[Service]}} section. This makes the {{ic|/boot/}}, {{ic|/etc/}} and {{ic|/usr/}} directories read-only. If you need rsyncd write system directories you can [[edit]] the unit and set {{ic|1=ProtectSystem=off}} in the {{ic|[Service]}} section of the overriding snippet.}}<br />
<br />
Usage from client, e.g. list server content:<br />
<br />
$ rsync rsync://''server/share''<br />
<br />
transfer file from client to server: <br />
<br />
$ rsync ''local-file'' rsync://''server/share/''<br />
<br />
Consider iptables to open port {{ic|873}} and user authentication.<br />
<br />
{{Note|All transferred data including user authentication are not encrypted.}}<br />
<br />
== See also ==<br />
<br />
* More usage examples can be searched in the [https://bbs.archlinux.org/viewforum.php?id=27 Community Contributions] and [https://bbs.archlinux.org/viewforum.php?id=33 General Programming] forums<br />
* [http://www.pointsoftware.ch/en/howto-local-and-remote-snapshot-backup-using-rsync-with-hard-links/ Howto – local and remote snapshot backup using rsync with hard links] Includes file deduplication with hard-links, MD5 integrity signature, 'chattr' protection, filter rules, disk quota, retention policy with exponential distribution (backups rotation while saving more recent backups than older)<br />
* [https://stackoverflow.com/questions/5527068/how-do-you-use-an-identity-file-with-rsync Using SSH keys/identity files with rsync]</div>Rpbritton