Difference between revisions of "Talk:Full system backup with rsync"

From ArchWiki
Jump to: navigation, search
(Doesn't Work: Close. See Help:Discussion.)
(-x flag: add2)
 
(78 intermediate revisions by 12 users not shown)
Line 1: Line 1:
== Comments ==
+
== command (and script) handle rsync errors poorly (such as caused by .gvfs) ==
  
Thanks! Neat method for a sync-type backup! The only question I have is what happens if you {{Ic|dd}} the boot sector (first 512 bytes) instead of running GRUB? --[[User:VitaminJ|VitaminJ]] 22:18, 11 November 2009 (EST)
+
This page is very useful but things could be improved. I used the script currently here for a recent backup but rsync failed saying
  
----
+
<code>rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9].</code>
  
I don't know if it works with <code>dd</code>, since the partitioning scheme can be different on the two harddrives.
+
Unfortunately, I could not tell ''which'' files were not transferred because the offending output had long since scrolled past the terminal window and even beyond my terminal's large history buffer. A dry run of rsync (with the <code>-n</code> switch) allowed me to track down the culprit, which turned out to only be <code>.gvfs</code> in my home directory. This is a GNOME thing (''G''nome ''V''irtual ''F''ile ''S''ystem) that uses FUSE which is a separate file system. Turns out that rsync fails on this directory when the command is run as root (but oddly it works if ran by the owner).
  
* I vote for merging this with [[rsync]].
+
I've never used Arch (some day) but Arch users can run GNOME so the <code>.gvfs</code> issue should affect them too. Reading forums and threads about rsync failing over <code>.gvfs</code> suggests that rsync's <code>-x</code> switch (to avoid cross filesystem files) does not fix this and the only way to fix it is with an <code>--exclude</code>. As this appears to be a common problem, an exclusion of <code>/home/*/.gvfs</code> should be added to the list.
* I second the vote.
+
  
[[User:Wooptoo|Wooptoo]] 07:14, 2 December 2009 (EST)
+
The bigger problem is the way the script handles errors. Ideally, it should at least log rsync errors to a separate file so that the details can be easily read (rather than just the last error message which will usually not contain enough information). I hacked together a [http://unix.stackexchange.com/questions/6430/how-to-redirect-stderr-out-to-different-files-and-also-display-in-terminal quick solution] for me based on
--[[User:Keiichi|Keiichi]] 20:59, 2 December 2009 (EST)
+
----
+
Yep, I think you are right, I think that GRUB hardcodes the path to menu.lst in the GRUB boot program, so if you change around disks it won't work.
+
  
As an alternative, an article on using tar instead might be useful because rsync may improperly preserves permissions across filesystems. For example if you backup from ext3 to XFS, and try to restore back to ext3.
+
<code>(cmd | tee stdout.log) 3>&1 1>&2 2>&3 | tee stderr.log</code>
  
:The only issue would be doing it with FAT or NTFS were permissions just aren't possible. Coincidentially, I've done this exact setup using XFS and ext4 wihout anything breaking down. I only had problems between reiserfs and *, but that was due to SELinux context. [[User:Dres|Dres]] 21:22, 18 January 2010 (EST)
+
but there are many possibilities here depending on whether you want stdout, stderr, or both logged. A "perfect solution" must also be careful that asynchronous output doesn't scramble the order the information in the log files.
  
----
+
Anyway, these are just two suggestions for improving this script. Cheers, [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 20:40, 11 January 2015 (UTC)
I think its a good idea to mention dirvish here. --[[User:Moere|Moere]] 03:37, 1 March 2010 (EST)
+
  
----
+
:Removed the script, the one-liner seems to have other issues. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 19:47, 25 February 2015 (UTC)
on the same topic, you can also look at a free script that i wrote to backup the whole disk with rsync here: http://blog.pointsoftware.ch/index.php/howto-local-and-remote-snapshot-backup-using-rsync-with-hard-links/
+
  
It uses file deduplication thanks to hard-links, uses also MD5 integrity signature, 'chattr' protection, filter rules, disk quota, retention policy with exponential distribution (backups rotation while saving more recent backups than older).
+
::Which ones? Where are [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=prev&oldid=362660 they] reported? -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 14:56, 26 February 2015 (UTC)
It was already used in Disaster Recovery Plans for banking companies, in order to replicate datacenters, using only little network bandwidth  and transport encryption tunnel.
+
  
Can be used locally on each servers or via network on a central remote backup server.
+
:::It was related to <s>[https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=362741&oldid=362685]</s> [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=362669&oldid=362661], though I soon saw my mistake and deleted the accuracy template (as mentioned in the section below). As you say, it would be good to know the exact cirumstances. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 15:32, 26 February 2015 (UTC) edit: refer to correct diff
windows server could also be backuped by using a linux box that mount smb shares from them.
+
  
i hope it will be useful to you guys
+
::::I see - at first I thought that somebody else reported that {{ic|/proc}} is not excluded, but now I presume the report was part of the accuracy template... -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 15:57, 26 February 2015 (UTC)
--[[User:Scheuref|Scheuref]] ([[User talk:Scheuref|talk]]) 14:03, 19 September 2012 (UTC)
+
  
-----
+
:::::Well, both (but I hadn't verified in practice until after editing). Anyways, added a note on GVFS [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=363613&oldid=362741], so this can be closed. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 23:01, 3 March 2015 (UTC)
  
Indeed, great article. One small comment, I had to recreate initramfs due to the filesystem being different on the target disk. The previous initramfs didn't have the correct fsck tool for the new disk. Running mkinitcpio -p linux is really simple, but it would be nice to prepare the reader of this case. [[User:Haritak|Haritak]] ([[User talk:Haritak|talk]]) 04:40, 19 April 2013 (UTC)
+
:Hi, [[User:Alad|Alad]] and [[User:Lahwaacz|Lahwaacz]]. Thanks for looking into this. Alad, might removing the script be throwing out the baby with the bath water? It was a valuable if imperfect script. Also, the note just saying {{ic|/home/''user''/.gvfs}} is problematic. Readers may wrongly interpret it to mean to exclude just their home directory's GVFS subdirectory when they must exclude ''all'' GVFS subdirectories. I'll add a parenthetical remarks about that. Using {{ic|/home/*/.gvfs}} is a quick pragmatic solution that should work most of the time. A full solution would require a command that generates a list of all users' home directories to exclude .gvfs from that. [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 23:09, 4 March 2015 (UTC)
  
-----
+
::I've modified the note to use the wildcard pattern.[https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=365970&oldid=365969] Can this be closed? -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 17:22, 18 March 2015 (UTC)
  
== What about hard links and delete options ==
+
:::Yes, we want to maintain as little scripts as possible on the wiki. OP: if you want to expand on the original script, open a github repo or gist and link it here. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 17:26, 18 March 2015 (UTC)
  
Why the -H option is not added?
+
::::The current wording here is still problematic. Users ''must'' exclude the {{ic|.gvfs}} subdirectories or rsync reports errors. The current wording just suggests to skip "unimportant" subdirectories, a suggestion which does not prepare the user for seeing "uninvestigatable error" reports (because scrolled too far) after backing up over night. [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 11:10, 20 March 2015 (UTC)
I've done a quick search in my arch system and there are a lot of hard link in the /usr/share folders.
+
Moreover should the -delet option be added to in order to be sure that the cloned system contains all and only the files which are present in the original one?
+
Thank you,
+
Xwang
+
  
== What happened to this page? ==
+
:::::I agree with you. How about [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=366319&oldid=365974] ? -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 17:29, 20 March 2015 (UTC)
  
Ok, I started this page, so I'm biased. It used to be clearly written, with an easy to understand example on how to use rsync and its exclude format. Now it's a one liner with the excludes mashed together and most of the page is dedicated to setting up your fstab. Why?
+
::::::Better. Cheers, [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 18:34, 20 March 2015 (UTC)
[[User:Wooptoo|Wooptoo]] ([[User talk:Wooptoo|talk]]) 19:47, 6 March 2013 (UTC)
+
  
==<s> Doesn't Work</s> ==
+
== Single command expanded ==
  
The current version here does not work. When attempting to use this as a guide for writing my own short rsync-based backup script, I found that the excludes do not work as given. I put a "--exclude" for each item and left off the leading "/" as rsync takes all excludes to be relative. This needs to be updated with a tested, working version (or possibly rolled back to a working version).
+
About [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=362672&oldid=362661], if you pass the original, shorter {{ic|--exclude}} argument to ''echo'' it appears to expand to the same, longer command. See [https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html]. Not sure what went wrong here... -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 19:41, 25 February 2015 (UTC)
[[User:BlueG|BlueG]] ([[User talk:BlueG|talk]]) 02:22, 28 May 2013 (UTC)
+
 
:Agreed. It just does infinite loops on media. I employed your solution, it works. -- [[User:Oldarney|Oldarney]] ([[User talk:Oldarney|talk]]) 04:47, 2 June 2013 (UTC)
+
:The longer version has portability in mind: not all shells allow brace expansion, including, for instance, older versions of BASH. Plus, although it is longer, it is less "syntaxy". This makes it easier to read and easier to spot the pattern to generalize by adding one's own particular excludes. It may be a good idea to prefer the longer version for these reasons. [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 23:25, 4 March 2015 (UTC)
 +
 
 +
::Thanks for clarifying. I'd probably add a note mentioning different shells (and possibly brace expansion). -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 17:28, 18 March 2015 (UTC)
 +
 
 +
:::There are probably many different Bash-like conventions on the wiki (see e.g. [[Help:Reading]]), so the ArchWiki is not meant to provide "portable" information. I'd just assume that users of alternative shells are aware of the differences, or at least they can find the solution to problems as they arise. -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 17:33, 18 March 2015 (UTC)
 +
 
 +
::::In light of the "Does the given command really work?" thread, the assumption about the users of alternative shells are aware of the problem is questionable. Although this wiki is intended for Arch users, that does not necessitate using the most compact version of a command that works under the ''default'' Arch shell. Both ways (the long and the short) would work under Arch and are in that sense both the "Arch way" of doing things. An Arch user could equally well be using a shell for which the command does not work, therefore this issue is independent of "Archiness". It's really just an issue of whether the mere shortness of the compact version is worth allowing some people to improperly use the command for their backups. One possibility is to use the long version and put in the notes box a bullet about the shortened syntax for some shells. [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 11:38, 30 March 2015 (UTC)
 +
 
 +
:::::I have to say I'm surprised at the number of readers using a different default shell than zsh or bash 4... how about [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=367830&oldid=366319]? (experimental, as I don't feel strongly towards the "short" command) -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 14:41, 30 March 2015 (UTC)
 +
 
 +
::::::I like it, looks more natural than describing the longer version by default and the short version in a note/tip. -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 16:34, 30 March 2015 (UTC)
 +
 
 +
::::::It's acceptable. One other comment: this page is ranked highly by Google for common search terms related to backup and rsync. This suggests that the page will get significant traffic from people looking to learn how to backup on linux in general, not just limited to Arch, due to the article title. In any case, I think it's an important point when knowing the readership of the article. [[User:Jason Quinn|Jason Quinn]] ([[User talk:Jason Quinn|talk]]) 19:56, 30 March 2015 (UTC)
 +
 
 +
== fstab example ==
 +
 
 +
Re [https://wiki.archlinux.org/index.php?title=Full_system_backup_with_rsync&diff=453644&oldid=453643], it probably makes sense to have a note in place which reminds why (multiple) existing fstab entries should be commented out. That, or add something to the ¨With a single command" section concerning restoring backups to multiple partitions. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 18:54, 13 October 2016 (UTC)
 +
 
 +
== -x flag ==
 +
 
 +
An aside in [[#command (and script) handle rsync errors poorly (such as caused by .gvfs)]], but point of its own, {{ic|rsync -x}} would replace most of the excludes? When you look at ''findmnt'' output, it should behave as expected [https://paste.xinu.at/yMxIfS/]
 +
 
 +
I suppose it makes sense if you have multiple partitions with user data, such as {{ic|/home}}. Cf. [[Talk:Full_system_backup_with_rsync#fstab_example]]. We could clarify on this regard; more generally, give more points for people to construct their own rsync command, rather than build an article around a single one. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 20:51, 16 October 2016 (UTC)

Latest revision as of 20:57, 16 October 2016

command (and script) handle rsync errors poorly (such as caused by .gvfs)

This page is very useful but things could be improved. I used the script currently here for a recent backup but rsync failed saying

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9].

Unfortunately, I could not tell which files were not transferred because the offending output had long since scrolled past the terminal window and even beyond my terminal's large history buffer. A dry run of rsync (with the -n switch) allowed me to track down the culprit, which turned out to only be .gvfs in my home directory. This is a GNOME thing (Gnome Virtual File System) that uses FUSE which is a separate file system. Turns out that rsync fails on this directory when the command is run as root (but oddly it works if ran by the owner).

I've never used Arch (some day) but Arch users can run GNOME so the .gvfs issue should affect them too. Reading forums and threads about rsync failing over .gvfs suggests that rsync's -x switch (to avoid cross filesystem files) does not fix this and the only way to fix it is with an --exclude. As this appears to be a common problem, an exclusion of /home/*/.gvfs should be added to the list.

The bigger problem is the way the script handles errors. Ideally, it should at least log rsync errors to a separate file so that the details can be easily read (rather than just the last error message which will usually not contain enough information). I hacked together a quick solution for me based on

(cmd | tee stdout.log) 3>&1 1>&2 2>&3 | tee stderr.log

but there are many possibilities here depending on whether you want stdout, stderr, or both logged. A "perfect solution" must also be careful that asynchronous output doesn't scramble the order the information in the log files.

Anyway, these are just two suggestions for improving this script. Cheers, Jason Quinn (talk) 20:40, 11 January 2015 (UTC)

Removed the script, the one-liner seems to have other issues. -- Alad (talk) 19:47, 25 February 2015 (UTC)
Which ones? Where are they reported? -- Lahwaacz (talk) 14:56, 26 February 2015 (UTC)
It was related to [1] [2], though I soon saw my mistake and deleted the accuracy template (as mentioned in the section below). As you say, it would be good to know the exact cirumstances. -- Alad (talk) 15:32, 26 February 2015 (UTC) edit: refer to correct diff
I see - at first I thought that somebody else reported that /proc is not excluded, but now I presume the report was part of the accuracy template... -- Lahwaacz (talk) 15:57, 26 February 2015 (UTC)
Well, both (but I hadn't verified in practice until after editing). Anyways, added a note on GVFS [3], so this can be closed. -- Alad (talk) 23:01, 3 March 2015 (UTC)
Hi, Alad and Lahwaacz. Thanks for looking into this. Alad, might removing the script be throwing out the baby with the bath water? It was a valuable if imperfect script. Also, the note just saying /home/user/.gvfs is problematic. Readers may wrongly interpret it to mean to exclude just their home directory's GVFS subdirectory when they must exclude all GVFS subdirectories. I'll add a parenthetical remarks about that. Using /home/*/.gvfs is a quick pragmatic solution that should work most of the time. A full solution would require a command that generates a list of all users' home directories to exclude .gvfs from that. Jason Quinn (talk) 23:09, 4 March 2015 (UTC)
I've modified the note to use the wildcard pattern.[4] Can this be closed? -- Lahwaacz (talk) 17:22, 18 March 2015 (UTC)
Yes, we want to maintain as little scripts as possible on the wiki. OP: if you want to expand on the original script, open a github repo or gist and link it here. -- Alad (talk) 17:26, 18 March 2015 (UTC)
The current wording here is still problematic. Users must exclude the .gvfs subdirectories or rsync reports errors. The current wording just suggests to skip "unimportant" subdirectories, a suggestion which does not prepare the user for seeing "uninvestigatable error" reports (because scrolled too far) after backing up over night. Jason Quinn (talk) 11:10, 20 March 2015 (UTC)
I agree with you. How about [5] ? -- Alad (talk) 17:29, 20 March 2015 (UTC)
Better. Cheers, Jason Quinn (talk) 18:34, 20 March 2015 (UTC)

Single command expanded

About [6], if you pass the original, shorter --exclude argument to echo it appears to expand to the same, longer command. See [7]. Not sure what went wrong here... -- Alad (talk) 19:41, 25 February 2015 (UTC)

The longer version has portability in mind: not all shells allow brace expansion, including, for instance, older versions of BASH. Plus, although it is longer, it is less "syntaxy". This makes it easier to read and easier to spot the pattern to generalize by adding one's own particular excludes. It may be a good idea to prefer the longer version for these reasons. Jason Quinn (talk) 23:25, 4 March 2015 (UTC)
Thanks for clarifying. I'd probably add a note mentioning different shells (and possibly brace expansion). -- Alad (talk) 17:28, 18 March 2015 (UTC)
There are probably many different Bash-like conventions on the wiki (see e.g. Help:Reading), so the ArchWiki is not meant to provide "portable" information. I'd just assume that users of alternative shells are aware of the differences, or at least they can find the solution to problems as they arise. -- Lahwaacz (talk) 17:33, 18 March 2015 (UTC)
In light of the "Does the given command really work?" thread, the assumption about the users of alternative shells are aware of the problem is questionable. Although this wiki is intended for Arch users, that does not necessitate using the most compact version of a command that works under the default Arch shell. Both ways (the long and the short) would work under Arch and are in that sense both the "Arch way" of doing things. An Arch user could equally well be using a shell for which the command does not work, therefore this issue is independent of "Archiness". It's really just an issue of whether the mere shortness of the compact version is worth allowing some people to improperly use the command for their backups. One possibility is to use the long version and put in the notes box a bullet about the shortened syntax for some shells. Jason Quinn (talk) 11:38, 30 March 2015 (UTC)
I have to say I'm surprised at the number of readers using a different default shell than zsh or bash 4... how about [8]? (experimental, as I don't feel strongly towards the "short" command) -- Alad (talk) 14:41, 30 March 2015 (UTC)
I like it, looks more natural than describing the longer version by default and the short version in a note/tip. -- Lahwaacz (talk) 16:34, 30 March 2015 (UTC)
It's acceptable. One other comment: this page is ranked highly by Google for common search terms related to backup and rsync. This suggests that the page will get significant traffic from people looking to learn how to backup on linux in general, not just limited to Arch, due to the article title. In any case, I think it's an important point when knowing the readership of the article. Jason Quinn (talk) 19:56, 30 March 2015 (UTC)

fstab example

Re [9], it probably makes sense to have a note in place which reminds why (multiple) existing fstab entries should be commented out. That, or add something to the ¨With a single command" section concerning restoring backups to multiple partitions. -- Alad (talk) 18:54, 13 October 2016 (UTC)

-x flag

An aside in #command (and script) handle rsync errors poorly (such as caused by .gvfs), but point of its own, rsync -x would replace most of the excludes? When you look at findmnt output, it should behave as expected [10]

I suppose it makes sense if you have multiple partitions with user data, such as /home. Cf. Talk:Full_system_backup_with_rsync#fstab_example. We could clarify on this regard; more generally, give more points for people to construct their own rsync command, rather than build an article around a single one. -- Alad (talk) 20:51, 16 October 2016 (UTC)