Talk:Full system backup with rsync

From ArchWiki
Jump to: navigation, search

command (and script) handle rsync errors poorly (such as caused by .gvfs)

This page is very useful but things could be improved. I used the script currently here for a recent backup but rsync failed saying

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9].

Unfortunately, I could not tell which files were not transferred because the offending output had long since scrolled past the terminal window and even beyond my terminal's large history buffer. A dry run of rsync (with the -n switch) allowed me to track down the culprit, which turned out to only be .gvfs in my home directory. This is a GNOME thing (Gnome Virtual File System) that uses FUSE which is a separate file system. Turns out that rsync fails on this directory when the command is run as root (but oddly it works if ran by the owner).

I've never used Arch (some day) but Arch users can run GNOME so the .gvfs issue should affect them too. Reading forums and threads about rsync failing over .gvfs suggests that rsync's -x switch (to avoid cross filesystem files) does not fix this and the only way to fix it is with an --exclude. As this appears to be a common problem, an exclusion of /home/*/.gvfs should be added to the list.

The bigger problem is the way the script handles errors. Ideally, it should at least log rsync errors to a separate file so that the details can be easily read (rather than just the last error message which will usually not contain enough information). I hacked together a quick solution for me based on

(cmd | tee stdout.log) 3>&1 1>&2 2>&3 | tee stderr.log

but there are many possibilities here depending on whether you want stdout, stderr, or both logged. A "perfect solution" must also be careful that asynchronous output doesn't scramble the order the information in the log files.

Anyway, these are just two suggestions for improving this script. Cheers, Jason Quinn (talk) 20:40, 11 January 2015 (UTC)

Removed the script, the one-liner seems to have other issues. -- Alad (talk) 19:47, 25 February 2015 (UTC)
Which ones? Where are they reported? -- Lahwaacz (talk) 14:56, 26 February 2015 (UTC)
It was related to [1] [2], though I soon saw my mistake and deleted the accuracy template (as mentioned in the section below). As you say, it would be good to know the exact cirumstances. -- Alad (talk) 15:32, 26 February 2015 (UTC) edit: refer to correct diff
I see - at first I thought that somebody else reported that /proc is not excluded, but now I presume the report was part of the accuracy template... -- Lahwaacz (talk) 15:57, 26 February 2015 (UTC)
Well, both (but I hadn't verified in practice until after editing). Anyways, added a note on GVFS [3], so this can be closed. -- Alad (talk) 23:01, 3 March 2015 (UTC)
Hi, Alad and Lahwaacz. Thanks for looking into this. Alad, might removing the script be throwing out the baby with the bath water? It was a valuable if imperfect script. Also, the note just saying /home/user/.gvfs is problematic. Readers may wrongly interpret it to mean to exclude just their home directory's GVFS subdirectory when they must exclude all GVFS subdirectories. I'll add a parenthetical remarks about that. Using /home/*/.gvfs is a quick pragmatic solution that should work most of the time. A full solution would require a command that generates a list of all users' home directories to exclude .gvfs from that. Jason Quinn (talk) 23:09, 4 March 2015 (UTC)
I've modified the note to use the wildcard pattern.[4] Can this be closed? -- Lahwaacz (talk) 17:22, 18 March 2015 (UTC)
Yes, we want to maintain as little scripts as possible on the wiki. OP: if you want to expand on the original script, open a github repo or gist and link it here. -- Alad (talk) 17:26, 18 March 2015 (UTC)
The current wording here is still problematic. Users must exclude the .gvfs subdirectories or rsync reports errors. The current wording just suggests to skip "unimportant" subdirectories, a suggestion which does not prepare the user for seeing "uninvestigatable error" reports (because scrolled too far) after backing up over night. Jason Quinn (talk) 11:10, 20 March 2015 (UTC)
I agree with you. How about [5] ? -- Alad (talk) 17:29, 20 March 2015 (UTC)
Better. Cheers, Jason Quinn (talk) 18:34, 20 March 2015 (UTC)

Single command expanded

About [6], if you pass the original, shorter --exclude argument to echo it appears to expand to the same, longer command. See [7]. Not sure what went wrong here... -- Alad (talk) 19:41, 25 February 2015 (UTC)

The longer version has portability in mind: not all shells allow brace expansion, including, for instance, older versions of BASH. Plus, although it is longer, it is less "syntaxy". This makes it easier to read and easier to spot the pattern to generalize by adding one's own particular excludes. It may be a good idea to prefer the longer version for these reasons. Jason Quinn (talk) 23:25, 4 March 2015 (UTC)
Thanks for clarifying. I'd probably add a note mentioning different shells (and possibly brace expansion). -- Alad (talk) 17:28, 18 March 2015 (UTC)
There are probably many different Bash-like conventions on the wiki (see e.g. Help:Reading), so the ArchWiki is not meant to provide "portable" information. I'd just assume that users of alternative shells are aware of the differences, or at least they can find the solution to problems as they arise. -- Lahwaacz (talk) 17:33, 18 March 2015 (UTC)
In light of the "Does the given command really work?" thread, the assumption about the users of alternative shells are aware of the problem is questionable. Although this wiki is intended for Arch users, that does not necessitate using the most compact version of a command that works under the default Arch shell. Both ways (the long and the short) would work under Arch and are in that sense both the "Arch way" of doing things. An Arch user could equally well be using a shell for which the command does not work, therefore this issue is independent of "Archiness". It's really just an issue of whether the mere shortness of the compact version is worth allowing some people to improperly use the command for their backups. One possibility is to use the long version and put in the notes box a bullet about the shortened syntax for some shells. Jason Quinn (talk) 11:38, 30 March 2015 (UTC)
I have to say I'm surprised at the number of readers using a different default shell than zsh or bash 4... how about [8]? (experimental, as I don't feel strongly towards the "short" command) -- Alad (talk) 14:41, 30 March 2015 (UTC)
I like it, looks more natural than describing the longer version by default and the short version in a note/tip. -- Lahwaacz (talk) 16:34, 30 March 2015 (UTC)
It's acceptable. One other comment: this page is ranked highly by Google for common search terms related to backup and rsync. This suggests that the page will get significant traffic from people looking to learn how to backup on linux in general, not just limited to Arch, due to the article title. In any case, I think it's an important point when knowing the readership of the article. Jason Quinn (talk) 19:56, 30 March 2015 (UTC)

fstab example

Re [9], it probably makes sense to have a note in place which reminds why (multiple) existing fstab entries should be commented out. That, or add something to the ¨With a single command" section concerning restoring backups to multiple partitions. -- Alad (talk) 18:54, 13 October 2016 (UTC)

-x flag

An aside in #command (and script) handle rsync errors poorly (such as caused by .gvfs), but point of its own, rsync -x would replace most of the excludes? When you look at findmnt output, it should behave as expected [10]

I suppose it makes sense if you have multiple partitions with user data, such as /home. Cf. Talk:Full_system_backup_with_rsync#fstab_example. We could clarify on this regard; more generally, give more points for people to construct their own rsync command, rather than build an article around a single one. -- Alad (talk) 20:51, 16 October 2016 (UTC)