Difference between revisions of "Synchronization and backup programs"

From ArchWiki
Jump to: navigation, search
m (Not rsync-based)
m (codeline -> ic)
Line 5: Line 5:
== Intro ==
== Intro ==
This wiki page contains information about various backup programs. It's a good idea to ''have'' regular backups of important data, most notably configuration files ({{Codeline|/etc/*}}) and local pacman database (usually {{Codeline|/var/lib/pacman/local/*}}).
This wiki page contains information about various backup programs. It's a good idea to ''have'' regular backups of important data, most notably configuration files ({{Ic|/etc/*}}) and local pacman database (usually {{Ic|/var/lib/pacman/local/*}}).
Few words to end the introduction: before you start trying programs out, try to think about your needs, e.g. consider the following questions:
Few words to end the introduction: before you start trying programs out, try to think about your needs, e.g. consider the following questions:

Revision as of 16:46, 11 December 2011

This template has only maintenance purposes. For linking to local translations please use interlanguage links, see Help:i18n#Interlanguage links.

Local languages: Català – Dansk – English – Español – Esperanto – Hrvatski – Indonesia – Italiano – Lietuviškai – Magyar – Nederlands – Norsk Bokmål – Polski – Português – Slovenský – Česky – Ελληνικά – Български – Русский – Српски – Українська – עברית – العربية – ไทย – 日本語 – 正體中文 – 简体中文 – 한국어

External languages (all articles in these languages should be moved to the external wiki): Deutsch – Français – Română – Suomi – Svenska – Tiếng Việt – Türkçe – فارسی


This wiki page contains information about various backup programs. It's a good idea to have regular backups of important data, most notably configuration files (/etc/*) and local pacman database (usually /var/lib/pacman/local/*).

Few words to end the introduction: before you start trying programs out, try to think about your needs, e.g. consider the following questions:

  • What backup medium do I have available?
    • cd / dvd
    • remote server (With what access? Ssh? Can I install some software on it (necessary for e.g. rsync-based solutions)?)
    • external harddrive
  • How often do I plan to backup?
    • daily?
    • weekly?
    • less often?
  • What goodies do I expect from the backup solution?
    • compression? (what algorithms?)
    • encryption? (gpg or something more straightforward?)
  • Most importantly: how do I plan to restore backups if needed?

All right, enough with this, let's see some options!

Incremental backups

The point with these is that they remember what has been backed up during the last run, and back up only what has changed. Great if you back up often.

Rsync-type backups

The main characteristic of this type of backups is that they maintain a copy of the directory you want to keep a backup of, in a traditional "mirror" fashion.

Certain rsync-type packages also do snapshot backups by storing files which describe how the contents of files and folders changed from the last backup (the so-called 'diffs'). Hence, they are inherently incremental, but usually they do not have compression/encryption. On the other hand, a working copy of everything is immediately available, no decompression/decryption needed. Finally the way it works makes it hard to burn backups to cd/dvd..

  • rsync (in extra repo)
    • rsync almost always makes a mirror of the source
    • Impossible to restore a full backup before the most recent backup (but you can use --backup to keep old versions of the files)
    • Standard install on all distros
    • Can run over SSH (port 22) or native rsync protocol (port 873)
    • Win32 version available
  • rdiff-backup (in community repo)
    • Stores most recent backup as regular files
    • To revert to older versions, you apply the diff files to recreate the older versions
    • It is granularly incremental (delta backup), it only stores changes to a file; will not create a new copy of a file upon change
    • Required python and librsync
    • Win32 version available
  • rsnapshot (in community repo)
    • Does not store diffs, instead it copies entire files if they have changed
    • Creates hard links between a series of backed-up trees (snapshots)
    • It is differential in that the size of the backup is only the original backup size plus the size of all files that have changed since the last backup.
    • Destination filesystem must support hard links
    • Requires perl
    • Win32 version available
  • SafeKeep (in AUR)
    • Enhancement to rdiff-backup
    • Integrates with Linux LVM and databases to create consistent backups
    • Bandwidth throttling
  • Link-Backup (AUR may be patched with additional features) is similar to rsync based scripts, but does not use rsync
    • Creates hard links between a series of backed-up trees (snapshots)
    • Intelligently handles renames, moves, and duplicate files without additional storage or transfer
    • dstdir/.catalog is a catalog of all unique file instances; backup trees hard-link to the catalog
    • Transfer occurs over standard I/O locally or remotely between a client and server instance of this script
    • It copies itself to the server; it does not need to be installed on the server
    • Requires SSH for remote backups
    • It resumes stopped backups; it can even be told to run for n minutes
  • Back In Time (in AUR)
    • Creates hard links between a series of backed-up trees (snapshots)
    • Inspired by FlyBack and TimeVault
    • Really is just a front-end to rsync, diff, cp
    • A new snapshot is created only if something changed since the last snapshot
  • FlyBack (in AUR)
    • A clone of Apple's Mac OS X Time Machine software
  • Areca Backup (in AUR)
    • Written in Java
    • Primarily archive-based (ZIP), but will do file-based backup as well
    • Delta backup supported (stores only changes)
  • TimeVault
    • Creates hard links between a series of backed-up trees (snapshots)
    • Imitates Windows Volume Shadow Copy feature in that it integrates with Nautilus to provide a "Previous Versions" tab in the Properties dialog.
  • Luckybackup (in AUR)
    • Luckybackup it's an easy program to backup and sync your files.
    • It is written in Qt and C++
    • It has Sync,Backup (with include and exclude options),Restore operations,Remote connection backup,Scheduled backups,Command line mode and more abilities.
  • syncbackup (in AUR)
    • Front-end for rsync provides a fast and extraordinarily copying tool. It offers the most common options that control its behavior and permit very flexible specification of the set of files to be copied.

Not rsync-based

They tend to create (big) archive files (like tar.bz2), and (of course) keep track of what's been archived. Now creating tar.bz2 or tar.gz archives has the advantage that you can extract the backups with just tar/bzip2/gzip, so you do not need to have the backup program around.

  • backup-manager (in AUR); homepage Backup Manager is a command line backup tool for GNU/Linux, designed to help you make daily archives of your file system. Written in bash and perl, it can make archives in lots of open formats (tar, gzip, bzip2, lzma, dar, zip) and provides lots of interesting features (such as network exports or CD/DVD automated-burning). The program is designed to be as easy to use as possible and is popular with desktop users and sysadmins. The whole backup process is defined in one full-documented configuration file which needs no more than 5 minutes to tune for your needs. It just works. Archives are kept for a given number of days and the upload system can use ftp or scp to transfer the generated archives to a list of remote hosts. Gettext is used for internationalization.
    • 1 configuration file, 5 minutes setup
    • Manually invoke backup process or run daily unattended via CRON
    • Backup files, MySQL databases and Subversion repositories
    • Specify multiple targets to backup at once (/etc, /home, etc…)
    • Ability to exclude files from backup
    • Automatically purge old backups
    • Full backup only or Full + Incremental backup
    • Backup to an attached disk, LAN or Internet
    • Burns backup to CD/DVD with MD5 checksum verification
    • Archives in lots of open formats: tar, gzip, bzip2, lzma, dar, zip
    • Slice archives to 2 GB if using dar archives format
    • Backup over SSH
    • Encrypts archives
    • Offsite remote upload of archives via FTP, SSH, RSYNC or Amazon S3
    • Can run with different configuration files concurrently
    • Easy external hooks
    • Simply uncompressed the open format backup archives with any command line or GUI tool
  • arch-backup (in community repo); website); trivial backup scripts with simple configuration:
    • compression method can be configured
    • possible to specify more directories to backup
  • hdup (in extra repo; website; it's no longer developed, the author develops rdup now (below); but it's still a decent one):
    • creates tar.gz or tar.bz2 archives
    • supports gpg encryption
    • supports pushing over ssh
    • possible to specify more directories to backup
  • rdup (in AUR) successor to hdup: the program *just determines* which files have changed since the last backup. It's completely up to you what do you want to do with that list. Some helper scripts are supplied, and with them it supports:
    • creating tar.gz archives or rsync-type copy
    • encryption (gpg and usual strong (eg. blowfish)), also applies for rsync-type copy
    • compression (also for rsync-type copy)
  • duplicity is a simple command-line utility which allows encrypted compressed incremental backup to nearly any storage. For even simpler usage there is duply frontend.
    • supports gpg encryption and signing
    • supports gzip compression
    • supports full or incremental backups, incremental backup stores only difference between new and old file
    • supports pushing over ftp, ssh/scp, rsync, WebDAV, WebDAVs, HSi and Amazon S3 or local filesystem
  • dar (in community repo):
    • it uses its own format for archives (so you need to have it around when you want to restore)
    • supports splitting backups into more files by size
    • makefile-type config files, some custom scripts are available along with it
    • supports basic encryption (not gpg; but also strong, but you need to supply a password every time)
    • some gui tools for inspecting backups are also available (kdar, in AUR, but current dar needs beta version)
    • a script suitable for running from cron is sarab (in AUR): supports pretty much any backup scheme (Towers of Hanoi, Grandfather-Father-Son, etc..)
  • backerupper (in AUR) Backerupper is a simple program for backing up selected directories over a local network. Its main intended purpose is backing up a user's personal data.
    • GUI based
    • creating tar.gz archives
    • possible to define : backup frequency, backup time, Max copies
  • Manent (in AUR) is an algorithmically strong backup and archival program. It's Python based and has the following features:
    • Efficient backup to anything that looks like a storage
    • Works well over a slow and unreliable network
    • Offers online access to the contents of the backup
    • Backed up storage is completely encrypted
    • Several computers can use the same storage for backup, automatically sharing data
    • Not reliant on timestamps of the remote system to detect changes
    • Cross-platform support for Unicode file names
  • Deja-Dup (in AUR) Déjà Dup is a simple backup program. It hides the complexity of doing backups the 'right way' (encrypted, off-site, and regular) and uses duplicity as the backend.

Cloud backups

  • Crashplan (in AUR)
    • Unlimited online space for very reasonable pricing.
    • Automatic and incremental backups to multiple destinations.
    • Intuitive GUI.
    • Offers encryption and de-duplication.
    • Software is generally free.
  • Dropbox (in AUR with Gnome support, and also AUR without Gnome dependencies).
    • A daemon monitors a specified directory, and uploads incremental changes to Dropbox.com.
    • Changes automatically show up on your other computers.
    • Includes file sharing and a public directory.
    • You can recover deleted files.
    • Community written add-ons.
    • Free accounts have 2GB storage.
  • Jungle Disk (in AUR)
    • Automatic backups to Amazon's servers.
    • Not free, but very low prices.
  • Tarsnap (in community repo)
    • Compressed encrypted backups to Amazon S3 Servers
    • Automate via cron
    • Incremental backups
    • Backup any files or directories
    • Command line only client
    • Very very low prices
    • Pay only for usage; bandwidth and storage
  • Wuala (in AUR, also possible to run as daemon)
    • Not Open Source, but free, also paid version available.
    • Free account holds 2GB, (as of 2011.10.03.) disk space trading has been discontinued.
    • Includes file sharing and a public directory.
    • Incremental backup and sync are both supported.
    • Very convenient GUI.
    • Social networking features.
    • All files in the cloud are first encrypted locally.

Not incremental backups

  • Q7Z (in AUR) is a P7Zip GUI for Linux, which attempts to simplify data compression and backup. It can create the following archive types: 7z, BZip2, Zip, GZip, Tar. Use Q7Z if you want to:
    • Update existing archives quickly
    • Backup multiple folders to a storage location
    • Create or extract protected archives
    • Lessen effort by using archiving profiles and lists
  • "Just copy everything into one big archive, but support writing to cd/dvd"-type: backup-manager (in AUR)
  • Partclone -- back up and restore only the used blocks of a partition
  • Redo Backup and Recovery - a nice liveCD that uses xPUD for the GUI and partclone for the backend.
  • Template:Package AUR -- simple bash script (was originally a MySQL backup script) that creates a rolling 7 days, rolling 4 weeks and static monthly backups in tar format. Good for servers without a GUI.
  • Clonezilla
    • Boots from live CD, USB flash drive, or PXE server
    • Uses Partimage, ntfsclone, partclone, and dd,
    • Compatability with many file systems (ext2, ext3, ext4, reiserfs, xfs, jfs of GNU/Linux, FAT, NTFS, and HFS+)
    • Multicasting server to restore to many machines at once
    • If file system is supported , only used blocks in harddisk are saved and restored. For unsupported file system, sector-to-sector copy is done
  • Partimage
  • Fsarchiver (in extra repo)
    • Support for basic file attributes (permissions, owner, ...)
    • Support for multiple file-systems per archive
    • Support for extended attributes (they are used by SELinux)
    • Support the basic file-system attributes (label, uuid, block-size) for all linux file-systems
    • Support for ntfs filesystems (ability to create flexible clones of windows partitions)
    • Checksumming of everything which is written in the archive (headers, data blocks, whole files)
    • Ability to restore an archive which is corrupt (it will just skip the current file)
    • Multi-threaded lzo, gzip, bzip2, lzma compression: if you have a dual-core / quad-core it will use all the power of your cpu
    • Lzma compression (slow but very efficient algorithm) to make your archive smaller.
    • Support for splitting large archives into several files with a fixed maximum size
    • Encryption of the archive using a password. Based on blowfish from libcrypto from openssl.
    • Support backup of a mounted root filesystem (-A option)
  • Mondo Rescue
    • Image-based backups, supporting Linux/Windows, I do not think it cares much about filesystems. Backs up MBR too, along with partition layout.
    • Compression rate adjustable
    • Can backup live systems (without having to halt it, unlike most, e.g. Clonezilla).
    • Can split image over as many as you want (just set the size for a CD - 700 MB to backup to CD, etc.).
    • Supports booting to a "live cd", in a sense, to perform a full restore.
    • Can backup/restore over NFS, cd's, tape drives and what not.
    • Can verify backups
    • Neat-o ncurses interface. Plus a free progressbar!
    • Customizable to all kinds of ends
    • Can get a bit confusing/time consuming to get up and running, and verify everything works.

Versioning systems

These are traditionally used for keeping track of software development; but if you want to have a simple way to manage your config files in one directory, it might be a good solution.

  • mercurial or git (both in extra repo)
  • gibak: a backup system based on git. it also supports binary diffs (for binaries, e-books, pictures, multimedia files, etc). on the homepage there is a short usage advice. it is meant to backup only the $HOME directory. one could also backup other directories (like /etc) by changing the $HOME variable to point to that directory (though i do not really recommend this). gibak is handy for people who are familiar with git. it uses .gitignore to filter files and one can use the git commands to restore files, browse through logs, diffs, etc. if one needs a gui, it is also possible to use gitk or qgit to browse through commits or do whatever these interfaces support. get it from AUR: http://aur.archlinux.org/packages.php?ID=18318.

External Resources