Difference between revisions of "User:Gen2ly/System backup and reinstall"

From ArchWiki
Jump to navigation Jump to search
m
m (spelling grammar formality)
Line 5: Line 5:
 
# You have limited hard disk space and are not capable of doing a full restore from backup.
 
# You have limited hard disk space and are not capable of doing a full restore from backup.
  
{{Tip|If you are a regular computer user, generally it is good practice to backup your primary drive to a backup hard drive after getting your system installed.  Secondary storage disks can be found realatively inexpensively and are able to store/restore an entire drive of programs, configurations, documents... in a safe and quick way.  If you have a hardware setup like this, look at programs like the [http://clonezilla.org/ Clonezilla CD], or the [http://partedmagic.com/ Parted Magic CD] (includes Clonzilla and other tools) which are both open-source and can image your hard drive to another drive.}}
+
{{Tip|If you are a regular computer user, generally it is good practice to backup your primary drive to a backup hard drive after getting your system installed.  Secondary storage disks can be found relatively inexpensively and are able to store/restore an entire drive of programs, configurations, documents... in a safe and quick way.  If you have a hardware setup like this, look at programs like the [http://clonezilla.org/ Clonezilla CD], or the [http://partedmagic.com/ Parted Magic CD] (includes Clonzilla and other tools) which are both open-source and can image your hard drive to another drive.}}
  
 
This document is designed to show you how to: quickly add commonly-used/necessary configurations in an include file, back them up on a regular basis, and how to restore all your configurations and packages when rebuilding your system.
 
This document is designed to show you how to: quickly add commonly-used/necessary configurations in an include file, back them up on a regular basis, and how to restore all your configurations and packages when rebuilding your system.
Line 15: Line 15:
 
=== Include/Exclude Files  ===
 
=== Include/Exclude Files  ===
  
tar has the ability to read from an include file that has a list of configurations, documents... to be included when compressing. The format is stored line by line with it's full base path. For example:
+
Tar has the ability to read from an "include" file that has a list of files (configurations, documents, etc.) to be included when compressing. The format is stored line by line with it's full base path. For example:
  
 
<pre>
 
<pre>
Line 46: Line 46:
 
</pre>
 
</pre>
  
Both scripts have the ability to read the full path so entering a relative path isn't a problem.  Put them in your script directory and make them executable to add a file anytime you think of it.
+
Both scripts have the ability to read the full path so entering a relative path is not a problem.  Put them in your script directory and make them executable to add a file anytime you think of it.
  
 
<pre>
 
<pre>
Line 69: Line 69:
 
fullpath=$(readlink -f "$@")
 
fullpath=$(readlink -f "$@")
  
# Display usage if full argument isn't given
+
# Display usage if full argument is not given
 
if [[ -z "$@" ]]; then
 
if [[ -z "$@" ]]; then
 
   echo " bca <file/folder/link> -  add files/folders/links to backup-cfgs' include file"
 
   echo " bca <file/folder/link> -  add files/folders/links to backup-cfgs' include file"
Line 77: Line 77:
 
# Test if path exists
 
# Test if path exists
 
if [ ! -e "$fullpath" ]; then
 
if [ ! -e "$fullpath" ]; then
   echo -e " File "$@" doesn't exist."
+
   echo -e " File "$@" does not exist."
 
   exit
 
   exit
 
fi
 
fi
Line 109: Line 109:
 
backdest="/opt/backup"
 
backdest="/opt/backup"
  
# Files contailing information to include and exclude
+
# Files containing information to include and exclude
 
include_file="/home/user/.scripts/backup/include.txt"
 
include_file="/home/user/.scripts/backup/include.txt"
 
exclude_file="/home/user/.scripts/backup/exclude.txt"
 
exclude_file="/home/user/.scripts/backup/exclude.txt"
Line 153: Line 153:
 
=== What not to include in the backup ===
 
=== What not to include in the backup ===
  
There are some files that shouldn't be included in the backup directly.  The reason for this will become apparently later in the document.  For now, if you want these files backed up copy them to a new name:
+
There are some files that should not be included in the backup directly.  The reason for this will become apparently later in the document.  For now, if you want these files backed up copy them to a new name:
  
 
<pre>
 
<pre>
Line 174: Line 174:
 
  bca /opt/backup/pkg-official
 
  bca /opt/backup/pkg-official
  
This will create a list of all pacman repositories enabled (core/extra/community/testing).  To create a list of all local packages (includes packages installed from the AUR):
+
This will create a list of all [[pacman]] repositories enabled (core/extra/community/testing).  To create a list of all local packages (includes packages installed from the [[AUR]]):
  
 
  pacman -Qqmt > /opt/backup/pkg-local
 
  pacman -Qqmt > /opt/backup/pkg-local
 
  bca /opt/backup/pkg-local
 
  bca /opt/backup/pkg-local
  
Alternately, you can use the [http://aur.archlinux.org/packages.php?ID=22747 packup] utility in the AUR.  Packup is a backup and restore from list utility with menu driven interface.  Packup will create a list and save it in the root folder.  If you use packup, create a seperate list for official and local packages for easier debugging later.
+
Alternately, you can use the [http://aur.archlinux.org/packages.php?ID=22747 packup] utility in the AUR.  Packup is a backup and restore from list utility with menu driven interface.  Packup will create a list and save it in the root folder.  If you use packup, create a separate list for official and local packages for easier debugging later.
  
 
==== Saving Package Tarballs ====
 
==== Saving Package Tarballs ====
Line 187: Line 187:
 
  pacman -Sc
 
  pacman -Sc
  
If you use Yaourt to install from the AUR, you might want to consider setting up a cache for it too.  Though Yaourt doesn't have the ablity to read from the cache before an install these packages can later be installed with {{Codeline|pacman -U <package>}}.  In {{Filename|/etc/yaourtrc}} (for example):
+
If you use [[Yaourt]] to install from the AUR, you might want to consider setting up a cache for it too.  Though Yaourt does not have the ablity to read from the cache before an install these packages can later be installed with {{Codeline|pacman -U <package>}}.  In {{Filename|/etc/yaourtrc}} (for example):
  
 
   ExportToLocalRepository /var/cache/yaourt/pkg
 
   ExportToLocalRepository /var/cache/yaourt/pkg
 
+
<!-- what is this ^?-->
 
  mkdir -p /var/cache/yaourt/pkg
 
  mkdir -p /var/cache/yaourt/pkg
 
  chmod 766 /var/cache/yaourt/pkg/
 
  chmod 766 /var/cache/yaourt/pkg/
Line 197: Line 197:
 
=== Storing the Backup ===
 
=== Storing the Backup ===
  
The examples above save the backups to the same partition (provided you don't have a seperate /opt partition).  If you have the hard disk space, it is better to put these on a seperate partition or drive.  Consider using the [http://partedmagic.com/ Parted Magic CD] to create one if you want to do this.  If you don't, you will need to put them on a seperate media before you restore.  If you absolutely have to, you can break up tar files and span them over [http://en.gentoo-wiki.com/wiki/Backup_to_DVD multiple disks].
+
The examples above save the backups to the same partition (provided you do not have a separate /opt partition).  If you have the hard disk space, it is better to put these in a separate partition or drive.  Consider using the [http://partedmagic.com/ Parted Magic CD] to create one if you want to do this.  If you do not you will need to put them in a separate media before you restore.  If you absolutely have to, you can break up tar files and span them over [http://en.gentoo-wiki.com/wiki/Backup_to_DVD multiple disks].
  
 
== Restoring ==
 
== Restoring ==
  
Restoring will require: reinstalling, extracting the backup configurations back on the new system, and checking for conflicts.
+
Restoring will require: reinstalling, extracting the backup configurations back to the new system, and checking for conflicts.
  
 
=== Installing/Restore configs ===
 
=== Installing/Restore configs ===
  
Install Arch Linux as you normally would and then mount your partition with your backup on it:
+
Install Arch Linux as you normally would and then mount your partition with your backup in it:
  
 
  mkdir /backup-part
 
  mkdir /backup-part
Line 214: Line 214:
 
  tar xvf hostname-arch-configs-date-tar.gz -C /mnt
 
  tar xvf hostname-arch-configs-date-tar.gz -C /mnt
  
The installer will have already mounted your install partition on /mnt.  This is where not including those files directly above comes in handy.  Doing this will overwrite any files that match on your backup and the new install.  Check your good mount file (fstab) and bootloader file 9menu.lst) with the new install version and update where necessary, then reboot.
+
The installer will have already mounted your install partition on /mnt.  This is where not including those files directly above comes in handy.  Doing this will overwrite any files that match with your backup and the new install.  Check your good mount file (fstab) and bootloader file (menu.lst) with the new install version and update where necessary, then reboot.
  
 
=== User/Yaourt ===
 
=== User/Yaourt ===
  
Continue your install as you normally would.  When creating a user consider giving the user a unique user id (uid).  This will help prevent conflicts in the future with other users and programs having the same uid (uid's for users generally start at 1000):
+
Continue your install as you normally would.  When creating a user consider giving the user a unique user id (UID).  This will help prevent conflicts in the future with other users and programs having the same UID (UIDs for users generally start at 1000):
  
 
<pre>
 
<pre>
Line 225: Line 225:
 
</pre>
 
</pre>
  
If you have restored a user home directory (/home/user) in your backup configurations, the '''-m''' switch will give a warning but will not effect anything.  The '''-p''' switch will input the password (in this case: abcdefg :) ).  Don't forget to change permissions in your home directory if you change the uid:
+
If you have restored a user home directory (/home/user) in your backup configurations, the '''-m''' switch will give a warning but will not effect anything.  The '''-p''' switch will input the password (in this case: abcdefg ).  Do not forget to change permissions in your home directory if you change the UID:
  
 
  chown -R username:users /home/user
 
  chown -R username:users /home/user
Line 240: Line 240:
 
=== Reinstalling Packages ===
 
=== Reinstalling Packages ===
  
Reinstall packages from the official repositories, the AUR, and locally installed packages seperately to better diagnose problems if they occur.
+
Reinstall packages from the official repositories, the AUR, and locally installed packages separately to better diagnose problems if they occur.
  
 
==== Official ====
 
==== Official ====
Line 274: Line 274:
 
</pre>
 
</pre>
  
The extra checks are to not include any packages already installed.  For example, if there are any problems installing a package from the AUR... you will not have to go through the whole list again and tell Yaourt what to install (i.e. Yaourt doesn't suppport the '''--needed''' switch).
+
The extra checks are to not include any packages already installed.  For example, if there are any problems installing a package from the AUR you will not have to go through the whole list again and tell Yaourt what to install (i.e. Yaourt does not suppport the '''--needed''' switch).
  
 
==== Local ====
 
==== Local ====
Line 282: Line 282:
 
  pacman -U <*--asdeps> <package>
 
  pacman -U <*--asdeps> <package>
  
== Resources ==
+
== External Resources ==
  
 
* [http://bbs.archlinux.org/viewtopic.php?id=76218 BBS - List AUR packages and only AUR packages]
 
* [http://bbs.archlinux.org/viewtopic.php?id=76218 BBS - List AUR packages and only AUR packages]

Revision as of 15:38, 30 October 2009

This is a disjuncted document for the reason that the only time you really need to restore from configurations only is if:

  1. For an unexplained reason a program or programs begin to behave unexpectedly and no help in the forums (or elsewhere) is available is able to fix the problem. By chance reinstalling your programs and other configurations might fix the problem.
  2. You have limited hard disk space and are not capable of doing a full restore from backup.
Tip: If you are a regular computer user, generally it is good practice to backup your primary drive to a backup hard drive after getting your system installed. Secondary storage disks can be found relatively inexpensively and are able to store/restore an entire drive of programs, configurations, documents... in a safe and quick way. If you have a hardware setup like this, look at programs like the Clonezilla CD, or the Parted Magic CD (includes Clonzilla and other tools) which are both open-source and can image your hard drive to another drive.

This document is designed to show you how to: quickly add commonly-used/necessary configurations in an include file, back them up on a regular basis, and how to restore all your configurations and packages when rebuilding your system.

Backup

The tar compression tool with a couple helper scripts can backup your configurations in a few steps.

Include/Exclude Files

Tar has the ability to read from an "include" file that has a list of files (configurations, documents, etc.) to be included when compressing. The format is stored line by line with it's full base path. For example:

/etc/pacman.conf
/etc/rc.conf
/home/user
...

And is invoked like this:

tar --files-from=include.txt --exclude-from=exclude.txt -cvpzf backupname.tar.gz

The name of the include file can be anything you want. Also an exclude file can be used like in the example above. The exclude file is like the include file but additionally has the ability to be able to use regexps and can be commented and have blank lines:

# Trashes not necessary
/home/*/.local/share/Trash/files
/home/*/.Trash

One step to add to (in|ex)clude files

To help add files to the include and exclude files, a couple bash scripts can help to accomplish this. bca (backup-config-add) and bce (backup-cfg-exclude) can be used from the command line:

bca /etc/X11/xorg.conf
cd /home/user
bce .thumbnails/

Both scripts have the ability to read the full path so entering a relative path is not a problem. Put them in your script directory and make them executable to add a file anytime you think of it.

#!/bin/bash
# bce (backup-config-exclude) - add files to be excluded in backup

excludeloc=/home/user/.scripts/backup/exclude.txt

# Add file/folder/link to list
if [[ -z "$@" ]]; then
  echo " bce <file/folder/list> - add exclusions to backing up configurations"
  else
  echo "`readlink -f "$@"`" >> $excludeloc
fi
#!/bin/bash
# bca (backup-cfg-add) - add file/folders to the backup-cfg include file

incfile=/home/user/.scripts/backup/include.txt
fullpath=$(readlink -f "$@")

# Display usage if full argument is not given
if [[ -z "$@" ]]; then
  echo " bca <file/folder/link> -  add files/folders/links to backup-cfgs' include file"
  exit
fi

# Test if path exists
if [ ! -e "$fullpath" ]; then
  echo -e " File "$@" does not exist."
  exit
fi

# Add entry
echo "$fullpath" >> "$incfile"
echo -e "${bldblu}*${txtrst} Added ${undwht}$fullpath${txtrst} in backup include file"

# Sort entries
sort -u "$incfile" -o "$incfile"

Backup Script

A tar backup script can help the process of backing up your configurations and be automated on a regular basis.

This backup script names your backup by hostname and date, specifies you include and exclude file locations, and removes old backups.

#!/bin/bash
# backup-cfg - backup configurations with tar

# Backup name
machine=$HOSTNAME
distro=arch
type=configs
date=`date "+%F"`
backupname=$machine-$distro-$type-$date

# Backup destination
backdest="/opt/backup"

# Files containing information to include and exclude
include_file="/home/user/.scripts/backup/include.txt"
exclude_file="/home/user/.scripts/backup/exclude.txt"

# Verify that the target directory exists.
if [ ! -d $backdest ]; then
  echo " Backup directory does not exist, creating"
  mkdir $backdest
  else
  echo " Backup directory exists."
fi

# Delete backups older than two months
if [[ -n "$(find "$backdest" -mtime +60)" ]]; then
  find "$backdest" -mtime +60 -exec rm {} \;
  echo -e " Backups older than two months deleted"
fi

# Unmount other drives/partitions
#umount -a

# Backup
tar --exclude-from=$exclude_file --files-from=$include_file -cvpzf \
$backdest/$backupname.tar.gz

To do automated backups (for example, every week), put the script in your root crontab (Template:Codeline):

#
# DO NOT EDIT THIS FILE MANUALLY!! USE crontab -e INSTEAD.
#

  50  00  *   *   0       /home/user/.scripts/backup/backup-cfg
# |   |   |   |   |   user
# |   |   |   |   weekday (0-6) (Sunday=0)
# |   |   |   month (1-12)
# |   |   day (0-31)
# |   hour (0-23)
# minute (0-59)

What not to include in the backup

There are some files that should not be included in the backup directly. The reason for this will become apparently later in the document. For now, if you want these files backed up copy them to a new name:

cp /boot/grub/menu.lst /boot/grub/menu.lst-good
cp /etc/fstab /etc/fstab-good
cp /etc/sudoers /etc/sudoers-good

Then add them to your include file.

Packages

A package list can be created that can be used to restore all installed packages. You might also want to consider (if you have the hard disk space available) of backing up the install packages for them.

Creating a Package List

You can create a list of official installed packages with:

pacman -Qqet | grep -v "$(pacman -Qqm)" > /opt/backup/pkg-official
bca /opt/backup/pkg-official

This will create a list of all pacman repositories enabled (core/extra/community/testing). To create a list of all local packages (includes packages installed from the AUR):

pacman -Qqmt > /opt/backup/pkg-local
bca /opt/backup/pkg-local

Alternately, you can use the packup utility in the AUR. Packup is a backup and restore from list utility with menu driven interface. Packup will create a list and save it in the root folder. If you use packup, create a separate list for official and local packages for easier debugging later.

Saving Package Tarballs

Pacman saves all package tarballs in /var/cache/pacman/pkg/. Saving these will increase the re-install speed so consider adding this folder to your include file. You can reduce the size of the cache to the last two installed versions of a package with:

pacman -Sc

If you use Yaourt to install from the AUR, you might want to consider setting up a cache for it too. Though Yaourt does not have the ablity to read from the cache before an install these packages can later be installed with Template:Codeline. In Template:Filename (for example):

  ExportToLocalRepository /var/cache/yaourt/pkg
mkdir -p /var/cache/yaourt/pkg
chmod 766 /var/cache/yaourt/pkg/
bca /var/cache/yaourt/pkg/

Storing the Backup

The examples above save the backups to the same partition (provided you do not have a separate /opt partition). If you have the hard disk space, it is better to put these in a separate partition or drive. Consider using the Parted Magic CD to create one if you want to do this. If you do not you will need to put them in a separate media before you restore. If you absolutely have to, you can break up tar files and span them over multiple disks.

Restoring

Restoring will require: reinstalling, extracting the backup configurations back to the new system, and checking for conflicts.

Installing/Restore configs

Install Arch Linux as you normally would and then mount your partition with your backup in it:

mkdir /backup-part
mount /dev/<your-partition> /backup-part

And extract your backup onto the new install:

tar xvf hostname-arch-configs-date-tar.gz -C /mnt

The installer will have already mounted your install partition on /mnt. This is where not including those files directly above comes in handy. Doing this will overwrite any files that match with your backup and the new install. Check your good mount file (fstab) and bootloader file (menu.lst) with the new install version and update where necessary, then reboot.

User/Yaourt

Continue your install as you normally would. When creating a user consider giving the user a unique user id (UID). This will help prevent conflicts in the future with other users and programs having the same UID (UIDs for users generally start at 1000):

 useradd -m -u 1050 -p '$1$UUiuruAK$5PrDq8BhI2094gMaN8TYE1' -G \
 audio,optical,power,storage,users,video -s /bin/bash user

If you have restored a user home directory (/home/user) in your backup configurations, the -m switch will give a warning but will not effect anything. The -p switch will input the password (in this case: abcdefg ). Do not forget to change permissions in your home directory if you change the UID:

chown -R username:users /home/user

If you previously had any AUR packages installed, you can install Yaourt to restore the packages:

wget http://aur.archlinux.org/packages/yaourt/yaourt.tar.gz
tar xvf yaourt.tar.gz && cd yaourt*
makepkg -s
pacman -U yaourt-*.pkg.tar.gz

Reinstalling Packages

Reinstall packages from the official repositories, the AUR, and locally installed packages separately to better diagnose problems if they occur.

Official

First reinstall packages from the official repositories;

pacman -Sy
pacman -S --needed $(cat /opt/backup/pkg-official)

Pacman will not install packages with file conflicts (i.e. if you have configurations placed that pacman detects an installing package has as well). Take a note of the conflicting packages and either: move the conflicting configuration (if you know what it is), or force pacman to overwrite the conflicting files:

pacman -S -f --needed $(cat /opt/backup/pkg-official)

If you do this, you will need to find out the conflicting file and reinstall it again. Pacman can tell you what files a certain package installs:

pacman -Ql <package>

And you can extract a file or files from an archive with:

tar xvf hostname-arch-configs-date-tar.gz etc/hosts etc/resolve.conf

and place them manually.

The AUR

From the AUR:

cat /opt/backup/pc-emach/cat /opt/backup/pkg-local | grep -v "$(pacman -Qqm)" \
| xargs yaourt -Ss

The extra checks are to not include any packages already installed. For example, if there are any problems installing a package from the AUR you will not have to go through the whole list again and tell Yaourt what to install (i.e. Yaourt does not suppport the --needed switch).

Local

If you created a Yaourt cache or have your own packages saved, you can install them with:

pacman -U <*--asdeps> <package>

External Resources