Difference between revisions of "User:Gen2ly/System backup and reinstall"

Revision as of 20:44, 22 March 2010

This article is intended to show you how to backup your configurations and your package list, then to do a full system restore from those backups and finally restore your configurations..

Motivation

The need for this documentation is uncommon in the sense that the need to restore from configurations only is only really necessary for the following reasons:

1. You would like to change your system architecture (e.g. 32bit to 64bit).
2. If a program or programs begin to behave unexpectedly and no help in the forums (or elsewhere) is available is able to fix the problem. By chance reinstalling your programs and other configurations might fix the problem.
3. You have limited hard disk space and are not capable of doing a full restore from backup.

Backup

Using Template:Codeline and a couple helper scripts can make recording common configurations in just a couple steps.

Include and Exclude Files

Tar has the ability to read from both an include and an exclude file. This means that you can tell tar everything you would like to include in the backup and exclude just by using two files. The format used is one line per file or directory that indicates the full path. For example:

/etc/pacman.conf
/etc/rc.conf
/home/user
...

And is invoked like this:

tar --files-from=include.txt --exclude-from=exclude.txt -cvpzf backupname.tar.gz

The name of the files can be anything you want. The exclude file is like the include file but additionally has the ability to be able to use regexps, as well as being able to be commented and have blank lines.

Add to Include and Exclude Files with a Script

To help add file names, folders... to the include and exclude files a couple bash scripts can be used. The scripts are named bci and bce (backup-config-include and backup-config-exclude) and are invoke like this:

bca /etc/X11/xorg.conf
cd /home/user
bce .thumbnails/

Both scripts detect the full path so writing a relative or partial path are acceptable. The program Template:Package AUR will be required. Put them in your script directory and make them executable to add a file anytime you think of it. Be sure to use your username and edit the location where you put your exclude and include file.

#!/bin/bash
# add files to be excluded in backup

excludeloc=/home/<user>/.scripts/backup/exclude.txt

# Add file/folder/link to list
if [[ -z "$@" ]]; then
  echo " bce <file/folder/list> - add exclusions to backing up configurations"
  else
  echo "`realpath -s "$@"`" >> $excludeloc
fi

#!/bin/bash
# add file/folders to the backup configurations include file

# Include file location
backlsdir="/home/<user>/.bin/root/backup"
incfile="$backlsdir/bckcfg-inc.txt"

# Program name from it's filename
prog=${0##*/}

# Text color variables
txtund=$(tput sgr 0 1)          # Underline
txtbld=$(tput bold)             # Bold
bldred=${txtbld}$(tput setaf 1) #  red
bldblu=${txtbld}$(tput setaf 4) #  blue
bldwht=${txtbld}$(tput setaf 7) #  white
txtrst=$(tput sgr0)             # Reset
info=${bldwht}*${txtrst}        # Feedback
pass=${bldblu}*${txtrst}
warn=${bldred}*${txtrst}

# Display usage if full argument isn't given
if [[ -z "$@" ]]; then
  echo " $prog <file/folder/link> -  add files/folders/links to backup-cfgs' include file"
  exit
fi

# Check if folder exists
if [ ! -d $backlsdir ]; then
  echo "Directory and include file do not exist: $incfile"
  echo "Exiting"
  exit
fi

# Check if file exists
if [ ! -f $incfile ]; then
  echo "Include file doesn't exist: $incfile"
  echo "Exiting"
  exit
fi

# Check link valid, then add
for file in "$@"; do
  fullpath=$(realpath -s "$file")
  if [ ! -e "$fullpath" ]; then
    echo "$warn File ${txtund}$fullpath${txtrst} doesnt exist."
    continue
  fi
  echo "$fullpath" >> "$incfile"
  echo "$pass Added ${txtund}$fullpath${txtrst} in backup include file"
done

# Sort entries
sort -u "$incfile" -o "$incfile"

Backup Script

A tar backup script can be built and then be automated to run on a regular basis. This backup script names your backup by several identifying variables, removes old backups, then backs up your configurations. Again, be sure to add the location of your include and exclude files as well as your backup directory.

#!/bin/bash
# backup configurations with tar

# Backup destination
backdest="/opt/backup"

# Backup name
machine=${HOSTNAME}
distro=arch
type=configs
date=`date "+%F"`
backupfile="$backdest/$distro-$type-$date.tar.gz"

# Include and exclude file locations
prog=${0##*/} # Program name from filename
bcdir="/home/<user>/.bin/root/backup"
include_file="$bcdir"/$prog-inc.txt
exclude_file="$bcdir"/$prog-exc.txt

# Text color variables
txtbld=$(tput bold)             # Bold
bldred=${txtbld}$(tput setaf 1) #  red
bldwht=${txtbld}$(tput setaf 7) #  white
txtrst=$(tput sgr0)             # Reset
info=${bldwht}*${txtrst}        # Feedback
warn=${bldred}!${txtrst}

# Verify that the target directory exists.
if [ ! -d $backdest ]; then
  echo "$warn Backup directory does not exist, exiting."
fi

# Delete backups older than a month
if [[ -n "$(find "$backdest" -mtime +30)" ]]; then
  echo "$info Deleting backups older than one month..."
  find "$backdest" -name "$distro-$type-*" -mtime +30 -exec rm {} \;
fi

# Backup
tar --exclude-from=$exclude_file --files-from=$include_file -cvpzf \
$backupfile

Packages

Package lists can be created that can re-install your programs upon a restore. If you have the hard disk space available, you might also want to consider saving the install packages (Template:Filename) as well.

Creating a Package List

You can create a list of all installed official packages with:

pacman -Qqet | grep -v "$(pacman -Qqm)" > pkglist-off.txt

This will create a list of all packages in the official, enabled pacman repositories (i.e. core, extra, community and testing).

To create a list of all local packages (includes packages installed from the AUR):

pacman -Qqmt > pkglist-loc.txt

Saving Package Tarballs

Pacman saves all package tarballs in Template:Filename. Saving these will increase your re-install speed so consider saving these as well. You might want to think about reducing the size of the cache before backing up too. Pacman has the ability to remove any uninstalled packages from the cache with:

pacman -Sc

If you use Yaourt to install packages from the AUR, you might want to consider setting up a cache for it (Yaourt by default does not save the built package tarballs). To setup a cache directory, edit Template:Filename to include one:

  ExportToLocalRepository /var/cache/pacman/pkg-local

Then give the directory the necessary permissions so Yaourt can write to it as a regular user:

mkdir -p /var/cache/pacman/pkg-local
chmod 766 /var/cache/pacman/pkg-local

Copy these packages to your seperate medium.

Storing the Backup

After you have made up your tarred configurations, package lists, and (optionally) your install packages, you are going to need to store them on a seperate medium than your install partition/drive. Do not put your package lists and install packages in your tarred configurations. This is because all packages must be reinstalled first before you restore your configurations to prevent file conflicts (pacman will not install packages with file conflicts). If you have large enough USB Flash Drive these work well. Optionally you can burn them to a CD or use a partition utility like Template:Package Official to create an extra partition. If using CD's you can span large archives by using the split utility. To create a new partition consider using the Parted Magic CD which has gparted on it.

Restoring

Restoring will involve:

1. Installing the base system through the AIF (Arch Installation Framework).
2. Changing root.
3. Reinstalling all your packages.
4. Extracting your configurations.
5. Adding a new user, and expect a few unexpecteds.

AIF Install

Install Arch Linux as you normally would through the AIF on the LiveCD.

Change Root

When finished, mount your USB Flash Drive (or whatever medium you choose to save your configurations... on).

mkdir /backup-files
mount /dev/<disk-drive-parition> /backup-files

Your Arch install will already be mounted on Template:Filename so now copy these files to your Arch install:

mkdir -p /mnt/opt/restore
cd /backup-files
cp -a * /mnt/opt/restore

Now you will need to Template:Codeline (Change Root) to your Arch install:

cd /mnt
cp /etc/resolv.conf /mnt/etc
mount -t proc none /mnt/arch/proc
mount -t sysfs none /mnt/arch/sys
mount -o bind /dev /mnt/arch/dev
chroot . /bin/bash

Reinstall your Packages

Reinstall packages from the official repositories, the AUR, and locally installed packages separately to better diagnose problems if they occur.

Official

First reinstall packages from the official repositories;

pacman -Sy
pacman -S --needed $(cat /opt/restore/pkglist-off.txt)

The AUR

Yaourt comes in handy here. To quickly install yaourt again:

wget http://aur.archlinux.org/packages/yaourt/yaourt.tar.gz
tar xvf yaourt.tar.gz && cd yaourt*
makepkg -s
pacman -U yaourt-*.pkg.tar.gz

Then to install AUR pakages from the list:

yaourt -S $(cat /opt/restore/pkglist-loc.txt | grep -vx "$(pacman -Qqm)")

Template:Codeline here is used to remove packages that are already installed. This comes in useful in case you have to restart the command because you had trouble installing one of the packages. If you have packages already built by yaourt and in your yaourt cache, you can avoid recompiling again by going to that cache and installing the packages manually (Template:Codeline).

Extract Configurations

Once all packages have been installed you can extract the configurations:

tar xvf /opt/restore/hostname-arch-configs-date-tar.gz -C /mnt

A couple things to look out for:

• Be aware of any changes to your partition layout. If you changed your partition, you will need to edit both Template:Filename and Template:Filename.
• If you had special options for the kernel ram disk (initrd), then you will have to re-compile it before your reboot to get your expected behavior.

Final Details

Good time to add your user now before you reboot. When creating a user, consider giving the user a unique user id (UID). This will help prevent conflicts in the future with other users and programs having the same UID (UIDs for users generally start at 1000):

useradd -m -u 1050 -G audio,optical,power,storage,users,video -s /bin/bash user

If you have restored a user home directory (/home/user) from your backup configurations, the -m switch will give a warning about an already existing home directory but will not alter the directory. Do not forget to change permissions in your home directory if your UIDs differ:

chown -R username:users /home/user

Now, reboot. Expect a few unexpected things here. No re-install is perfect. ALSA may pop up a warning and may have to be configure again and there may be a few other things unconsidered. That's it. Congratulations on your reinstall.