Difference between revisions of "User:Gen2ly/System backup and reinstall"

From ArchWiki
Jump to navigation Jump to search
m
(remove beginners' guide link, see Talk:Installation guide#The Great Merge)
 
(110 intermediate revisions by 16 users not shown)
Line 1: Line 1:
[[Category: Backup (English)]][[Category: Tutorial (English)]]
+
This tutorial is to explain how to backup the system configurations and other likewise items in order to expedite a reinstall process.
This is an disjuncted document for the reason that the only time you really need to restore from configurations only is if:
 
<!-- having trouble understanding this ^ sentence -->
 
# For an unexplained reason a program or programs begin to behave unexpectedly and no help in the forums (or elsewhere) is available is able to fix the problem.  By chance reinstalling your programs and other configurations might fix the problem.
 
# You have limited hard disk space and are not capable of doing a full restore from backup.
 
  
{{Tip|If you are a regular computer user, generally it is good practice to backup your primary drive to a backup hard drive after getting your system installed.  Secondary storage disks can be found realatively inexpensively and are able to store/restore an entire drive of programs, configurations, documents... in a safe and quick way.  If you have a hardware setup like this, look at programs like the [http://clonezilla.org/ Clonezilla CD], or the [http://partedmagic.com/ Parted Magic CD] (includes Clonzilla and other tools) which are both open-source and can image your hard drive to another drive.}}
+
== Justification ==
  
This document is designed to show you how to: quickly add commonly-used/necessary configurations in an include file, back them up on a regular basis, and how to restore all your configurations and packages when rebuilding your system.
+
The justification for this tutorial generally is:
  
== Backup ==
+
* The system architecture is desired to be changed (e.g. 32-bit to 64-bit)
 +
* A system problem has occurred and troubleshooting has failed (external help from the wiki or forums is unavailable)
  
The tar compression tool with a couple helper scripts can backup your configurations in a few steps.
+
The discouragement for this tutorial is:
  
=== Include/Exclude Files  ===
+
* The system installation that is extremely outdated. Because package and configurations change over time this method may be ineffectual and possibly even harmful (in terms of dataloss). For installations older than six months (and sometimes even less) consider reinstalling from the onset according to the [[Installation guide]].
  
tar has the ability to read from an include file that has a list of configurations, documents... to be included when compressing.  The format is stored line by line with it's full base path.  For example:
+
== Notes ==
  
<pre>
+
A few things to know about this tutorial before reading:
/etc/pacman.conf
 
/etc/rc.conf
 
/home/user
 
...
 
</pre>
 
  
And is invoked like this:
+
* A backup destination has been throughout defined:
  
  tar --files-from=include.txt --exclude-from=exclude.txt -cvpzf backupname.tar.gz
+
  /mnt/backup/amiga_$(date +%F).../
  
The name of the include file can be anything you want. Also an exclude file can be used like in the example above. The exclude file is like the include file but additionally has the ability to be able to use [http://www.regular-expressions.info/ regexps] and can be commented and have blank lines:
+
* The system installation is recommended to be updated and configurations merged (if doable). This will keep backup configurations equivocal with their reinstalled counterparts.
 +
* The backup is recommended to be saved on a separate storage disk. This will ease the process of the backup.
  
<pre>
+
== Backup ==
# Trashes not necessary
 
/home/*/.local/share/Trash/files
 
/home/*/.Trash
 
</pre>
 
  
=== One step to add to (in|ex)clude files ===
+
The backup process will involve these steps:
  
To help add files to the include and exclude files, a couple bash scripts can help to accomplish this.  bca (backup-config-add) and bce (backup-cfg-exclude) can be used from the command line:
+
# Backup system configurations
 +
# Record the package list(s)
 +
# Backup the package tarballs
  
<pre>
+
=== Configurations ===
bca /etc/X11/xorg.conf
 
cd /home/user
 
bce .thumbnails/
 
</pre>
 
  
Both scripts have the ability to read the full path so entering a relative path isn't a problem.  Put them in your script directory and make them executable to add a file anytime you think of it.
+
To create a backup in Linux, two common utilities are generally employed [[tar]] and [[rsync]].
  
<pre>
+
A typical method to do multi-file backups is to create a list of the required files. {{ic|tar}} and {{ic|rsync}} both have the ability to define a file-list in an include and an exclude file. The format to use is one file per line with the full path. For example:
#!/bin/bash
 
# bce (backup-config-exclude) - add files to be excluded in backup
 
  
excludeloc=/home/user/.scripts/backup/exclude.txt
+
{{bc|include.txt
 
+
/etc/fstab
# Add file/folder/link to list
+
/etc/modules-load.d/
if [[ -z "$@" ]]; then
+
/etc/pacman.conf
  echo " bce <file/folder/list> - add exclusions to backing up configurations"
+
...
  else
+
}}
  echo "`readlink -f "$@"`" >> $excludeloc
 
fi
 
</pre>
 
 
 
<pre>
 
#!/bin/bash
 
# bca (backup-cfg-add) - add file/folders to the backup-cfg include file
 
 
 
incfile=/home/user/.scripts/backup/include.txt
 
fullpath=$(readlink -f "$@")
 
 
 
# Display usage if full argument isn't given
 
if [[ -z "$@" ]]; then
 
  echo " bca <file/folder/link> -  add files/folders/links to backup-cfgs' include file"
 
  exit
 
fi
 
 
 
# Test if path exists
 
if [ ! -e "$fullpath" ]; then
 
  echo -e " File "$@" doesn't exist."
 
  exit
 
fi
 
 
 
# Add entry
 
echo "$fullpath" >> "$incfile"
 
echo -e "${bldblu}*${txtrst} Added ${undwht}$fullpath${txtrst} in backup include file"
 
 
 
# Sort entries
 
sort -u "$incfile" -o "$incfile"
 
</pre>
 
 
 
=== Backup Script ===
 
 
 
A tar backup script can help the process of backing up your configurations and be automated on a regular basis.
 
 
 
This backup script names your backup by hostname and date, specifies you include and exclude file locations, and removes old backups.
 
 
 
<pre>
 
#!/bin/bash
 
# backup-cfg - backup configurations with tar
 
 
 
# Backup name
 
machine=$HOSTNAME
 
distro=arch
 
type=configs
 
date=`date "+%F"`
 
backupname=$machine-$distro-$type-$date
 
 
 
# Backup destination
 
backdest="/opt/backup"
 
 
 
# Files contailing information to include and exclude
 
include_file="/home/user/.scripts/backup/include.txt"
 
exclude_file="/home/user/.scripts/backup/exclude.txt"
 
 
 
# Verify that the target directory exists.
 
if [ ! -d $backdest ]; then
 
  echo " Backup directory does not exist, creating"
 
  mkdir $backdest
 
  else
 
  echo " Backup directory exists."
 
fi
 
 
 
# Delete backups older than two months
 
if [[ -n "$(find "$backdest" -mtime +60)" ]]; then
 
  find "$backdest" -mtime +60 -exec rm {} \;
 
  echo -e " Backups older than two months deleted"
 
fi
 
 
 
# Unmount other drives/partitions
 
#umount -a
 
 
 
# Backup
 
tar --exclude-from=$exclude_file --files-from=$include_file -cvpzf \
 
$backdest/$backupname.tar.gz
 
</pre>
 
 
 
To do automated backups (for example, every week), put the script in your root crontab ({{Codeline|crontab -e}}):
 
 
 
<pre>
 
#
 
# DO NOT EDIT THIS FILE MANUALLY!! USE crontab -e INSTEAD.
 
#
 
 
 
  50  00  *  *  0      /home/user/.scripts/backup/backup-cfg
 
# |  |  |  |  |  user
 
# |  |  |  |  weekday (0-6) (Sunday=0)
 
# |  |  |  month (1-12)
 
# |  |  day (0-31)
 
# |  hour (0-23)
 
# minute (0-59)
 
</pre>
 
 
 
=== What not to include in the backup ===
 
 
 
There are some files that shouldn't be included in the backup directly.  The reason for this will become apparently later in the document.  For now, if you want these files backed up copy them to a new name:
 
 
 
<pre>
 
cp /boot/grub/menu.lst /boot/grub/menu.lst-good
 
cp /etc/fstab /etc/fstab-good
 
cp /etc/sudoers /etc/sudoers-good
 
</pre>
 
 
 
Then add them to your include file.
 
 
 
=== Packages ===
 
 
 
A package list can be created that can be used to restore all installed packages.  You might also want to consider (if you have the hard disk space available) of backing up the install packages for them.
 
 
 
==== Creating a Package List ====
 
 
 
You can create a list of official installed packages with:
 
 
 
pacman -Qqet | grep -v "$(pacman -Qqm)" > /opt/backup/pkg-official
 
bca /opt/backup/pkg-official
 
 
 
This will create a list of all pacman repositories enabled (core/extra/community/testing). To create a list of all local packages (includes packages installed from the AUR):
 
 
 
pacman -Qqmt > /opt/backup/pkg-local
 
bca /opt/backup/pkg-local
 
 
 
Alternately, you can use the [http://aur.archlinux.org/packages.php?ID=22747 packup] utility in the AUR.  Packup is a backup and restore from list utility with menu driven interface.  Packup will create a list and save it in the root folder.  If you use packup, create a seperate list for official and local packages for easier debugging later.
 
 
 
==== Saving Package Tarballs ====
 
 
 
Pacman saves all package tarballs in /var/cache/pacman/pkg/.  Saving these will increase the re-install speed so consider adding this folder to your include file.  You can reduce the size of the cache to the last two installed versions of a package with:
 
 
 
pacman -Sc
 
 
 
If you use Yaourt to install from the AUR, you might want to consider setting up a cache for it too.  Though Yaourt doesn't have the ablity to read from the cache before an install these packages can later be installed with {{Codeline|pacman -U <package>}}.  In {{Filename|/etc/yaourtrc}} (for example):
 
 
 
  ExportToLocalRepository /var/cache/yaourt/pkg
 
 
 
mkdir -p /var/cache/yaourt/pkg
 
chmod 766 /var/cache/yaourt/pkg/
 
bca /var/cache/yaourt/pkg/
 
 
 
=== Storing the Backup ===
 
 
 
The examples above save the backups to the same partition (provided you don't have a seperate /opt partition).  If you have the hard disk space, it is better to put these on a seperate partition or drive.  Consider using the [http://partedmagic.com/ Parted Magic CD] to create one if you want to do this.  If you don't, you will need to put them on a seperate media before you restore.  If you absolutely have to, you can break up tar files and span them over [http://en.gentoo-wiki.com/wiki/Backup_to_DVD multiple disks].
 
  
== Restoring ==
+
{{bc|exclude.txt
 +
# only testing
 +
/etc/modules-load.d/loop*
 +
}}
  
Restoring will require: reinstalling, extracting the backup configurations back on the new system, and checking for conflicts.
+
To create a backup from the file-list, with {{ic|tar}} and {{ic|rsync}} respectively do:
  
=== Installing/Restore configs ===
+
tar --create --xz --files-from=include.txt --exclude-from=exclude.txt --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz
 +
rsync --archive --files-from=include.txt --exclude-from=exclude.txt / /mnt/backup/amiga_$(date +%F)_syscfg/
  
Install Arch Linux as you normally would and then mount your partition with your backup on it:
+
(The include file ''must'' list valid files or the programs will exit with an error. However, the exclude file does ''not'' have to list valid files because it allows glob pattern matches.)
  
mkdir /backup-part
+
(The {{ic|rsync}} command defines the source directory as the root directory ({{ic|/}}), this is necessary to coincide with the full path specified in the file listing.)
mount /dev/<your-partition> /backup-part
 
  
And extract your backup onto the new install:
+
See also [[Full System Backup with tar]] and [[Full system backup with rsync]].
  
tar xvf hostname-arch-configs-date-tar.gz -C /mnt
+
=== Package lists ===
  
The installer will have already mounted your install partition on /mnt.  This is where not including those files directly above comes in handy. Doing this will overwrite any files that match on your backup and the new install.  Check your good mount file (fstab) and bootloader file 9menu.lst) with the new install version and update where necessary, then reboot.
+
A listing can be created of installed packages of both ''official'' repository packages and ''local'' packages. These lists can be used upon the reinstall to define the package selection.
  
=== User/Yaourt ===
+
To create a list of all ''official'' repository installed packages do:
  
Continue your install as you normally would. When creating a user consider giving the user a unique user id (uid).  This will help prevent conflicts in the future with other users and programs having the same uid (uid's for users generally start at 1000):
+
  $ pacman -Qqe | grep -v "$(pacman -Qqm)" > /mnt/backup/amiga_$(date +%F)_pkglist-off.txt
  
<pre>
+
To create a list of all ''local'' installed packages do (includes packages installed from the [[AUR]]):
useradd -m -u 1050 -p '$1$UUiuruAK$5PrDq8BhI2094gMaN8TYE1' -G \
 
audio,optical,power,storage,users,video -s /bin/bash user
 
</pre>
 
  
If you have restored a user home directory (/home/user) in your backup configurations, the '''-m''' switch will give a warning but will not effect anything.  The '''-p''' switch will input the password (in this case: abcdefg :) ). Don't forget to change permissions in your home directory if you change the uid:
+
$ pacman -Qqm > /mnt/backup/amiga_$(date +%F)_pkglist-loc.txt
  
chown -R username:users /home/user
+
See also [[Pacman tips#Backing up and retrieving a list of installed packages]].
  
If you previously had any AUR packages installed, you can install Yaourt to restore the packages:
+
=== Package tarballs ===
  
<pre>
+
By creating a backup of package tarballs that have been previously downloaded time can be saved. During a reinstall a package cache can be defined to the package manager of the previous package tarballs.
wget http://aur.archlinux.org/packages/yaourt/yaourt.tar.gz
 
tar xvf yaourt.tar.gz && cd yaourt*
 
makepkg -s
 
pacman -U yaourt-*.pkg.tar.gz
 
</pre>
 
  
=== Reinstalling Packages ===
+
To remove package tarballs that are uninstalled or older versions do:
  
Reinstall packages from the official repositories, the AUR, and locally installed packages seperately to better diagnose problems if they occur.
+
# pacman -Sc
  
==== Official ====
+
To backup package tarballs:
  
First reinstall packages from the official repositories;
+
# cp /var/cache/pacman/pkg/* /mnt/backup/amiga_$(date +%F)_pkg/
  
<pre>
+
== Reinstall ==
pacman -Sy
 
pacman -S --needed $(cat /opt/backup/pkg-official)
 
</pre>
 
  
Pacman will not install packages with file conflicts (i.e. if you have configurations placed that pacman detects an installing package has as well).  Take a note of the conflicting packages and either: move the conflicting configuration (if you know what it is), or force pacman to overwrite the conflicting files:
+
The reinstall process will involve these steps:
  
pacman -S -f --needed $(cat /opt/backup/pkg-official)
+
# Install the base system
 +
# Generate a locale
 +
# Install previous packages
 +
# Restore previous configurations
  
If you do this, you will need to find out the conflicting file and reinstall it again.  Pacman can tell you what files a certain package installs:
+
=== Base install ===
  
pacman -Ql <package>
+
Install the base system normally as directed in the [[Installation guide]] except without editing the configurations.
  
And you can extract a file or files from an archive with:
+
=== Locale generation ===
  
tar xvf hostname-arch-configs-date-tar.gz etc/hosts etc/resolve.conf
+
Generating a locale will be the one exception where editing a configuration will be necessary (a few packages use scripts to install certain features that require a locale be present). Read [[Locale]] to learn more.
  
and place them manually.
+
=== Package reinstall ===
  
==== The AUR ====
+
To reinstall packages from the ''official'' repositories (using the backup package cache and the package list):
  
From the AUR:
+
pacman --needed --cachedir /mnt/backup/amiga_$(date +%F)_pkg/ -S - < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt
  
<pre>
+
To reinstall packages that are ''local'' will depend on the method or program that was originally used. If the package tarballs exist for them and they are in the same cache, the above command can be used with the {{ic|...pkglist-loc.txt}} file. If the packages do not have a package tarball and are AUR packages, use a [[AUR Helper]] to reinstall them:
cat /opt/backup/pc-emach/cat /opt/backup/pkg-local | grep -v "$(pacman -Qqm)" \
 
| xargs yaourt -Ss
 
</pre>
 
  
The extra checks are to not include any packages already installed. For example, if there are any problems installing a package from the AUR... you will not have to go through the whole list again and tell Yaourt what to install (i.e. Yaourt doesn't suppport the '''--needed''' switch).
+
  AURHELPER ... < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt
  
==== Local ====
+
=== Restore configurations ===
  
If you created a Yaourt cache or have your own packages saved, you can install them with:
+
When all the packages have been installed the configurations can be restored with {{ic|tar}} and {{ic|rsync}} respectively:
  
  pacman -U <*--asdeps> <package>
+
  tar --extract --verbose --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz -C /mnt/install
 +
rsync --archive /mnt/backup/amiga_$(date +%F)_syscfg/ /mnt/install/
  
== Resources ==
+
== Details ==
  
* [http://bbs.archlinux.org/viewtopic.php?id=76218 BBS - List AUR packages and only AUR packages]
+
* Installation guide procedures should still be observed and keep in mind that they change temporally.
 +
* Partition layout changes will need to be attention paid to. If device designation has changed {{ic|/etc/fstab}}, bootloader configuration will need to be updated.
 +
* Non-default options for the kernel ram disk will require the rebuilding if it before rebooting.
 +
* Edits in the kernel ram disk configuration will necessitate its rebuilding before rebooting.

Latest revision as of 22:34, 26 August 2016

This tutorial is to explain how to backup the system configurations and other likewise items in order to expedite a reinstall process.

Justification

The justification for this tutorial generally is:

  • The system architecture is desired to be changed (e.g. 32-bit to 64-bit)
  • A system problem has occurred and troubleshooting has failed (external help from the wiki or forums is unavailable)

The discouragement for this tutorial is:

  • The system installation that is extremely outdated. Because package and configurations change over time this method may be ineffectual and possibly even harmful (in terms of dataloss). For installations older than six months (and sometimes even less) consider reinstalling from the onset according to the Installation guide.

Notes

A few things to know about this tutorial before reading:

  • A backup destination has been throughout defined:
/mnt/backup/amiga_$(date +%F).../
  • The system installation is recommended to be updated and configurations merged (if doable). This will keep backup configurations equivocal with their reinstalled counterparts.
  • The backup is recommended to be saved on a separate storage disk. This will ease the process of the backup.

Backup

The backup process will involve these steps:

  1. Backup system configurations
  2. Record the package list(s)
  3. Backup the package tarballs

Configurations

To create a backup in Linux, two common utilities are generally employed tar and rsync.

A typical method to do multi-file backups is to create a list of the required files. tar and rsync both have the ability to define a file-list in an include and an exclude file. The format to use is one file per line with the full path. For example:

include.txt
/etc/fstab
/etc/modules-load.d/
/etc/pacman.conf
...
exclude.txt
# only testing
/etc/modules-load.d/loop*

To create a backup from the file-list, with tar and rsync respectively do:

tar --create --xz --files-from=include.txt --exclude-from=exclude.txt --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz
rsync --archive --files-from=include.txt --exclude-from=exclude.txt / /mnt/backup/amiga_$(date +%F)_syscfg/

(The include file must list valid files or the programs will exit with an error. However, the exclude file does not have to list valid files because it allows glob pattern matches.)

(The rsync command defines the source directory as the root directory (/), this is necessary to coincide with the full path specified in the file listing.)

See also Full System Backup with tar and Full system backup with rsync.

Package lists

A listing can be created of installed packages of both official repository packages and local packages. These lists can be used upon the reinstall to define the package selection.

To create a list of all official repository installed packages do:

$ pacman -Qqe | grep -v "$(pacman -Qqm)" > /mnt/backup/amiga_$(date +%F)_pkglist-off.txt

To create a list of all local installed packages do (includes packages installed from the AUR):

$ pacman -Qqm > /mnt/backup/amiga_$(date +%F)_pkglist-loc.txt

See also Pacman tips#Backing up and retrieving a list of installed packages.

Package tarballs

By creating a backup of package tarballs that have been previously downloaded time can be saved. During a reinstall a package cache can be defined to the package manager of the previous package tarballs.

To remove package tarballs that are uninstalled or older versions do:

# pacman -Sc

To backup package tarballs:

# cp /var/cache/pacman/pkg/* /mnt/backup/amiga_$(date +%F)_pkg/

Reinstall

The reinstall process will involve these steps:

  1. Install the base system
  2. Generate a locale
  3. Install previous packages
  4. Restore previous configurations

Base install

Install the base system normally as directed in the Installation guide except without editing the configurations.

Locale generation

Generating a locale will be the one exception where editing a configuration will be necessary (a few packages use scripts to install certain features that require a locale be present). Read Locale to learn more.

Package reinstall

To reinstall packages from the official repositories (using the backup package cache and the package list):

pacman --needed --cachedir /mnt/backup/amiga_$(date +%F)_pkg/ -S - < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt

To reinstall packages that are local will depend on the method or program that was originally used. If the package tarballs exist for them and they are in the same cache, the above command can be used with the ...pkglist-loc.txt file. If the packages do not have a package tarball and are AUR packages, use a AUR Helper to reinstall them:

AURHELPER ... < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt

Restore configurations

When all the packages have been installed the configurations can be restored with tar and rsync respectively:

tar --extract --verbose --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz -C /mnt/install
rsync --archive /mnt/backup/amiga_$(date +%F)_syscfg/ /mnt/install/

Details

  • Installation guide procedures should still be observed and keep in mind that they change temporally.
  • Partition layout changes will need to be attention paid to. If device designation has changed /etc/fstab, bootloader configuration will need to be updated.
  • Non-default options for the kernel ram disk will require the rebuilding if it before rebooting.
  • Edits in the kernel ram disk configuration will necessitate its rebuilding before rebooting.