Difference between revisions of "User:Gen2ly/System backup and reinstall"

From ArchWiki
Jump to: navigation, search
(Backup Script: The wiki is not an appropriate location to host entire scripts. Create a repo and link to it instead.)
(remove beginners' guide link, see Talk:Installation guide#The Great Merge)
 
(39 intermediate revisions by 9 users not shown)
Line 1: Line 1:
[[Category:System recovery]]
+
This tutorial is to explain how to backup the system configurations and other likewise items in order to expedite a reinstall process.
This article is intended to show you how to backup your configurations and your package list, then to do a full system restore, restore packages, and finally restore your configurations..
+
  
== Motivation ==
+
== Justification ==
  
The need for this documentation is uncommon in the sense that the need to restore from ''configurations only'' is only really necessary for the following reasons:
+
The justification for this tutorial generally is:
  
# You would like to change your system architecture (e.g. 32bit to 64bit).
+
* The system architecture is desired to be changed (e.g. 32-bit to 64-bit)
# If a program or programs begin to behave unexpectedly and no help in the forums (or elsewhere) is available is able to fix the problem.  By chance reinstalling your programs and other configurations might fix the problem.
+
* A system problem has occurred and troubleshooting has failed (external help from the wiki or forums is unavailable)
# You have limited hard disk space and are not capable of doing a full restore from backup.
+
  
{{Tip|If you are a regular computer user, generally it is good practice to backup your primary drive to a backup hard drive after getting your system installed.  Secondary storage disks can be found relatively inexpensively these days and are able to store/restore an entire drive of programs, configurations, documents, etc. in a safe and reliable way.  If you have a hardware setup like this, look at programs like the [http://clonezilla.org/ Clonezilla CD], or the [http://partedmagic.com/ Parted Magic CD] (includes Clonzilla and other tools) which are both open-source and can image your hard drive for later restoration.}}
+
The discouragement for this tutorial is:
  
== Backup ==
+
* The system installation that is extremely outdated. Because package and configurations change over time this method may be ineffectual and possibly even harmful (in terms of dataloss). For installations older than six months (and sometimes even less) consider reinstalling from the onset according to the [[Installation guide]].
  
Using {{Ic|tar}} in a script can make archiving configurations get done in just a couple steps.
+
== Notes ==
  
=== Include and Exclude Files ===
+
A few things to know about this tutorial before reading:
  
Tar has the ability to read from both an include and an exclude file.  This means that you can tell tar everything you would like to include in the backup and exclude just by using two files.  The format used is one line per file or directory that indicates the full path.  For example:
+
* A backup destination has been throughout defined:
  
<pre>
+
/mnt/backup/amiga_$(date +%F).../
/etc/pacman.conf
+
/etc/rc.conf
+
/home/user
+
...
+
</pre>
+
  
And is invoked like this:
+
* The system installation is recommended to be updated and configurations merged (if doable). This will keep backup configurations equivocal with their reinstalled counterparts.
 +
* The backup is recommended to be saved on a separate storage disk. This will ease the process of the backup.
  
tar --files-from=include.txt --exclude-from=exclude.txt -c --xz -f backupname.tar.xz
+
== Backup ==
  
The name of the files can be anything you want.  The exclude file is like the include file but additionally has the ability to be able to use [http://www.regular-expressions.info/ regexps], as well as being able to be commented and have blank lines.
+
The backup process will involve these steps:
  
=== Packages ===
+
# Backup system configurations
 +
# Record the package list(s)
 +
# Backup the package tarballs
  
Package lists can be created that can re-install your programs upon a restore.  If you have the hard disk space available, you might also want to consider saving the install packages ({{ic|*.pkg.tar.gz}}) as well.
+
=== Configurations ===
  
==== Creating a Package List ====
+
To create a backup in Linux, two common utilities are generally employed [[tar]] and [[rsync]].
  
You can create a list of all installed ''official'' packages with:
+
A typical method to do multi-file backups is to create a list of the required files. {{ic|tar}} and {{ic|rsync}} both have the ability to define a file-list in an include and an exclude file. The format to use is one file per line with the full path. For example:
  
pacman -Qqe | grep -v "$(pacman -Qqm)" > pkglist-off.txt
+
{{bc|include.txt
 
+
/etc/fstab
This will create a list of all packages in the official, enabled [[pacman]] repositories (i.e. core, extra, community and testing).
+
/etc/modules-load.d/
 
+
/etc/pacman.conf
To create a list of all ''local'' packages (includes packages installed from the [[AUR]]):
+
...
 
+
}}
pacman -Qqm > pkglist-loc.txt
+
 
+
==== Saving Package Tarballs ====
+
 
+
Pacman saves all package tarballs in {{ic|/var/cache/pacman/pkg/}}. Saving these will increase your re-install speed so consider saving these as well.  You might want to think about reducing the size of the cache before backing up too.  Pacman has the ability to remove any uninstalled packages from the cache with:
+
 
+
pacman -Sc
+
 
+
===== Yaourt =====
+
 
+
If you use [[Yaourt]] to install packages from the AUR, you might want to consider setting up a cache for it (Yaourt by default does not save the built package tarballs).  To setup a cache directory, edit {{ic|/etc/yaourtrc}} to include one:
+
 
+
  ExportToLocalRepository /var/cache/pacman/pkg-local
+
 
+
Then give the directory the necessary permissions so Yaourt can write to it as a regular user:
+
 
+
mkdir -p /var/cache/pacman/pkg-local
+
chmod 766 /var/cache/pacman/pkg-local
+
 
+
Copy these packages to your seperate medium.
+
  
=== Storing the Backup ===
+
{{bc|exclude.txt
 +
# only testing
 +
/etc/modules-load.d/loop*
 +
}}
  
After you have made up your tarred configurations, package lists, and (optionally) your install packages, you are going to need to store them on a seperate medium than your install partition/drive.  Do not put your package lists and install packages in your tarred configurations. This is because all packages must be reinstalled first before you restore your configurations to prevent file conflicts (pacman will not install packages with file conflicts).  If you have large enough USB Flash Drive these work well.  Optionally you can burn them to a CD or use a partition utility like {{Pkg|gparted}} to create an extra partition.  If using CD's you can span large archives by using the [http://en.gentoo-wiki.com/wiki/Backup_to_DVD split utility].  To create a new partition consider using the [http://partedmagic.com/ Parted Magic CD] which has gparted on it.
+
To create a backup from the file-list, with {{ic|tar}} and {{ic|rsync}} respectively do:
  
== Restoring ==
+
tar --create --xz --files-from=include.txt --exclude-from=exclude.txt --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz
 +
rsync --archive --files-from=include.txt --exclude-from=exclude.txt / /mnt/backup/amiga_$(date +%F)_syscfg/
  
Restoring will involve:
+
(The include file ''must'' list valid files or the programs will exit with an error. However, the exclude file does ''not'' have to list valid files because it allows glob pattern matches.)
  
# Installing the base system through the AIF (Arch Installation Framework).
+
(The {{ic|rsync}} command defines the source directory as the root directory ({{ic|/}}), this is necessary to coincide with the full path specified in the file listing.)
# Changing root.
+
# Reinstalling all your packages.
+
# Extracting your configurations.
+
# Adding a new user.
+
  
=== AIF Install ===
+
See also [[Full System Backup with tar]] and [[Full system backup with rsync]].
  
Install Arch Linux as you normally would through the AIF on the LiveCD.
+
=== Package lists ===
  
=== Change Root ===
+
A listing can be created of installed packages of both ''official'' repository packages and ''local'' packages. These lists can be used upon the reinstall to define the package selection.
  
When finished, mount your USB Flash Drive (or whatever medium you choose to save your configurations... on).
+
To create a list of all ''official'' repository installed packages do:
  
  mkdir /backup-files
+
  $ pacman -Qqe | grep -v "$(pacman -Qqm)" > /mnt/backup/amiga_$(date +%F)_pkglist-off.txt
mount /dev/<disk-drive-parition> /backup-files
+
  
Your Arch install will already be mounted on {{ic|/mnt}} so now copy these files to your Arch install:
+
To create a list of all ''local'' installed packages do (includes packages installed from the [[AUR]]):
  
  mkdir -p /mnt/opt/restore
+
  $ pacman -Qqm > /mnt/backup/amiga_$(date +%F)_pkglist-loc.txt
cd /backup-files
+
cp -a * /mnt/opt/restore
+
  
Now you will need to {{Ic|chroot}} ([[Change Root]]) to your Arch install:
+
See also [[Pacman tips#Backing up and retrieving a list of installed packages]].
  
cd /mnt
+
=== Package tarballs ===
cp /etc/resolv.conf /mnt/etc
+
mount -t proc none /mnt/arch/proc
+
mount -t sysfs none /mnt/arch/sys
+
mount -o bind /dev /mnt/arch/dev
+
chroot . /bin/bash
+
  
=== Reinstall your Packages ===
+
By creating a backup of package tarballs that have been previously downloaded time can be saved. During a reinstall a package cache can be defined to the package manager of the previous package tarballs.
  
Reinstall packages from the official repositories, the AUR, and locally installed packages separately to better diagnose problems if they occur.
+
To remove package tarballs that are uninstalled or older versions do:
  
==== Official ====
+
# pacman -Sc
  
First reinstall packages from the official repositories;
+
To backup package tarballs:
  
<pre>
+
# cp /var/cache/pacman/pkg/* /mnt/backup/amiga_$(date +%F)_pkg/
pacman -Sy
+
pacman -S --needed $(cat /opt/restore/pkglist-off.txt)
+
</pre>
+
  
==== The AUR ====
+
== Reinstall ==
  
Yaourt comes in handy here.  To quickly install yaourt again:
+
The reinstall process will involve these steps:
  
<pre>
+
# Install the base system
wget http://aur.archlinux.org/packages/yaourt/yaourt.tar.gz
+
# Generate a locale
tar xvf yaourt.tar.gz && cd yaourt*
+
# Install previous packages
makepkg -s
+
# Restore previous configurations
pacman -U yaourt-*.pkg.tar.gz
+
</pre>
+
  
Then to install AUR pakages from the list:
+
=== Base install ===
  
yaourt -S $(cat /opt/restore/pkglist-loc.txt | grep -vx "$(pacman -Qqm)")
+
Install the base system normally as directed in the [[Installation guide]] except without editing the configurations.
  
{{Ic|grep -vx ...}} here is used to remove packages that are already installed.  This comes in useful in case you have to restart the command because you had trouble installing one of the packages.  If you have packages already built by yaourt and in your yaourt cache, you can avoid recompiling again by going to that cache and installing the packages manually ({{Ic|pacman -U ...}}).
+
=== Locale generation ===
  
=== Extract Configurations ===
+
Generating a locale will be the one exception where editing a configuration will be necessary (a few packages use scripts to install certain features that require a locale be present). Read [[Locale]] to learn more.
  
Once all packages have been installed you can extract the configurations:
+
=== Package reinstall ===
  
tar xvf /opt/restore/hostname-arch-configs-date-tar.gz -C /mnt
+
To reinstall packages from the ''official'' repositories (using the backup package cache and the package list):
  
A couple things to look out for:
+
pacman --needed --cachedir /mnt/backup/amiga_$(date +%F)_pkg/ -S - < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt
  
* Be aware of any changes to your partition layout. If you changed your partition, you will need to edit both {{ic|/etc/fstab}} and {{ic|/boot/grub/menu.lst}}.
+
To reinstall packages that are ''local'' will depend on the method or program that was originally used. If the package tarballs exist for them and they are in the same cache, the above command can be used with the {{ic|...pkglist-loc.txt}} file. If the packages do not have a package tarball and are AUR packages, use a [[AUR Helper]] to reinstall them:
* If you had special options for the kernel ram disk (initrd), then you will have to re-compile it before your reboot to get your expected behavior.
+
  
=== Final Details ===
+
AURHELPER ... < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt
  
Good time to add your user now before you reboot.  When creating a user, consider giving the user a unique user id (UID).  This will help prevent conflicts in the future with other users and programs having the same UID (UIDs for users generally start at 1000):
+
=== Restore configurations ===
  
<pre>
+
When all the packages have been installed the configurations can be restored with {{ic|tar}} and {{ic|rsync}} respectively:
useradd -m -u 1050 -G audio,optical,power,storage,users,video -s /bin/bash user
+
</pre>
+
  
If you have restored a user home directory (/home/user) from your backup configurations, the '''-m''' switch will give a warning about an already existing home directory but will not alter the directoryDo not forget to change permissions in your home directory if your UIDs differ:
+
tar --extract --verbose --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz -C /mnt/install
 +
  rsync --archive /mnt/backup/amiga_$(date +%F)_syscfg/ /mnt/install/
  
chown -R username:users /home/user
+
== Details ==
  
Now, '''reboot'''. Expect a few unexpected things here. No re-install is perfect.  ALSA may pop up a warning and may have to be configure again and there may be a few other things unconsidered. That's it. Congratulations on your reinstall.
+
* Installation guide procedures should still be observed and keep in mind that they change temporally.
 +
* Partition layout changes will need to be attention paid to. If device designation has changed {{ic|/etc/fstab}}, bootloader configuration will need to be updated.
 +
* Non-default options for the kernel ram disk will require the rebuilding if it before rebooting.
 +
* Edits in the kernel ram disk configuration will necessitate its rebuilding before rebooting.

Latest revision as of 22:34, 26 August 2016

This tutorial is to explain how to backup the system configurations and other likewise items in order to expedite a reinstall process.

Justification

The justification for this tutorial generally is:

  • The system architecture is desired to be changed (e.g. 32-bit to 64-bit)
  • A system problem has occurred and troubleshooting has failed (external help from the wiki or forums is unavailable)

The discouragement for this tutorial is:

  • The system installation that is extremely outdated. Because package and configurations change over time this method may be ineffectual and possibly even harmful (in terms of dataloss). For installations older than six months (and sometimes even less) consider reinstalling from the onset according to the Installation guide.

Notes

A few things to know about this tutorial before reading:

  • A backup destination has been throughout defined:
/mnt/backup/amiga_$(date +%F).../
  • The system installation is recommended to be updated and configurations merged (if doable). This will keep backup configurations equivocal with their reinstalled counterparts.
  • The backup is recommended to be saved on a separate storage disk. This will ease the process of the backup.

Backup

The backup process will involve these steps:

  1. Backup system configurations
  2. Record the package list(s)
  3. Backup the package tarballs

Configurations

To create a backup in Linux, two common utilities are generally employed tar and rsync.

A typical method to do multi-file backups is to create a list of the required files. tar and rsync both have the ability to define a file-list in an include and an exclude file. The format to use is one file per line with the full path. For example:

include.txt
/etc/fstab
/etc/modules-load.d/
/etc/pacman.conf
...
exclude.txt
# only testing
/etc/modules-load.d/loop*

To create a backup from the file-list, with tar and rsync respectively do:

tar --create --xz --files-from=include.txt --exclude-from=exclude.txt --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz
rsync --archive --files-from=include.txt --exclude-from=exclude.txt / /mnt/backup/amiga_$(date +%F)_syscfg/

(The include file must list valid files or the programs will exit with an error. However, the exclude file does not have to list valid files because it allows glob pattern matches.)

(The rsync command defines the source directory as the root directory (/), this is necessary to coincide with the full path specified in the file listing.)

See also Full System Backup with tar and Full system backup with rsync.

Package lists

A listing can be created of installed packages of both official repository packages and local packages. These lists can be used upon the reinstall to define the package selection.

To create a list of all official repository installed packages do:

$ pacman -Qqe | grep -v "$(pacman -Qqm)" > /mnt/backup/amiga_$(date +%F)_pkglist-off.txt

To create a list of all local installed packages do (includes packages installed from the AUR):

$ pacman -Qqm > /mnt/backup/amiga_$(date +%F)_pkglist-loc.txt

See also Pacman tips#Backing up and retrieving a list of installed packages.

Package tarballs

By creating a backup of package tarballs that have been previously downloaded time can be saved. During a reinstall a package cache can be defined to the package manager of the previous package tarballs.

To remove package tarballs that are uninstalled or older versions do:

# pacman -Sc

To backup package tarballs:

# cp /var/cache/pacman/pkg/* /mnt/backup/amiga_$(date +%F)_pkg/

Reinstall

The reinstall process will involve these steps:

  1. Install the base system
  2. Generate a locale
  3. Install previous packages
  4. Restore previous configurations

Base install

Install the base system normally as directed in the Installation guide except without editing the configurations.

Locale generation

Generating a locale will be the one exception where editing a configuration will be necessary (a few packages use scripts to install certain features that require a locale be present). Read Locale to learn more.

Package reinstall

To reinstall packages from the official repositories (using the backup package cache and the package list):

pacman --needed --cachedir /mnt/backup/amiga_$(date +%F)_pkg/ -S - < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt

To reinstall packages that are local will depend on the method or program that was originally used. If the package tarballs exist for them and they are in the same cache, the above command can be used with the ...pkglist-loc.txt file. If the packages do not have a package tarball and are AUR packages, use a AUR Helper to reinstall them:

AURHELPER ... < /mnt/backup/amiga_$(date +%F)_pkglist-off.txt

Restore configurations

When all the packages have been installed the configurations can be restored with tar and rsync respectively:

tar --extract --verbose --file /mnt/backup/amiga_$(date +%F)_syscfg.tar.xz -C /mnt/install
rsync --archive /mnt/backup/amiga_$(date +%F)_syscfg/ /mnt/install/

Details

  • Installation guide procedures should still be observed and keep in mind that they change temporally.
  • Partition layout changes will need to be attention paid to. If device designation has changed /etc/fstab, bootloader configuration will need to be updated.
  • Non-default options for the kernel ram disk will require the rebuilding if it before rebooting.
  • Edits in the kernel ram disk configuration will necessitate its rebuilding before rebooting.