Enhance system stability
- 1 Introduction
- 2 Setting Up Arch
- 2.1 Arch Specific Tips
- 2.1.1 Set Up a Large /var Partition and Keep Old Packages
- 2.1.2 Activate as Few Modules and Daemons in rc.conf as Possible
- 2.1.3 Use Recommended Configurations
- 2.1.4 Select the Core and Extra Software Repositories
- 2.1.5 Use Up to Date Mirrors
- 2.1.6 Avoid Development Packages
- 2.1.7 Consider Using yaourt to Simplify SysAdmin Tasks
- 2.2 Generic Best Practices
- 2.1 Arch Specific Tips
- 3 Maintaining Arch
- 3.1 Arch Specific Tips
- 3.1.1 Upgrade Entire System with Reasonable Frequency
- 3.1.2 Set IgnorePkg
- 3.1.3 Read Before Upgrading the System
- 3.1.4 Act on Alerts During an Upgrade
- 3.1.5 Deal Promptly with .pacnew, .pacsave, and .pacorig Files
- 3.1.6 Consider Using Pacmatic
- 3.1.7 Avoid Certain Pacman Commands
- 3.1.8 Revert Package Upgrades That Cause Instability
- 3.1.9 Regularly Purge Cruft
- 3.1.10 Regularly Backup a List of Installed Packages
- 3.1.11 Regularly Backup the Pacman Database
- 3.2 Generic Best Practices
- 3.1 Arch Specific Tips
The purpose of this wiki article is to provide tips on how to make an Arch Linux system as stable as possible. While Arch Developers and Trusted Users work hard to produce high quality packages, given Arch's rolling release system and rapid package turnover, an Arch system may not be suitable for a mission critical, commercial production environment.
However, Arch is inherently stable due to its commitment to simplicity in configuration, coupled with a rapid bug-report/bug-fix cycle, and the use of unpatched upstream source code. Thus, by following the advice below on setting up and maintaining Arch, the user should be able enjoy a very stable system. Furthermore, advice is included that will ease system repair in the event of a major malfunction.
How stable can Arch Linux really be? There are numerous reports in the Arch forums of skilled system administrators successfully using Arch for production servers. Archlinux.org is one such example. On the desktop, a properly configured and maintained Arch installation can offer excellent stability.
Setting Up Arch
When first installing and configuring Arch Linux, the user has a variety of choices to make about configuration, software, and drivers. These choices will impact overall system stability.
Arch Specific Tips
Set Up a Large /var Partition and Keep Old Packages
When setting up partitions during installation, always be sure to allocate plenty of space for a large, separate /var partition. A /var partition should have a generous 6 to 8 GB of space - more for some server uses. Pacman archives all of the previously installed packages in /var/cache/pacman/pkg, which requires significant amounts of storage space. Retaining these packages is helpful if a recent package upgrade causes instability, requiring a downgrade to an older, archived package. See the section below entitled, Revert Package Upgrades That Cause Instability.
Activate as Few Modules and Daemons in rc.conf as Possible
During the initial configuration of an Arch system, use the fewest number of services possible. When editing the /etc/rc.conf file, only activate those modules and daemons that are absolutely necessary to run the system. The fewer services running, the less chance of instability problems caused by buggy software or corner-case software conflicts.
Use Recommended Configurations
In the detailed Arch Linux installation and configuration documentation, there is often more than one way to configure a specific aspect of the system. For example, there are several ways to configure a login manager to run during startup. Always choose the recommended, default configuration when setting up the system. (In the case of login manager configuration, this is the inittab method). The recommended, default configurations reflect best practices, chosen for optimum system stability and ease of system repair.
Select the Core and Extra Software Repositories
Edit the /etc/pacman.conf file to use only the core and extra repositories. These software repositories contain the most well developed and tested Arch packages. Information on how to do this is found in the Repositories section of the Pacman Arch wikipage.
If need be, also activate the community repository, but be aware that software from this repository might not be as mainstream nor as well tested and packaged as software from core and extra. Only use AUR and 3rd party repositories if absolutely necessary. Avoid any use of the testing repository, or individual packages from testing.
Use Up to Date Mirrors
Use mirrors that are frequently updated with the latest packages from the main Arch FTP server. Review the Arch Mirror Check webpage to verify that your chosen mirror is up to date. By using recently rsync'd mirrors, this ensures that your system will always have the freshest packages and package databases available during the course of routine maintenance.
Also, if it is used, edit the mirror list in /etc/pacman.d by placing local mirrors, those within your country or region, at the top of the list. Refer to the Enabling your favorite mirror Arch wikipage section for additional details, including the installation of the rankmirrors script to enable the fastest mirrors. These steps will ensure that the system uses the fastest, most reliable mirrors.
After changing the server mirror used for updates, to ensure that the mirror and local pacman databases are fully synchronized, always execute the following command:
Avoid Development Packages
To prevent serious breakage of the system, do not install any development packages, which are usually found in AUR and occasionally in community. These are packages taken directly from upstream development branches, and usually feature one of the following words appended to the package name: dev, devel, svn, cvs, git, hg, or darcs.
Most especially, avoid installing any development version of crucial system packages such as the kernel or glibc, such as those found in the testing or community repositories.
If building a custom package using makepkg, be sure that the PKGBUILD follows the Arch Packaging Standards, including a provides array. Use namcap to check the final .tar.gz or PKGBUILD file.
Consider Using yaourt to Simplify SysAdmin Tasks
Yaourt, which stands for Yet AnOther User Repository Tool, is a wrapper for pacman. Yaourt provides a variety of useful package management services, in addition to those provided by pacman. It uses the same command syntax as pacman, and invokes pacman to perform many functions.
Yaourt also simplifies several important SysAdmin tasks. After removing a package, yaourt immediately identifies orphan dependencies and offers to quickly remove them as well. Yaourt provides an easy way to make timestamped snapshot backups of the pacman database. Finally, yaourt provides the pacdiffviewer tool, which greatly simplifies finding and dealing with .pacnew and .pacsave files.
Information about installing yaourt is found on the yaourt Arch wikipage. Detailed usage instructions are available through the command:
Be aware that one of the functions of yaourt is to easily install packages from the AUR. AUR PKGBUILD files have not undergone the thorough vetting of the packages found in core and extra, and might not be as stable. Furthermore, AUR packages present potentially serious security risks. Yaourt executes user-submitted Bash scripts which may delete, by malicious design or by accident, any file to which the user has write permissions. Anything you can do in your user's Bash shell, the AUR package installation can do. However, Yaourt provides a means of reading through the PKGBUILD and *.install files of AUR packages prior to build and install. Doing so is highly recommended. Finally, it is extremely unwise to ever run yaourt or the manual tool makepkg as the root user.
Generic Best Practices
Minimize the Number of Installed Packages
Only install those software packages that are needed, and no more. For example, servers rarely require the installation and configuration of xorg and GUI software, so do not install them on server machines. Avoid installing multiple desktop environments or window managers. Do not install software packages that duplicate the functions of already installed software. Try not to mix GUI toolkits - stick to an all GTK or all QT system if possible. Keep the system spare and lean. The fewer the number of packages, and the simpler the selection of packages; the less chance of software bugs causing instability.
Use Proven, Mainstream Software Packages
Install mature, proven, mainstream software; while avoiding cutting edge software that is still buggy. Try to avoid installing "point-oh", aka x.y.0, software releases. For example, instead of installing Foobar 2.5.0, wait until Foobar 2.5.1 is available. Do not deploy newly developed software until it is proven to be reliable. For example, PulseAudio's early versions could be unreliable. Users interested in maximum stability would use the ALSA sound system instead. Finally, use software that has a strong and active development community.
Choose Open Source Drivers
Wherever possible, choose open source drivers. Try to avoid proprietary drivers. Most of the time, open source drivers are more stable and reliable than proprietary drivers. Open source driver bugs are fixed more easily and quickly. While proprietary drivers can offer more features and capabilities, this can come at the cost of stability. To avoid this dilemma, choose hardware components known to have mature open source driver support with full features. Information about hardware with open source Linux drivers is available at linux-drivers.org.
In addition to configuring Arch for stability, there are steps one can take during maintenance which will enhance stability. Paying attention to a few SysAdmin details will help to ensure continued system reliability.
Arch Specific Tips
Upgrade Entire System with Reasonable Frequency
Many Arch users update frequently, even upgrading their systems daily using pacman -Syu. While updating so frequently is not necessary, one should upgrade fairly often to enjoy the latest bugfix and security updates. Weekly or biweekly upgrades are thus a good idea.
If the system has packages from the AUR, upgrade all AUR packages with the following command:
yaourt -Syu --aur
Be aware that such an update can take considerably more time than a normal system upgrade invoked by using pacman -Syu or yaourt -Syu.
In the /etc/pacman.conf file, there is a section for listing packages to be ignored during upgrades. Uncomment the IgnorePkg line, and list the packages that should not be changed during upgrades. Refer to the Skip upgrade package section of the Pacman Arch wikipage for further details.
For example, when a new major kernel release comes out, such as 184.108.40.206, one might want to wait until the first point release, 220.127.116.11, before upgrading the kernel. In such a case, edit the pacman.conf file so that the IgnorePkg line reads as follows (be sure to include any necessary firmware kernel modules):
IgnorePkg = kernel26 kernel26-firmware
With proprietary video drivers, one might want to hold back updating the driver itself, as well as the kernel and xorg-server packages, until a new video driver compatible with the latest kernel and xorg-server packages is available.
Read Before Upgrading the System
Before upgrading Arch, always read the latest Arch News to find out if there are any major software or configuration changes with the latest packages. Before upgrading fundamental software, such as the kernel, xorg, or glibc to a new version; look over the appropriate webforum to see if there have been any reported problems.
Act on Alerts During an Upgrade
When upgrading the system, be sure to pay attention to the alert notices provided by pacman. If any additional actions are required by the user, be sure to take care of them right away. If a pacman alert is confusing, search the forums and the recent news posts for more detailed instructions.
Deal Promptly with .pacnew, .pacsave, and .pacorig Files
When pacman removes a package that has a configuration file, it normally creates a backup copy of that config file and appends .pacsave to the name of the file. Likewise, when pacman upgrades a package which includes a new config file created by the maintainer differing from the currently installed file, it writes a .pacnew config file. Occasionally, under special circumstances, a .pacorig file is created. Pacman provides notice when these files are written.
Users must deal with these files promptly when pacman creates them, in order to ensure optimum system stability. Users are referred to the Pacnew and Pacsave Files wiki page for detailed instructions.
There are various tools to help resolve .pacnew and .pacsave file issues. Yaourt provides a CLI program, pacdiffviewer, which assists with the proper resolution of .pacnew and .pacsave issues, offering an array of viewer/editor tools. The pacman-contrib package includes a tool, pacdiff, which helps to sort through such files. Finally, the pacnews bash script provides similar functionality. Both pacdiff and pacnews use vimdiff to compare and edit .pacnew and .pacsave files.
Consider Using Pacmatic
Pacmatic is a pacman wrapper which automates the process of checking Arch News prior to upgrading. Pacmatic also ensures that the local pacman database is correctly synchronized with online mirrors, thus avoiding potential problems with botched pacman -Sy database updates. Finally, it provides more stringent warnings about updated or obsolete config files. Pacmatic is available from the AUR. To use pacmatic with yaourt, edit the /etc/yaourtrc file so that the PacmanBin line reads:
Avoid Certain Pacman Commands
Never use pacman -Sy Package-Name or pacman -Syy Package-Name to install a package. This could cause synchronization errors between the local pacman database, and the online mirror's pacman database. The result will be a tedious system repair. Instead, use one of the two following commands:
pacman -S Package-Name pacman -Syu Package-Name
Avoid using the -f option with pacman, for example, pacman -Syuf or pacman -Uf. The --force option skips dependency and conflict checks and overwrites installed configuration files. In a properly maintained system, it should never need to be used.
Never run pacman -Scc unless there is a desparate need for the disk space, and little or no need for archived package files. It is safer to keep older packages available in the cache archives in the event a package upgrade causes problems, requiring a package reversion. Instead, just use the following command to clean out the archived packages in the pacman cache, of packages previously removed from the pacman database:
Make sure to only use this command if there is no intention of re-installing recently removed packages. If such packages are re-installed after this command has been executed, there will be no older, archived versions of the packages in the pacman cache.
In the event that /var disk space becomes scarce, move all archived packages to the home directory using the fdup-archpkg script. Move all but the last previously installed package versions in the pacman cache archives, to the home directory (this makes package reversion much easier), by using the fduppkg script.
Revert Package Upgrades That Cause Instability
In the event that a particular package upgrade results in system instability, remove the package and any of its dependencies, which might also be part of the problem. This is done by executing the command:
pacman -Rs Package-Name
Then, install the last known stable version of the package from the local pacman cache using the following command:
pacman -U /var/cache/pacman/pkg/Package-Name.pkg.tar.gz
If the package has dependencies, before installing the package; install the dependencies by name using the same command and archive as listed above. For more detailed information on reverting to older packages, consult the Arch wikipage, Downgrade packages.
Once the package is reverted, temporarily add it to the IgnorePkg section of pacman.conf, until the difficulty with the updated package is resolved. Consult the Arch wiki and/or webforums for advice, and file a bug report if necessary.
Regularly Purge Cruft
At regular intervals, review the activated modules and daemons in the /etc/rc.conf file, and remove those that are no longer needed.
Use the yaourt package manager to safely and automatically remove orphan packages, which it does as part of its routine use. With pacman, find and remove orphan packages no longer used as dependencies as follows:
sudo pacman -Rs $(pacman -Qtdq)
Use caution - this is a powerful command that can damage the system if improperly used.
Run the following command to clean out archived packages from /var/cache/pacman/pkg, which have been permanently removed from the system:
To optimize the pacman database for best speed, run the command:
pacman-optimize && sync
Regularly Backup a List of Installed Packages
At regular intervals, create a list of installed packages and store a copy on one or more offline media, such as a USB stick, external hard drive, or CD-R. Use the following command to create a pkglist:
pacman -Qqe | grep -vx "$(pacman -Qqm)" > /path/to/chosen/directory/pkg.list
In the event of a catastrophic system failure requiring a complete re-installation, these packages can be quickly reinstalled using the command:
xargs -a /path/to/chosen/directory/pkg.list pacman -S --needed*
Regularly Backup the Pacman Database
At regular intervals - possibly before each system upgrade, using yaourt, execute the following command to backup the local pacman database:
yaourt -B /path/to/chosen/directory/
Yaourt can be used to restore the backup pacman database file with the following command:
yaourt -B /path/to/chosen/directory/Name-of-Backup-File.tar.bz2
The following command will accomplish the same task, and can be run as a cronjob:
tar -cjf /path/to/chosen/directory/pacman-database.tar.bz2 /var/lib/pacman/local
Store the backup pacman database file on one or more offline media, such as a USB stick, external hard drive, or CD-R.
Restore the backup pacman database file by moving the pacman-database.tar.bz2 file into the /var/lib/pacman directory and executing the following command:
tar -xjvf pacman-database.tar.bz2
If the pacman database files are corrupted, and there is no backup file available, there exists some hope of rebuilding the pacman database. Consult the Arch wikipage, How To Restore Pacman's Local Database.
Generic Best Practices
Subscribe to NVD/CVE Alerts and Only Upgrade On a Security Alert
Subscribe to the Common Vulnerabilities and Exposure Security Alert updates, made available by National Vulnerability Database, and found on the NVD Download webpage. Only update the Arch system when a security alert is issued for a package installed on that particular system.
This is the alternative to upgrading the entire system frequently. It ensures that security problems in various packages are resolved promptly, while keeping all the rest of the packages frozen in a known, stable configuration. However, reviewing the frequent CVE Alerts to see if any apply to installed packages can be time consuming.
Always Backup Config Files Before Editing
Before editing any configuration file, always back up a known working version of that config file. In the event that changes in the config file cause problems, one can revert to the previous stable config file. Do this from a text editor by first saving the file to a backup copy before making any alterations; or execute the following command:
cp Config-File Config-File.bak
Regularly Backup the /home, /srv, and /var Directories
At regular intervals, backup the /home directory to an external hard drive, Network Attached Server, or online backup service. Occasionally verify the integrity of the backup process by comparing original files and directories with their backups.
Server installations should have the /srv directory regularly backed up. There may be additional directories in /var, such a /var/spool/mail or /var/lib which also require backup and occasional verification.
Regularly Backup the /etc Directory
At regular intervals, backup the /etc directory by executing the following command as root or as a cronjob:
tar -cjf /path/to/chosen/directory/etc-backup.tar.bz2 /etc
Store the /etc backup file on one or more offline media, such as a USB stick, external hard drive, or CD-R. Occasionally verify the integrity of the backup process by comparing original files and directories with their backups.
Restore corrupted /etc files by extracting the etc-backup.tar.bz2 file in a temporary working directory, and copying over individual files and directories as needed. To restore the entire /etc directory with all its contents, move the etc-backup.tar.bz2 files into the / directory. As root, execute the following command:
tar -xvjf etc-backup.tar.bz2
Test Updates On A Non-Critical System
If possible, test changes to configuration files as well as updates to software packages on a non-critical duplicate system first. Then, if no problems arise, roll out the changes to the production system.