Installing with Software RAID or LVM
Note: This is currently a work in progress! I will delete this comment when it is no longer applicable -- that is, once I have tested it through at least once fully. At this point, many parts have been tested, but new parts recently added may not be exactly correct yet.
- 1 Disclaimer
- 2 General Approach
- 3 Get the Arch Installer CD
- 4 Outline
- 5 Procedure
- 5.1 Boot the Installer CD
- 5.2 Partition the Hard Drives
- 5.3 Create the RAID Redundant Partitions
- 5.4 Create and Mount the Main Filesystems
- 5.5 Setup LVM and Create the /home LVM Volume
- 5.6 Install and Configure Arch
- 5.7 Install Grub on the Primary Hard Drive
- 5.8 Unmount Filesystems and Reboot
- 5.9 Install Grub on the Alternate Boot Drives
- 5.10 Archive your Filesystem Partition Scheme
- 5.11 Install the mdadm Tool
- 6 Conclusion
- 7 Credits
Installing a system with RAID is a complex process. Anything could go wrong. You could make a mistake, I could make a mistake, there could be a bug in something. Backup all your data first. Make sure only the drives involved in the installation are attached while doing the install. You've been warned!
For starters, note that this document seeks primarily to give you a good example walkthrough of how to install Arch with Software RAID or LVM support for a typical case. It won't try to explain all the possible things you can do -- it's more to give you an example of something that will work that you can then tweak to your own purposes.
In this example, the machine I'm using will have three similar IDE hard drives, at least 80GB each in size, installed as primary master, primary slave, and secondary master, with my installation CD-ROM drive as the secondary slave. I will assume these can be reached as /dev/discs/disc0, /dev/discs/disc1, and /dev/discs/disc2, and that the cdrom drive is /dev/cdroms/cdrom0.
To make this easy, I won't be trying to put the root filesystem on lvm. I don't know if this is possible now (it wasn't when I started setting up lvm systems), but I generally think it to be a complicating factor, and I don't like complicating factors in my boot processes.
We'll create a 100MB /boot partition, a 2048MB (2GB) swap partition, a 40GB root partition, and leave the rest of the drivespace for LVM. The boot and swap partitions will be RAID 1, while the root and LVM partitions will be RAID5. Why RAID 1? For boot, it's so you can boot the kernel from grub (which has no RAID drivers!), and for swap, it's for speed.
Each RAID 1 redundant partition will have three physical partitions, all the same size, one on each of the drives. The total storage capacity will be the size of a single one of these physical partitions. A RAID 1 redundant partition with 3 physical partitions can lose any two of its physical partitions and still function.
Each RAID 5 redundant partition will also have three physical partitions, all the same size, one on each of the drives. The total storage capacity will be the combined size of two of these physical partitions, with the third drive being consumed to provide parity information. A RAID 5 redundant partition with 3 physical partitions can lose any one of its physical partitions and still function.
In this example, we'll create the /home partition in LVM, and mount it prior to install.
Get the Arch Installer CD
Please note that in order to use LVM, you need the
dev-mapper packages installed, otherwise you won't be able to see any LVM partitions on reboot, until you install those packages. Note that the Arch 0.7 Base installer does not contain these packages, but the Arch 0.7 Full installer does. So if you're going to use LVM, you'll need to download the bigger ISO. My example will describe you using the Full installer; the changes should be minimal if you wish to use the Base installer instead.
Just to give you an idea of how all this will work, I'll outline the steps. The details for these will be filled in below.
- Boot the Installer CD
- Partition the Hard Drives
- Create the RAID Redundant Partitions
- Create and Mount the Main Filesystems
- Setup LVM and Create the /home LVM Volume
- Install and Configure Arch
- Install Grub on the Primary Hard Drive
- Unmount Filesystems and Reboot
- Install Grub on the Alternate Boot Drives
- Archive your Filesystem Partition Scheme
- Install the mdadm Tool
Boot the Installer CD
First, load all your drives in the machine. Then boot the Arch Linux 0.7 Full installation CD.
At the syslinux boot prompt, hit enter: we want to use the SCSI kernel, which has support for RAID and LVM built in.
So far, this is easy. Don't worry, it gets harder.
Partition the Hard Drives
cfdisk to do this partitioning. We want to create 4 partitions on each of the three drive:
Partition 1 (/boot): 100MB, type FD, bootable
Partition 2 (swap): 2048MB, type FD
Partition 3 (/): 20000MB, type FD
Partition 4 (LVM): <Rest of the drive>, type FD
Note that in general, in
cfdisk, you can use the first letter of each
Bracketed Option to select it; however, this is not true for the
Write command, presumably for safety's sake, and so you have to use the arrow keys to select it.
# cfdisk /dev/discs/disc0/disc
Create each partition in order:
- Hit Enter to make it a
- Type the appropriate size (in MB), or for Partition 4, just hit enter to select the remainder of the drive.
- Hit Enter to choose to place the partition at the
Type, hit enter to see the second page of the list, and then type
fdfor the Linux RAID Autodetect type.
- For Partition 1 on each drive, select
- Hit down arrow (selecting the remaining freespace) to go on to the next partition to be created.
When you're done, select
Write, and confirm
y-e-s that you want to write the partition information to disk.
Repeat this for the other two drives:
# cfdisk /dev/discs/disc1/disc # cfdisk /dev/discs/disc2/disc
Create the same exact partitions on each disk. If a group of partitions of different sizes are assembled to create a redundant RAID partition, it will work, but the redundant partition will be in multiples of the size of the smallest one, leaving the rest of the allocated drive space to waste.
Create the RAID Redundant Partitions
Now that you've created all the physical partitions, you're ready to set up RAID. The tool you use to create RAID arrays is
To create /dev/md/0 (/boot):
# mdadm --create /dev/md/0 --level=1 --raid-devices=3 /dev/discs/disc0/part1
To create /dev/md/1 (swap):
# mdadm --create /dev/md/1 --level=1 --raid-devices=3 /dev/discs/disc0/part2
To create /dev/md/2 (/):
# mdadm --create /dev/md/2 --level=5 --raid-devices</code>3 /dev/discs/disc0/part3
To create /dev/md/3 (LVM):
# mdadm --create /dev/md/3 --level=5 --raid-devices=3 /dev/discs/disc0/part4
At this point, you should have working RAID partitions. When you create the RAID partitions, they need to sync themselves so the contents of all three physical partitions are the same on all three drives. The hard drives lights will come on as they try to sync up. You can monitor the progress by typing:
# cat /proc/mdstat
You can also get particular information about, say, the root partition by typing:
# mdadm --misc --detail /dev/md/2
You don't have to wait for synchronization to finish -- you may proceed with the installation while syncronization is still occurring. You can even reboot at the end of the installation with synchronization still going.
Create and Mount the Main Filesystems
I like Reiser (3.x), so I use it for almost everything. GRUB supports it for booting, and it handles small files well. It's about as well tested as EXT3. You can choose other types if you wish.
To create /boot:
# mkreiserfs /dev/md/0
To create swap space:
# mkswap /dev/md/1
To create /:
# mkreiserfs /dev/md/2
Now, mount the boot and root partitions where the installer expects them:
# mount /dev/md/2 /mnt # mkdir /mnt/boot # mount /dev/md/0 /mnt/boot
Setup LVM and Create the /home LVM Volume
First, you have to create and mount a sysfs partition on /sys, to keep lvm from getting cranky. If you forget to do this, instead of giving you an intelligent error message, lvm will simply Segmentation fault at various inconvenient times. Be warned!
To mount the sysfs partition, do:
# mkdir /sys # mount -t sysfs none /sys
Now you need to do is tell LVM you have a Physical Volume for it to use. It's really a virtual RAID volume (
/dev/md/3), but LVM doesn't know this, or really care. Do:
# lvm pvcreate /dev/md/3
LVM should report back that it has added the Physical Volume. You can confirm this with:
# lvm pvscan
Now it's time to create a Volume Group (which I'll call
array) which has control over the LVM Physical Volume we created. Do:
# lvm vgcreate array /dev/md/3
LVM should report that it has created the Volume Group
array. You can confirm this with:
# lvm vgscan
Next, we create a Logical Volume called
home in Volume Group
array which is 50GB in size:
# lvm lvcreate --size 50G --name home array
LVM should report that it created the Logical Volume
home. You can confirm this with:
# lvm lvscan
This Logical Volume will now be available as the device
/dev/array/home. We create a ReiserFS filesystem on it, and mount it:
# mkreiserfs /dev/array/home # mkdir /mnt/home # mount /dev/array/home /mnt/home
We've created all our filesystems! And we're ready to install the OS!
Install and Configure Arch
This section doesn't attempt to teach you all about the Arch Installer. It leaves out some details here and there for brevity, but still seeks to be basically follow-able. If you're having trouble with the installer, you may wish to seek help elsewhere in the Wiki or forums.
Here's the walkthrough:
/arch/setupto launch the main installer.
< OK >at the opening screen.
1 CD_ROMto install from CD-ROM.
- Now at the main menu, Select
2 Select Packagesand select all the packages in the base and lvm categories.
3 Install Packages. This will take a little while.
4 Install Kernel. Choose to install the
5 Configure System.
/etc/rc.conf. It should contain a
USELVM entry already, which you should change to:
Please Note: The
rc.sysconfig which parses the
USELVM variable entry will accept either
YES, however it will not accept mixed case. Please be sure you've got your capitalization correct.
Edit your Grub
/boot/grub/menu.lst so that your boot entry looks like:
# (0) Arch Linux title Arch Linux [[/boot/vmlinuz26]] root (hd0,0) kernel /vmlinuz26 root=/dev/md/2 ro
/etc/fstab to contain the entries:
/dev/md/2 / reiserfs defaults 0 0 /dev/md/1 swap swap defaults 0 0 /dev/md/0 /boot reiserfs defaults 0 0 /dev/array/home /home reiserfs defaults 0 0
At this point, make any other configuration changes you need to other files.
Then exit the configuration menu.
Since you will not be installing Grub from the installer, select
7 Exit Install to leave the installer program.
Install Grub on the Primary Hard Drive
This is the last and final step before you have a bootable system!
As an overview, the basic concept is to copy over the grub bootloader files into /boot/grub, mount a procfs and a devfs inside of /mnt, then chroot to /mnt so you're effectively inside your new system, and then run grub to install the bootloader in the boot area of your first hard drive.
# cp -a /mnt/usr/share/grub/i386-pc/* /mnt/boot/grub # sync # mount -t devfs none /mnt/dev # mount -t proc none /mnt/proc # chroot /mnt
At this point, you may no longer be able to see keys you type at your console. I'm not sure of the reason for this (NOTE: try "chroot /mnt /bin/<shell>"), but it you can fix it by typing
reset at the prompt.
Once you've got console echo back on, type:
After a short wait while grub does some looking around, it should come back with a grub prompt. Do:
grub> root (hd0,0) grub> setup (hd0) grub> quit
Unmount Filesystems and Reboot
The hard part is all over! Now just a few commands to unmount filesystems, and then you get to reboot into your new system:
sh-3.00# exit # cd / # umount /mnt/dev /mnt/proc /mnt/home /mnt/boot # umount /mnt <pre> Now remove the CD from your CD-ROM drive, and type: <pre> # reboot
Install Grub on the Alternate Boot Drives
Once you've successfully booted your new system for the first time, you will want to install Grub onto the other two disks so that, in the event of disk failure, the system can be booted from another drive. Log in to your new system as root and do:
# grub grub> device (hd0) /dev/discs/disc1/disc grub> root (hd0,0) grub> setup (hd0) grub> device (hd0) /dev/discs/disc2/disc grub> root (hd0,0) grub> setup (hd0) grub> exit
Archive your Filesystem Partition Scheme
Now that you're done, it's worth taking a second to archive off the partition state of each of your drives. This guarantees that it will be trivially easy to replace/rebuild a disk in the event that one fails. You do this with the
sfdisk tool and the following steps:
# mkdir /etc/partitions # sfdisk --dump /dev/discs/disc0/disc >/etc/partitions/disc0.partitions # sfdisk --dump /dev/discs/disc1/disc >/etc/partitions/disc1.partitions # sfdisk --dump /dev/discs/disc2/disc >/etc/partitions/disc2.partitions
Install the mdadm Tool
You should install the
mdadm package, which gives you facilities to monitor the health of your RAID partitions, as well as making changes or viewing the details of them.
You're done! I hope you've succeeded in setting up Arch Linux on your server with RAID and LVM!
This document was written by Paul Mattal with with significant help from others. Comments and suggestions are welcome at paul at archlinux dot org.
Thanks to all who have contributed information and suggestions! This includes:
- Carl Chave
- Guillaume Darbonne