From ArchWiki
Jump to: navigation, search

Using multiple hard drives, LVM & encryption


This section explains how to use multiple physical hard drives that should be encrypted an highly dynamic with LVM usage. Imagine the following scenarion: You have two or more phsyical drives that should be merged into a single volume group with LVM (and one or multiple logical volumes afterwards). Also you don't want these drives to be mounted on boot (so not a system drive).

Generally, you have two options on the order: either use LVM first and then encrypt the created drives, or use encryption first and then create the LVM inside. If you have a single phyiscal drive, it is easy to decide on encryption first, simply, because it is the most convenient way and you only need to enter / provide a passphrase or keyfile once. But if you have multiple physical drives, you could want to create the LVM first and then the encryption on that LVM drive. The problem: LVM is supposed to be dynamic and you destroy all of this dynamic by applying encryption: Since LVM does not understand the filesystem type, it cannot neither reduce nor extend space (and hence adding a new physical drive would be a very hard task).

So again it seems that encryption first is a good option but imagine having 10 harddrives that all need to be decrypted. Using a passphrase for each drive is not very convenient, because you have to enter it everytime (if you can't cache it or don't want to). Using a keyfile has the disadvantage of having an unencrypted file ready to decrypt every drive (rendering encryption useless). So the best approach is to have one (or multiple) keyfiles which are encrypted. While there are guides for this during boot-up nothing has been suggested for runtime usage.


Note: This guide uses a lot of commands from LVM and Dm-crypt_with_LUKS, so if anything problematic occurs, look at the articles for guidance.
Warning: I assume that you have read Dm-crypt_with_LUKS#Initial_Setup to get the hang of preparing your drive(s) for encryption. Do not just execute what is written here - understand it!

Since we want to use a keyfile, create one - we will use random binary data, but you can choose whatever you want (see Dm-crypt_with_LUKS#Using_LUKS_to_Format_Partitions_with_a_Keyfile):

dd if=/dev/random bs=512 count=4 | gpg -v --cipher-algo aes256 --digest-algo sha512 -c -a  > keyfile.gpg


Enter a passphrase and you will have an encrypted keyfile for your system encryption. I personally prefer /dev/random over dev/urandom for increased entropy, but you may use either one (just make sure you know the difference).

We will never want to have our keyfile lying around in an unencrypted way on any persistent, unencrypted storage, i.e. do not place it on the harddrive if it's not encrpyted. So there are two possible solution: Either have an encrypted drive (or partition) - you should be save there - or use the RAM to store your unencrypted passphrase via a ramdisk. Add the following to your /etc/fstab

none     /mnt/ramdisk     ramfs  defaults   0     0

You can modify the path and place it where you like. Now go ahead and create and mount your ramdisk:

mkdir /mnt/ramdisk && mount /mnt/ramdisk

Now you can decrypt your keyfile into this mountpoint.

Warning: Make sure it is actually mounted or your keyfile will be place on your root partition!
Note: Do not use either /run/ or /tmp or any other tmpfs as they can be moved to swap (which may be unencrypted), giving you a security risk!
gpg -d -o /mnt/ramdisk/keyfile keyfile.gpg

Now you can take a look at your keyfile to ensure it is filled with random data and nothing went wrong. This file will now be used to encrypt all of your drives (Yes, we will use a single file for all drives but feel free to adjust this to one file for each drive). Simply create an encrypted partition as usual:

cryptsetup -c aes-xts-plain -s 512 luksFormat /dev/sdb1 /mnt/ramdisk/keyfile

replace /dev/sdb1 with the drive which is to be encrypted. Execute this step for any drive you want to include. Next we want to make them all available to us:

cryptsetup --key-file /mnt/ramdisk/keyfile luksOpen /dev/sdb1 drive1

Choose any name you like (replace drive1) but I recommend to just increment numbers if the purpose is similar. You will not make contact with those names in the final solution, since you will have LVM work with them in the next step. Now all your encrypted drives are open and you can start with the creation of LVM as you like. I will assume a simple merge of all available drives into one big logical volume. To do this add all drives as physical volumes:

pvcreate /dev/mapper/drive1

Repeat for every drive you want to include (all those drives you encrypted and opened above). Then issue:

vgcreate VolGroup00 /dev/mapper/drive1

And issue the following for every following drive you want to add:

vgextend VolGroup00 /dev/mapper/drive2

The name VolGroup00 can be chosen in any way you like. For example if you are creating a multimedia drive, name it media. We will later name our logical group as well, so you may want to read on before choosing a name, if this is important to you. Our next step is to create our large logical volume:

lvcreate -l +100%FREE VolGroup00 -n media

In this example I named my new drive media giving us two paths to our new drive: /dev/mapper/VolGroup00-media and {ic|/dev/VolGroup00/media}}. If you adjusted your name for the volume group you need to adjust these paths and the command as well.

Note: While /dev/media does not exist on my system, it is a very common name and might lead to conflicts, so you maybe want to choose a more arbitray name.

Now that our new logical volume is created we can proceed and create the filesystem with

mkfs.ext3 /dev/VolGroup00/media

Depending on the size of the drive this may take some time. Now you can mount your new drive, for example:

mount /dev/VolGroup00/media /mnt/media

This finishes the setup phase but since half of those commands are setup commands, you might want to automate the process.


As mentioned in the introduction, we do not want to mount our system on boot (for me it is a headless machine so I can't enter anything during boot), so we will write a script to do all the work for us. Here is what we will need to do:

  • Mount ramdisk (only if you specifiec it as noauto in /etc/fstab
  • Decrypt keyfile to ramdisk
  • Open encrypted volumes with luksOpen
  • Discover all LVM drives (since they were encrypted, LVM does not know about them yet)
  • Mount all LVM drives in the fashion desired (e.g. to /mnt/media
  • Delete the keyfile
  • Execute possible actions afterwards (e.g. start a program that serves or processes your data)

A sample script could look like this:

 #!/bin/sh -e

declare -ar DRIVES=('/dev/sdb1'
if ! grep -qs "' $MOUNTPOINT '" /proc/mounts; then
    let i=1
    while (($i<=${#DRIVES[@]})); do
        if [ ! -L "/dev/mapper/$DRIVEPREFIX$i" ]; then
            if [ ! -f $KEYFILE ]; then
                gpg -o $KEYFILE $KEYFILE_GPG
            cryptsetup --key-file $KEYFILE luksOpen ${DRIVES[$((i-1))]} "$DRIVEPREFIX$i"
            echo "Device ${DRVIVES[$((i-1))]} already mapped to $DRIVEPREFIX$i..."
        let i++
	vgchange -ay
	echo "Already mounted, nothing to do..."
if [ -f $KEYFILE ]; then
	echo "Passphrase already deleted... That is good!"

Note: Since the drive is not always mounted, either make sure that no software automatically will want to write to it or regularly check the unmounted path of your drive so nothing got written to it (this can lead to ugly full root partitions).

Multi-Threading & Performance in dm-crypt

While dm-crypt does support multi-threading(REF HERE), this is a different notion than a user might expect. The original non-multi-threaded version meant that each process that does IO had to go through the same dm-crypt process that performed the de- and encryption. That meant that this would be a bottleneck for parallel IO operations. The improved version with multi-threading means that this bottleneck is removed by allowing multiple dm-crypt threads or processes such that parallel IO operations do not get slowed down by a single dm-crypt process. However, a single process doing IO still only gets a single dm-crypt process. So, for example, running something like rsync on a large data set would still result in de- or encryption going only through a single process.

This only proves itself as a problem when a single core is slower when doing encryption than the hard drive has read or write speed. This becomes painfully obvious when using an SSD, because their speeds range from 200-600 MB/s (or when using RAID0/5/etc.) and the CPU does not support the AES-NI instructions (old & slow CPUs and even some new laptops may be missing it). The solution to this problem would be to allow dm-crypt to spawn multiple threads