User:Kayvlim/Install UEFI and BIOS compatible Arch Linux with Encrypted ZFS and ZFSBootMenu

This article is not officially supported.

The Arch Linux community does not offer support for the information contained in this page; for installation procedures, the Installation guide is the only officially supported document. The content below is mainly maintained by User:Kayvlim, who last reviewed it on 8 September 2021, and it may be out of date or inaccurate.

This article is under active construction and should not be used.

This is a work in progress (started in Aug 31 2021), and will probably be left unfinished for several days or weeks. Please wait until this message is removed.

Pre-installation

Follow the entire Installation guide#Pre-installation section up until "Partition the disks".

You should always prefer the most recent ISO, but should any issues arise, the latest ISO used with this article was the following:

FILE: archlinux-2021.12.01-x86_64.iso
MD5 : c8beaaea6dd5986d36dd532d1df4dc26
SHA1: e303b93788220dbe8e0ba1804c90beebe432cf63

Archive link: https://archive.archlinux.org/iso/2021.12.01/

Partition the disks

Constraints to keep in mind

UEFI...

requires GPT.
requires an EFI system partition (known as the esp).
- No smaller than 100MB. No larger than 1GB if you also want Windows to work on the same disk.
- Recommended size is 550MiB. Stick to this unless you know what you are doing.

Constraint 1: we must use GPT.
Constraint 2: we must create a ESP.

BIOS...

may require MBR, but GPT is already required by UEFI. For older BIOS systems that do not support GPT (therefore not supporting UEFI either), a hybrid MBR may be necessary, but this is unlikely to make sense unless you're building a broadly-compatible removable device.
- MBR can't address more than 2TiB.
- MBR has a limit of 4 primary partitions. A hybrid MBR can only "hybridize" 3 of these (the other being the EFI GPT partition), and none of them can span past the 2TiB mark if your disk is large enough.
- Since GPT will be used instead of MBR, there will be no post-MBR gap for GRUB's stage 1.5, so a BIOS boot partition would be required. Instead of juggling an EFI and a BIOS boot partition, we will be using SYSLINUX instead, stored inside the ESP, for BIOS booting.

Constraint 3: we may use a hybrid MBR instead of a protective MBR.
Constraint 4: we must keep our partition table under 4 partitions.
Constraint 5: we are limited to the first 2 TiB of the device; anything beyond that will be GPT-only.
Constraint 6: we will be using SYSLINUX for BIOS booting. If GRUB is preferred, a different partition structure, with a BIOS boot partition, probably within an extended partition, will be necessary.

ZFS...

is currently unable to hold a swap device - it deadlocks if a swap device resides within a ZFS pool (see OpenZFS issue #7734 and ZFS#Swap_volume).

Constraint 7: any swap partitions must not be a part of the zpool (until the issue is resolved).

Swap...

should not be necessary in a bootable removable device.
is elsewhere (especially on a laptop) useful for suspend-to-disk ("hibernate"). Make its size your RAM + 2GB.
- You will install more RAM in the future? Plan your partitioning accordingly: you won't have space to change this later. You can't shrink ZFS.

Keep in mind constraints 4 and 7: if swap is required, it must be its own partition, and partitions are limited to 3. That leaves us with ESP, SWAP and ZPOOL. If you require more partitions, they will not be a part of the hybrid MBR, so they will be missing when booting via BIOS.

Final layout

Mount point on the installed system	Partition	Partition type GUID	Size
`/efi`	`/dev/sdz1`	`C12A7328-F81F-11D2-BA4B-00A0C93EC93B`: EFI system partition	550 MiB
`[SWAP]`	`/dev/sdz2`	`0657FD6D-A4AB-43C4-84E5-0933C84B4F4F`: Linux swap	18 GiB
`[zpool]`	`/dev/sdz3`	`0FC63DAF-8483-4772-8E79-3D69D8477DE4`: Linux filesystem	Remainder of the device

Commands

Warning: These commands will repartition a whole disk. Existing partitions will be deleted. Identify the disk to repartition (and erase) and set the following variable accordingly:

# export DISK=/dev/sdz

In some cases (like when DISK=/dev/nvme0n1), any instructions below referring to partitions such as ${DISK}1 should be modified accordingly (to ${DISK}p1 for /dev/nvme0n1p1).

Create GPT label

# parted $DISK

(parted) mklabel gpt

Create each partition

(parted) mkpart "EFI System Partition" fat32 0% 550MiB
(parted) mkpart SWAP linux-swap 550MiB 18GiB
(parted) mkpart zpool 18GiB 100%

Set the correct type for each partition

(parted) set 1 boot on 
(parted) set 2 swap on

Build a hybrid MBR

Warning: Build a hybrid MBR only if you really know you need it. It makes sense for a removable drive that will need to work on several different machines. It does NOT make sense for a fixed drive on a UEFI/GPT-capable machine, even if you will be LEGACY or UEFI-CSM booting, since these modes can usually handle GPT just fine.

If your machine is BIOS-only, use MBR, not GPT with a hybrid MBR. If you plan on converting it to GPT later, just leave some room for the future ESP in free space so you won't spend a primary partition, but stay with MBR.

A lot can go wrong when using a hybrid MBR, especially if you later decide to modify your partition tables and forget to keep MBR and GPT in sync.

If you're really going down this road, make sure you have read Hybrid MBRs: The Good, the Bad, and the So Ugly You'll Tear Your Eyes Out before proceeding.

# gdisk $DISK

GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective            <--------------------------- "protective"
  BSD: not present
  APM: not present
  GPT: present

Command (? for help): r
Recovery/transformation command (? for help): h

Then select the partitions 1 2 3 with all defaults and bootable flag on the first partition only:

Recovery/transformation command (? for help): h

WARNING! Hybrid MBRs are flaky and dangerous! If you decide not to use one,
just hit the Enter key at the below prompt and your MBR partition table will
be untouched.

Type from one to three GPT partition numbers, separated by spaces, to be
added to the hybrid MBR, in sequence: 1 2 3
Place EFI GPT (0xEE) partition first in MBR (good for GRUB)? (Y/N): y <-------------- Your choice, which depends on whether
                                                                                      it's a removable device and whether
Creating entry for GPT partition #1 (MBR partition #2)                                you want to support Windows.
Enter an MBR hex code (default EF):
Set the bootable flag? (Y/N): y                                                       Make sure you have read the article
                                                                                      mentioned above.
Creating entry for GPT partition #2 (MBR partition #3)
Enter an MBR hex code (default 82):
Set the bootable flag? (Y/N): n

Creating entry for GPT partition #3 (MBR partition #4)
Enter an MBR hex code (default 83):
Set the bootable flag? (Y/N): n

Save and quit:

Recovery/transformation command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdz.
The operation has completed successfully.

Verify your partitions

None of these commands should be necessary, but their outputs are recorded here for eventual troubleshooting.

Note: Be wary of using parted on a disk with a hybrid MBR. parted will replace back the hybrid MBR with a protective MBR on write.

# parted $DISK "align-check opt "{1,2,3}
1 aligned
2 aligned
3 aligned

Then, open the gdisk command, which should mention the hybrid MBR if that's your case:

# gdisk $DISK

GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: hybrid            <--------------------------- "hybrid"
  BSD: not present
  APM: not present
  GPT: present

and confirm the ESP contains the GUID code C12A7328-F81F-11D2-BA4B-00A0C93EC93B:

Command (? for help): i
Partition number (1-3): 1
Partition GUID code: C12A7328-F81F-11D2-BA4B-00A0C93EC93B (EFI system partition)

Load the ZFS module on the archiso system

Alternative 1: the easy way

Warning: This is the EASY way, not the safest way. You should never curl | bash, especially as root (which is the case here). This is a shortcut. It might be good enough for you, but keep in mind that the script may be changed by the author at any time, for any reason. You should always inspect a script before running it.

# curl -s https://eoli3n.github.io/archzfs/init | bash

For more information, see eoli3n/archiso-zfs.

Format the filesystems

ESP

See: EFI_system_partition

# mkfs.fat -F32 ${DISK}1

Swap

See: Swap and Dm-crypt/Swap_encryption. See also Power_management/Suspend_and_hibernate if suspend-to-disk is desired.

This will create an encrypted swap device (the dd is simply to fill it with random data):

# cryptsetup luksFormat ${DISK}2
# cryptsetup luksOpen ${DISK}2 swapvol
# dd if=/dev/zero of=/dev/mapper/swapvol bs=8M status=progress
# mkswap /dev/mapper/swapvol

This passphrase will need to be typed every time the system boots. If suspend-to-disk is not required, better options exist - see the links above for more information.

ZFS Pool

Note: You are expected to understand ZFS, as well as understand the following commands, modify them adequately for your needs, and you need to make sure you are not using dynamic paths like /dev/sdz3. Use /dev/disk/by-id/ paths only.

Tip: We will not be using GRUB, so this zpool does not need to be GRUB-compatible. All features may be enabled unless you plan on having non-Linux systems access this zpool, in which case some features may not be supported by them - see warning below.

Find out the persistent name of the partition where the zpool will be stored. This is just an example, you should modify it appropriately for your setup:

# find /dev/disk/by-id/ -lname \*${DISK##*/}3\* -name 'ata-*'
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_1TB_X1X1X1X1X1X1X1X-part3

or

# find /dev/disk/by-id/ -lname \*${DISK##*/}3\* -name 'usb-*'
/dev/disk/by-id/usb-SanDisk_Ultra_Fit_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef12345678-0:0-part3

or for NVMe devices (where partitions end in p3 and multiple entries may show up)

# find /dev/disk/by-id/ -lname \*${DISK##*/}p3\* -name 'nvme-*'                 
/dev/disk/by-id/nvme-SK_hynix_XX000_XXX000XXXXXX-00A0A_AA0AAA00000000000-part3
/dev/disk/by-id/nvme-eui.xxx00x00000x00x0-part3

Either of these entries should be fine, since both are symlinks to ../../nvme0n1p3. In fact, the following should also work:

# find -L /dev/disk/by-id -samefile ${DISK}p3

Then run zpool-create:

Warning: Not all features available in Linux OpenZFS are compatible with OpenZFS in other operating systems. If you plan on running a non-Linux system such as FreeBSD or Illumos/OpenIndiana, or if you intend on accessing your zpool from such a system, you need to be especially mindful of the features you enable (see OpenZFS issue #9889 for an example regarding the userobj_accounting feature, disabled below).
Refer to the OpenZFS Feature Flags page to make an informed decision. The following command attempts to create a zpool compatible with FreeBSD, Illumos, and OS X, according to the feature flags implementation matrix, but was not tested in these operating systems.

zpool create -f                            \
       -o feature@userobj_accounting=disabled  \
       -o feature@bookmark_written=disabled    \
       -o feature@device_rebuild=disabled      \
       -o feature@draid=disabled               \
       -o feature@livelist=disabled            \
       -o feature@log_spacemap=disabled        \
       -o feature@redacted_datasets=disabled   \
       -o feature@redaction_bookmarks=disabled \
       -o feature@zstd_compress=disabled       \
       -o ashift=12                        \
       -O acltype=posixacl                 \
       -O relatime=off                     \
       -O xattr=sa                         \
       -O atime=off                        \
       -O dnodesize=legacy                 \
       -O normalization=formD              \
       -O mountpoint=none                  \
       -O canmount=off                     \
       -O devices=off                      \
       -O compression=lz4                  \
       -O encryption=aes-256-gcm           \
       -O keyformat=passphrase             \
       -O keylocation=prompt               \
       -R /mnt                             \
       yourzpoolname                       \
       /dev/disk/by-id/usb-SanDisk_Ultra_Fit_0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef12345678-0:0-part3

Choose a passphrase for your encrypted zpool:

Enter new passphrase:
Re-enter new passphrase:

And your zpool is ready to use.

Create the datasets

The factual accuracy of this article or section is disputed.

Reason: This structure for the datasets needs to be reviewed for both usage in boot environments and usage in multiple OS sharing the same zpool (Discuss in User talk:Kayvlim/Install UEFI and BIOS compatible Arch Linux with Encrypted ZFS and ZFSBootMenu)

The procedure will be similar to Install_Arch_Linux_on_ZFS#Create_your_datasets, but we will not be using GRUB and we will be choosing ZFSBootMenu to manage Boot Environments.

The specific datasets you create are up to you, but the ones used in this article follow:

Note: Keep in mind that ZFSBootMenu expects /boot to be present in the root dataset (that is, where mountpoint=/). Don't put /boot in a new dataset or in a different partition/zpool, or ZFSBootMenu will fail to identify the environment.

ZPOOL=yourzpoolname
ZFS_ROOT=arch

zfs create                        -o canmount=off    ${ZPOOL}/ROOT
zfs create -o mountpoint=/        -o canmount=noauto ${ZPOOL}/ROOT/${ZFS_ROOT}

zfs create                        -o canmount=off ${ZPOOL}/DATA
zfs create                        -o canmount=off ${ZPOOL}/DATA/${ZFS_ROOT}
zfs create -o mountpoint=legacy                   ${ZPOOL}/DATA/${ZFS_ROOT}/home
zfs create -o mountpoint=legacy                   ${ZPOOL}/DATA/${ZFS_ROOT}/root

zfs create                        -o canmount=off ${ZPOOL}/DATA/${ZFS_ROOT}/var
zfs create -o mountpoint=legacy                   ${ZPOOL}/DATA/${ZFS_ROOT}/var/log
zfs create                        -o canmount=off ${ZPOOL}/DATA/${ZFS_ROOT}/var/lib
zfs create -o mountpoint=legacy                   ${ZPOOL}/DATA/${ZFS_ROOT}/var/lib/libvirt
zfs create -o mountpoint=legacy                   ${ZPOOL}/DATA/${ZFS_ROOT}/var/lib/docker

zfs mount ${ZPOOL}/ROOT/${ZFS_ROOT}
zpool set bootfs=${ZPOOL}/ROOT/${ZFS_ROOT} ${ZPOOL}

Create the directory structure

Ensure that the pool's altroot was defined during zpool-create as -R /mnt, and the root dataset was mounted by zfs-mount in the previous step.

Create the directory structure for the mountpoints you created in the previous section (adjusting for any changes):

mkdir /mnt/home
mkdir /mnt/root
mkdir /mnt/var
mkdir /mnt/var/log
mkdir /mnt/var/lib
mkdir /mnt/var/lib/libvirt
mkdir /mnt/var/lib/docker

Mount the datasets

The mountpoints for these datasets are legacy for the future possibility of hosting more systems in this zpool. That means we need to mount them with mount instead of zfs-mount.

This will only be needed once. genfstab will ensure these mountpoints are persisted in /etc/fstab in future steps.

mount -t zfs ${ZPOOL}/DATA/${ZFS_ROOT}/home /mnt/home
mount -t zfs ${ZPOOL}/DATA/${ZFS_ROOT}/root /mnt/root
mount -t zfs ${ZPOOL}/DATA/${ZFS_ROOT}/var/log /mnt/var/log
mount -t zfs ${ZPOOL}/DATA/${ZFS_ROOT}/var/lib/libvirt /mnt/var/lib/libvirt
mount -t zfs ${ZPOOL}/DATA/${ZFS_ROOT}/var/lib/docker /mnt/var/lib/docker

Confirm the mounted datasets

# zfs mount

yourzpoolname/ROOT/arch                  /mnt
yourzpoolname/DATA/arch/home             /mnt/home
yourzpoolname/DATA/arch/root             /mnt/root
yourzpoolname/DATA/arch/var/log          /mnt/var/log
yourzpoolname/DATA/arch/var/lib/libvirt  /mnt/var/lib/libvirt
yourzpoolname/DATA/arch/var/lib/docker   /mnt/var/lib/docker

Installation

With the datasets created and mounted inside /mnt, you can continue following the Installation_guide#Installation and Installation_guide#Configure_the_system.

After you go through both of these sections of the Installation Guide, while still inside the new system chroot, you need to do the following:

Execute again the steps on section #Load the ZFS module on the archiso system. This is so your new system supports ZFS, not just the live archiso.
Edit the /etc/mkinitcpio.conf file and replace the HOOKS with HOOKS=(base udev autodetect modconf block keyboard zfs filesystems). See Install_Arch_Linux_on_ZFS#Install_and_configure_Arch_Linux for more information.
Regenerate your initramfs with mkinitcpio -P. Otherwise, your new system won't boot because the initramfs is not ZFS-aware.

Warning: If the regeneration of initramfs fails with errors like module not found: zfs, your kernel may be more recent than the kernel supported by ZFS. Read: #Downgrading_to_a_ZFS-compatible_kernel.

Install any tools you might need. For instance:

pacman -S alsa-utils base btrfs-progs clonezilla cryptsetup ddrescue dhclient dhcpcd diffutils dmraid dnsmasq dosfstools e2fsprogs edk2-shell efibootmgr ethtool exfatprogs gnu-netcat gpart gpm gptfdisk grml-zsh-config hdparm htop intel-ucode iw iwd jfsutils kitty-terminfo less lftp libfido2 libusb-compat lsscsi lvm2 lynx man-db man-pages mc mdadm memtest86+ mkinitcpio nano nbd ncdu ndisc6 nfs-utils nilfs-utils nmap ntfs-3g nvme-cli openconnect openssh openvpn partclone parted partimage pcsclite ppp pptpclient pv qemu-guest-agent reflector rsync rxvt-unicode-terminfo screen sdparm sg3_utils smartmontools sof-firmware squashfs-tools sudo syslinux systemd-resolvconf tcpdump terminus-font testdisk tmux tpm2-tss tree udftools usb_modeswitch usbmuxd usbutils vim vpnc wireless_tools wireless-regdb wpa_supplicant wvdial xfsprogs xl2tpd zsh

Downgrading to a ZFS-compatible kernel

Considering that ZFS is not officially supported by Linux and Arch Linux is a rolling distribution, you will often face the situation where the kernel in Arch moves faster than the latest kernel supported by OpenZFS. You will need to wait before you are able to upgrade your system to the most recent kernel version, but during installation you are installing the kernel for the first time, so you don't have that choice.

Instead, you will need to downgrade the new system you have just installed. To do so, refer to the instructions in Arch_Linux_Archive#How_to_restore_all_packages_to_a_specific_date, but keep reading:

This article was written using the ArchISO from 2021-09-01, which carries the kernel 5.13.13-arch1-1.

As of 2021-09-10, the latest kernel available is 5.14.2-arch1-2, as can be confirmed here: https://archive.archlinux.org/repos/2021/09/10/core/os/x86_64/ (notice the file linux-5.14.2.arch1-2-x86_64.pkg.tar.zst).

Scrolling back in time through the files in the Arch Archive, the last day with kernel version 5.13.13-arch1-1 was 2021-09-09: https://archive.archlinux.org/repos/2021/09/09/core/os/x86_64/ .

Therefore, in order to downgrade:

backup the file /etc/pacman.d/mirrorlist and create a new one with a single entry: Server=https://archive.archlinux.org/repos/2021/09/09/$repo/os/$arch.
update the pacman database and force downgrade: pacman -Syyuu

An example interaction follows:

[root@archiso /]# pacman -Syyuu
:: Synchronizing package databases...
 core                       135.9 KiB   496 KiB/s 00:00 [##############################] 100%
 extra                     1572.9 KiB  8.73 MiB/s 00:00 [##############################] 100%
 community                    5.8 MiB  22.8 MiB/s 00:00 [##############################] 100%
 archzfs                     14.1 KiB   151 KiB/s 00:00 [##############################] 100%
:: Starting full system upgrade...
warning: ca-certificates-mozilla: downgrading from version 3.70-1 to version 3.69.1-1
warning: edk2-shell: downgrading from version 202108-1 to version 202105-1
warning: fuse-common: downgrading from version 3.10.5-1 to version 3.10.4-1
warning: fuse3: downgrading from version 3.10.5-1 to version 3.10.4-1
warning: gdbm: downgrading from version 1.21-1 to version 1.20-1
warning: krb5: downgrading from version 1.19.2-1 to version 1.19.1-1
warning: libcap: downgrading from version 2.56-1 to version 2.53-1
warning: libedit: downgrading from version 20210714_3.1-1 to version 20210522_3.1-1
warning: libnsl: downgrading from version 2.0.0-1 to version 1.3.0-2
warning: libsamplerate: downgrading from version 0.2.2-1 to version 0.2.1-1
warning: liburing: downgrading from version 2.1-1 to version 2.0-1
warning: libxml2: downgrading from version 2.9.12-2 to version 2.9.10-9
warning: linux: downgrading from version 5.14.2.arch1-2 to version 5.13.13.arch1-1
warning: nss: downgrading from version 3.70-1 to version 3.69.1-1
warning: ntfs-3g: downgrading from version 2021.8.22-1 to version 2017.3.23-5
warning: pacman: downgrading from version 6.0.1-1 to version 6.0.0-5
warning: partclone: downgrading from version 0.3.17-2 to version 0.3.17-1
warning: python: downgrading from version 3.9.7-1 to version 3.9.6-1
warning: smbclient: downgrading from version 4.14.7-2 to version 4.14.7-1
warning: testdisk: downgrading from version 7.1-3 to version 7.1-2
warning: vim: downgrading from version 8.2.3412-1 to version 8.2.2891-1
warning: vim-runtime: downgrading from version 8.2.3412-1 to version 8.2.2891-1
resolving dependencies...
looking for conflicting packages...

Packages (22) ca-certificates-mozilla-3.69.1-1  edk2-shell-202105-1  fuse-common-3.10.4-1
              fuse3-3.10.4-1  gdbm-1.20-1  krb5-1.19.1-1  libcap-2.53-1
              libedit-20210522_3.1-1  libnsl-1.3.0-2  libsamplerate-0.2.1-1  liburing-2.0-1
              libxml2-2.9.10-9  linux-5.13.13.arch1-1  nss-3.69.1-1  ntfs-3g-2017.3.23-5
              pacman-6.0.0-5  partclone-0.3.17-1  python-3.9.6-1  smbclient-4.14.7-1
              testdisk-7.1-2  vim-8.2.2891-1  vim-runtime-8.2.2891-1

Total Download Size:   159.89 MiB
Total Installed Size:  282.03 MiB
Net Upgrade Size:       -2.27 MiB

:: Proceed with installation? [Y/n]

After downgrading, execute again the steps on section #Load the ZFS module on the archiso system, and then mkinitcpio -P should work with no ZFS-related issues.

Warning: Don't forget to bring back the file /etc/pacman.d/mirrorlist from backup. If you lost it, build a new one with reflector -n 20 > /etc/pacman.d/mirrorlist.

After you bring back the previous /etc/pacman.d/mirrorlist, notice that you can't upgrade the kernel:

[root@archiso /]# pacman -Syu
:: Synchronizing package databases...
 core                       136.1 KiB  1791 KiB/s 00:00 [##############################] 100%
 extra                     1587.8 KiB  23.5 MiB/s 00:00 [##############################] 100%
 community                    5.8 MiB  25.1 MiB/s 00:00 [##############################] 100%
 archzfs is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
error: failed to prepare transaction (could not satisfy dependencies)
:: installing linux (5.14.2.arch1-2) breaks dependency 'linux=5.13.13.arch1-1' required by zfs-linux

Install the bootloaders

rEFInd

It is often helpful to have rEFInd available while getting UEFI to work.

Install rEFInd:

# pacman -S refind

Install rEFInd to your ESP (replace /dev/sdXY with your ${DISK}1:

# refind-install --usedefault /dev/sdXY

ShimSource is none
Installing rEFInd on Linux....
Note: IA32 (x86) binary not installed!
Copied rEFInd binary files

Copying sample configuration file as refind.conf; edit this file to configure
rEFInd.

ZFSBootMenu

See: https://get.zfsbootmenu.org/

refind-install may have left the EFI mounted in /tmp/refind_install. Confirm it is there, or mount it yourself.

Then, create a new entry for ZFSBootMenu:

# cd /tmp/refind_install/EFI
# mkdir ZBM
# cd ZBM
# curl -O -L https://get.zfsbootmenu.org/zfsbootmenu.EFI

Syslinux

TODO: BIOS boot.