User:Bai-Chiang/Arch Linux installation with Bcachefs, unified kernel image (UKI), secure boot, and common setups

These notes are based on my previous ones User:Bai-Chiang/Arch Linux installation with unified kernel image (UKI), full disk encryption, secure boot, btrfs snapshots, and common setups.

Reader should already familiar the official installation guide. Also I will not give detailed explanation for each step, instead I will provide a link to corresponding wiki page for those interested.

Feature list:

bcachefs as root filesystem.
Unified kernel image boot directly from UEFI.
Secure boot with your own keys.
Optionally, systemd-homed (It could encrypts your home directory when system is suspended, but no session manager supports this feature yet, see systemd-homed#Forget key on suspend)
Optionally, SELinux for adventurous users (unofficial repository, see current status and issues)

Note I've created an installation script and Ansible playbooks to automate this setup. The script will bootstrap a base system. Then reboot into new system, and run Ansible playbooks to finish post installation configuration.

(The automation script and Ansible playbooks are not well tested with bcachefs.)

Pre-installation

Prepare an archiso with latest packages

On your existing Arch Linux machine install the archiso package. Copy the releng profile

$ mkdir -p /tmp/archiso-tmp/out
$ cp -r /usr/share/archiso/configs/releng/ /tmp/archiso-tmp/archlive

Add bcachefs-tools, bcachefs-dkms and linux-headers to packages.x86_64 if not exist.

$ echo "bcachefs-tools" >> /tmp/archiso-tmp/archlive/packages.x86_64
$ echo "bcachefs-dkms" >> /tmp/archiso-tmp/archlive/packages.x86_64
$ echo "linux-headers" >> /tmp/archiso-tmp/archlive/packages.x86_64

Create the image

# mkarchiso -v -w /tmp/archiso-tmp -o /tmp/archiso-tmp/out /tmp/archiso-tmp/archlive/

The ISO image will located /tmp/archiso-tmp/out/archlinux-YYYY.mm.dd-x86_64.iso

Put the firmware in "Setup Mode"

This step depends on BIOS implementation. First you need to enable UEFI boot only (no legacy boot). Then change Secure Boot option to Setup Mode. Usually it is under Secure Boot section, the exact name may be different, sometimes called Custom Mode. To check the Secure Boot is set to the Setup Mode, boot into the live iso. Then run

# bootctl status | grep "Secure Boot"

...
Secure Boot: disabled (setup)
...

It should have (setup).

UEFI Boot manager efibootmgr(8)

Display current boot settings

# efibootmgr --unicode

BootCurrent: 0004
BootNext: 0003
BootOrder: 0004,0000,0001,0002,0003
Timeout: 30 seconds
Boot0000* Diskette Drive(device:0)
Boot0001* CD-ROM Drive(device:FF)
Boot0002* Hard Drive(Device:80)/HD(Part1,Sig00112233)
Boot0003* PXE Boot: MAC(00D0B7C15D91)
Boot0004* Linux

Delete unused boot options. For example

# efibootmgr --bootnum 0004 --delete-bootnum --unicode

will delete boot entry 0004.

Set the console keyboard layout

# loadkeys us

Check UEFI mode

If booted as UFEI mode

# cat /sys/firmware/efi/fw_platform_size

should should return 64 or 32 without error.

Check internet connection

# ping archlinux.org

Recommend using wired connection during installation. Use a USB to Ethernet dongle if necessary.

Check system clock

# timedatectl

Partition the disks

Use fdisk create three partitions:

/dev/sdX1 1GiB EFI system partition.
/dev/sdX2 root partition.
(Optionally) /dev/sdX3 swap partition. Setup swap encryption later.

Format root partition

Enable zstd compression level 1

# bcachefs format --fs_label=ArchLinux --compression=zstd:1 /dev/sdX2

Encrypted root

# bcachefs format --fs_label=ArchLinux --encrypted /dev/sdX2

Mutiple drives RAID 0

# bcachefs format --fs_label=ArchLinux --data_replicas=1 --metadata_replicas=2 /dev/sdX2 /dev/sdY /dev/sdZ ...

Multiple drives with RAID 1 and tiered storage

# bcachefs format --fs_label=ArchLinux \
    --replicas=2 \
    --foreground_target=ssd \
    --promote_target=ssd \
    --metadata_target=ssd \
    --background_target=hdd \
    --label=ssd.ssd1 /dev/sdX2 \
    --label=ssd.ssd2 /dev/sdY \
    --label=hdd.hdd1 /dev/sdZ \
    --label=hdd.hdd2 /dev/sdA \
    ...

Tiered storage with two copies of data on hdd but only one on ssd (no redundancy for ssd)

# bcachefs format --fs_label=ArchLinux \
    --compression=zstd:1 \
    --encrypted \
    --data_replicas=2 \
    --metadata_replicas=2 \
    --foreground_target=ssd \
    --promote_target=ssd \
    --metadata_target=ssd \
    --background_target=hdd \
    --label=hdd.hdd1 /dev/sdY \
    --label=hdd.hdd2 /dev/sdZ \
    --durability=2 --discard --label=ssd.ssd1 /dev/sdX2

Format other the partitions

EFI partition
```
# mkfs.fat -n boot -F 32 /dev/sdX1
```
(Optionally) swap partition
```
# mkswap -L swap /dev/sdX3
```
swap encryption will be configured after arch-bootstrap.

Mount all file systems

Single drive

# bcachefs mount -o noatime /dev/sdX2 /mnt

Multiple drives

# bcachefs mount -o noatime /dev/sdX2:/dev/sdY:/dev/sdZ:... /mnt

Mount /efi partition with options fmask=0177,dmask=0077,noexec,nosuid,nodev, see Talk:EFI system partition#Mountpoint umask

# mkdir /mnt/efi
# mount -o fmask=0177,dmask=0077,noexec,nosuid,nodev /dev/sdX1 /mnt/efi

(Optionally) mount swap partition

# swapon /dev/sdX3

Installation

Add SELinux repository (optional)

Warning Unofficial repository use at your own risk.

Enable Unofficial user repositories#selinux by adding these lines to the end of /etc/pacman.conf

[selinux]
Server = https://github.com/archlinuxhardened/selinux/releases/download/ArchLinux-SELinux
SigLevel = PackageOptional

Install essential packages

Without SELinux:

# pacstrap -K /mnt base base-devel linux linux-firmware man-db vim dosfstools e2fsprogs bcachefs-tools bcachefs-dkms linux-headers cpu_manufacturer-ucode

If installing other kernels also add their headers like linux-zen-headers

With SELinux: replace base and base-devel with base-selinux and base-devel-selinux. Also add archlinux-keyring to the list:

# pacstrap -K /mnt base-selinux base-devel-selinux linux linux-firmware man-db vim dosfstools e2fsprogs bcachefs-tools bcachefs-dkms linux-headers archlinux-keyring cpu_manufacturer-ucode

The base-selinux^AUR depends selinux-refpolicy-arch^AUR, no need to install extra policy.

Replace cpu_manufacturer-ucode with amd-ucode or intel-ucode depends on the CPU manufacturer.

Configure the system

Fstab

Generate an fstab

# genfstab -U /mnt >> /mnt/etc/fstab

Chroot

# arch-chroot /mnt
# export PS1="(chroot) ${PS1}"

Time

(chroot) # ln -sf /usr/share/zoneinfo/Region/City /etc/localtime
(chroot) # hwclock --systohc

Localization

Uncomment en_US.UTF-8 UTF-8 in /etc/locale.gen, then

(chroot) # locale-gen

Create /etc/locale.conf and /etc/vconsole.conf

(chroot) # echo "LANG=en_US.UTF-8" > /etc/locale.conf
(chroot) # echo "KEYMAP=us" > /etc/vconsole.conf

Network configuration

Hostname

(chroot) # echo archlinux > /etc/hostname

Creating the /etc/resolv.conf symlink need to be done outside of chroot environment, see systemd-resolved#DNS.

(chroot) # exit
# ln -sf /run/systemd/resolve/stub-resolv.conf /mnt/etc/resolv.conf
# arch-chroot /mnt
# export PS1="(chroot) ${PS1}"

Choose only one network manager.

systemd-networkd is my preferred choice for devices that only need wired connection. iwd works well for WPA-Personal WiFi, but not easy to configure for the WPA-Enterprise WiFi.
NetworkManager with wpa_supplicant for devices need wireless connection.

systemd-networkd

Create configuration file

/etc/systemd/network/20-ethernet.network

[Match]
Name=en*
Name=eth*

[Network]
DHCP=yes
IPv6PrivacyExtensions=yes

Enable systemd-resolved.service and systemd-networkd.service.

NetworkManager

Install networkmanager and wpa_supplicant. Then enable systemd-resolved.service and NetworkManager.service.

Add bcachefs to initramfs

Edit /etc/mkinitcpio.conf, add bcachefs to the HOOKS and MODULES

/etc/mkinitcpio.conf

...
MODULES=(bcachefs)
...
HOOKS=(... bcachefs filesystems ...)
...

For using encrypted bcachefs root partition, remove systemd and sd-vconsole from HOOKS and add udev and consolefont to HOOKS. This will let mkinitcpio build busybox based initramfs. Bcachefs currently only support busybox based initramfs for encrypted root Bcachefs#Encrypted root filesystem.

/etc/mkinitcpio.conf

...
MODULES=(bcachefs)
...
HOOKS=(base systemd udev autodetect ... sd-vconsole consolefont ... bcachefs filesystems ...)
...

Might also need to add keyboard usbhid xhci_hcd i2c_hid_acpi modules to MODULES, and move keyboard hook before autodetect. Dm-crypt/System_configuration#Keyboard or keyfile on filesystem unavailable for unlocking

Swap encryption

To use UUID instead of /dev/sdX3 in the /etc/crypttab, we need to create a small 1MiB sized filesystem at /dev/sdX3, see dm-crypt/Swap encryption#UUID and LABEL. First deactivate swap partition

(chroot) # swapoff /dev/sdX3

Then create a 1MiB ext2 filesystem with label cryptswap

(chroot) # mkfs.ext2 -F -F -L cryptswap /dev/sdX3 1M

Add /etc/crypttab entry

/etc/crypttab

# <name>   <device>         <password>   <options>
cryptswap  UUID=SWAP_UUID  /dev/urandom  swap,offset=2048

Get SWAP_UUID with command lsblk -dno UUID /dev/sdX3. The option offset is the start offset in 512-byte sectors. So 2048 of 512 bytes is 1MiB.

Edit the swap entry in /etc/fstab, change UUID=xxxx to /dev/mapper/cryptswap

/etc/fstab

# <filesystem>    <dir>  <type>  <options>  <dump>  <pass>
/dev/mapper/swap  none   swap    defaults   0       0

zram

Install the zram-generator package. Then create

/etc/systemd/zram-generator.conf

[zram0]
zram-size = min(ram / 2, 4 * 1024)
compression-algorithm = zstd
fs-type = swap

zram-size is a function of ram variable, your total memory size. min(ram / 2, 4 * 1024) will create a zram as a swap device with size of half of your total memory, up to 4GB maximum. See zram-generator.conf(5) § OPTIONS for more supported arithmetic operators and functions.

Create following config file to optimize zram

/etc/sysctl.d/99-vm-zram-parameters.conf

vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0

Since officially supported kernels have zswap enabled by default. To disable it, we need to add zswap.enabled=0 to kernel parameters in the next step.

Unified kernel image

Kernel command line

Create /etc/kernel/cmdline file, which will contains kernel parameters for the unified kernel image. You can also create a fallback images that uses /etc/kernel/cmdline_fallback as kernel parameters.

/etc/kernel/cmdline

root=UUID=ROOT_UUID modprobe.blacklist=pcspkr zswap.enabled=0

/etc/kernel/cmdline_fallback

root=UUID=ROOT_UUID modprobe.blacklist=pcspkr zswap.enabled=0

You can get ROOT_UUID with command lsblk -dno UUID /dev/sdX2. modprobe.blacklist=pcspkr disable the PC speaker (or beeper) globally, see PC speaker#Disabling the PC speaker.

Note If you edited kernel parameters after the installation, do not forget to regenerate the initramfs. Then run sbctl sign-all to re-sign the unified kernel images.

If SELinux enabled also add lsm=landlock,lockdown,yama,integrity,selinux,bpf to the end.

Modify .preset file

Edit all .preset files for the kernels you installed, eg. linux.preset, linux-zen.preset, linux-lts.preset etc.

Optionally, add 'fallback' to PRESETS to create a fallback UKI.
Edit default_uki= (and fallback_uki=) to create UKI under path /efi/EFI/Linux/.
Comment out default_image= and fallback_imag=
Optionally, add Arch splash screen by appending --splash to default_options=

/etc/mkinitcpio.d/linux.preset

# mkinitcpio preset file for the 'linux' package

#ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
#default_image="/boot/initramfs-linux.img"
default_uki="/efi/EFI/Linux/arch-linux.efi"
default_options="--splash /usr/share/systemd/bootctl/splash-arch.bmp"

#fallback_config="/etc/mkinitcpio.conf"
#fallback_image="/boot/initramfs-linux-fallback.img"
fallback_uki="/efi/EFI/Linux/arch-linux-fallback.efi"
fallback_options="-S autodetect --cmdline /etc/kernel/cmdline_fallback"

This will generate unified kernel image arch-linux.efi and arch-linux-fallback.efi.

If installed linux-zen kernel then linux-zen.preset would looks like

/etc/mkinitcpio.d/linux-zen.preset

# mkinitcpio preset file for the 'linux-zen' package

#ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux-zen"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
#default_image="/boot/initramfs-linux-zen.img"
default_uki="/efi/EFI/Linux/arch-linux-zen.efi"
default_options="--splash /usr/share/systemd/bootctl/splash-arch.bmp"

#fallback_config="/etc/mkinitcpio.conf"
#fallback_image="/boot/initramfs-linux-zen-fallback.img"
fallback_uki="/efi/EFI/Linux/arch-linux-zen-fallback.efi"
fallback_options="-S autodetect --cmdline /etc/kernel/cmdline_fallback"

This will generate unified kernel image arch-linux-zen.efi and arch-linux-zen-fallback.efi.

Then create the /efi/EFI/Linux directory and regenerate the initramfs

(chroot) # mkdir -p /efi/EFI/Linux
(chroot) # mkinitcpio -P

and remove any leftover initramfs-*.img from /boot or /efi.

Note If your machine have faulty firmware that can't add boot entries at #UEFI boot entries step and you use only one kernel, you can set default_uki="/efi/EFI/BOOT/BOOTX64.EFI" in the .preset file. Then create /efi/EFI/BOOT directory and regenerate the initramfs. Then you can set to boot from SSD drive in your BIOS and it will boot this UKI without adding boot entries.

Secure boot

Install the sbctl package.

Create keys

(chroot) # sbctl create-keys

Enroll keys

Warning Some plug-in cards have firmware that's signed by Microsoft's keys. Such a card will not work (at least, not from the firmware) if you enable Secure Boot without the matching public key. (The card should still work once you've booted an OS. The biggest concern is for cards that must work in a pre-boot environment, such as a video card or for PXE-booting a network card.) You can add Microsoft's keys back to the environment to make such cards work, but this eliminates the advantages of not having those keys, if that's a significant factor for you.[1] Without Microsoft 3rd Party UEFI CA certificate these devices would not work before OS booted. Which could be a problem if your CPU does not have an integrated graphics card or need PXE-booting. If your CPU does not have integrated graphics card, and discrete graphics would not work before OS booted, then you won't be able to access BIOS. Similarly, network card need to work before OS for PXE-booting. For these cases you need to enroll Microsoft key.

If you want to enroll Microsoft key, or not sure about the warning message, run

(chroot) # sbctl enroll-keys --microsoft

If you are certain that you don't want to enroll Microsoft key (you may not be able to access BIOS) run

(chroot) # sbctl enroll-keys

Note

If sbctl says You need to chattr -i files in efivarfs, first run
```
(chroot) # chattr -i /sys/firmware/efi/efivars/{PK,KEK,db}*
```
then enroll key again.
If running libvirt virtual machine, you may add --yes-this-might-brick-my-machine to force enroll keys.

Sign unified kernel image

Sign all unified kernel images

(chroot) # sbctl sign --save /efi/EFI/Linux/arch-linux.efi
(chroot) # sbctl sign --save /efi/EFI/Linux/arch-linux-fallback.efi

If installed linux-zen, you also need to sign those images

(chroot) # sbctl sign --save /efi/EFI/Linux/arch-linux-zen.efi
(chroot) # sbctl sign --save /efi/EFI/Linux/arch-linux-zen-fallback.efi

This will replace old unsigned images with new signed ones.

Note If you install extra kernels after initial installation, do not forget to modify its .preset file and regenerate the initramfs. Then sign the unified kernel images and add UEFI boot entries. Finally run sbctl sign-all since you regenerated initramfs.

Note sbctl will install /usr/share/libalpm/hooks/zz-sbctl.hook. This hook will sign unified kernel image when it updated.

UEFI boot entries

Install efibootmgr.

For linux kernel

(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux" --loader 'EFI\Linux\arch-linux.efi' --unicode
(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux-fallback" --loader 'EFI\Linux\arch-linux-fallback.efi' --unicode

--disk is the disk containing boot loader do not include part number, it is /dev/sdX not /dev/sdX1. The --part specify the partition number. If the EFI partition is /dev/sdX2 then it is --disk /dev/sdX --part 2.

If you install linux-zen kernel also need to add them to boot entry, for example

(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux-zen" --loader 'EFI\Linux\arch-linux-zen.efi' --unicode
(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux-zen-fallback" --loader 'EFI\Linux\arch-linux-zen-fallback.efi' --unicode

Warning The last created entry will boot first. Check your current boot order with command efibootmgr.

Change the boot order, for example I want first boot entry 0003 then 0004, then 0005, run

(chroot) # efibootmgr --bootorder 0003,0004,0005 --unicode

Set root password

(chroot) # passwd

Reboot into BIOS

(chroot) # exit
# umount -R /mnt
# systemctl reboot --firmware-setup

Now you can enable Secure Boot (also called User Mode) in the BIOS. Some motherboard manufacture will automatically change to User Mode if you enrolled your own key.
Then set a BIOS password.
Now finger crossed and boot into your new system.

Post-installation

SELinux#Post-installation steps

Label your filesystem

# restorecon -r /
# systemctl reboot

Check SELinux Status:

# sestatus

It should be permissive mode. To temporary switch to enforcing mode

# echo 1 > /sys/fs/selinux/enforce

Or edit /etc/selinux/config to switch permanently.

Enable restorecond.service to maintain correct context.

Add user

Run command

# EDITOR=editor visudo

and uncomment line

%wheel ALL=(ALL:ALL) ALL

to allow users in wheel group run sudo.

Then add user tux member of wheel group with default shell /bin/bash.

# useradd -m -G wheel -s /bin/bash tux
# passwd tux

systemd-homed

From Lennart Poettering's blog:

It (systemd-homed) also allows us to correct another major issue with traditional Linux systems: the way how data encryption works during system suspend. Traditionally on Linux the disk encryption credentials (e.g. LUKS passphrase) is kept in memory also when the system is suspended. This is a bad choice for security, since many (most?) of us probably never turn off their laptop but suspend it instead. But if the decryption key is always present in unencrypted form during the suspended time, then it could potentially be read from there by a sufficiently equipped attacker.

In other words, my data is still safe even if I leave my laptop suspended in a hotel. In my humble opinion, systemd-homed does not provide much benefits for server or single user desktop use cases. Also it needs extra configuration if running SSH server, see systemd-homed#SSH remote unlocking.

Note Forgetting key on suspend feature is not supported by any session managers yet, see systemd-homed#Forget key on suspend.

Start and enable systemd-homed.service.

# homectl create tux-homed --uid=1001 --member-of=wheel --shell=/bin/bash --storage=luks --fs-type=btrfs

Will add user tux-homed with UID=1001 and a member of wheel group. Its home directory is a encrypted LUKS volume, and the filesystem is btrfs.

Disable root login

# passwd -d root
# passwd -l root

Time servers

/etc/systemd/timesyncd.conf

[Time]
NTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org
FallbackNTP=0.pool.ntp.org 1.pool.ntp.org 0.fr.pool.ntp.org

Then start and enable systemd-timesyncd.service

Pacman#Enabling parallel downloads

Uncomment ParallelDownloads in /etc/pacman.conf.

reflector

Install the reflector package.

/etc/xdg/reflector/reflector.conf

--save /etc/pacman.d/mirrorlist
--protocol https
--country us
--latest 5
--sort age

Enable reflector.service and reflector.timer.

Pacman#Cleaning the package cache

Install the pacman-contrib package. Enable paccache.timer.

Solid state drive#Periodic TRIM

Enable fstrim.timer.

Optimize AUR building

Remove any -march and -mtune CFLAGS, then add -march=native in /etc/makepkg.conf.

/etc/makepkg.conf

CFLAGS="-march=native -O2 -pipe ..."

Add -C target-cpu=native to RUSTFLAGS:

/etc/makepkg.conf

RUSTFLAGS="-C opt-level=2 -C target-cpu=native"

Enable parallel compilation

/etc/makepkg.conf

MAKEFLAGS="-j$(nproc)"

firewall

Install the firewalld package. Start and enable firewalld.service.

Restore dotfiles from a Git bare repository

Note To setup a Git bare repository for the first time, see this

Install the git package.

Clone your dotfiles repository

$ git clone --bare dotfiles-repo-url $HOME/.dotfiles
$ alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'

Checkout the repository

$ dotfiles checkout

It may show an error of conflicting files that already exist in your home directory would be overwritten. Since this is fresh installed system, we simply delete them. Or run this command then checkout again

$ dotfiles checkout 2>&1 | egrep "\s+\." | awk {'print $1'} | xargs -I{} rm {}
$ dotfiles checkout

If using GitHub change the remote url to use SSH key

~/.dotfiles/config

...
[remote "origin"]
    url = git@github.com:username/dotfiles.git
...

Install the openssh package or openssh-selinux^AUR if using SELinux. Generate new SSH keys#Ed25519 pairs

$ ssh-keygen -t ed25519

After installed Graphical user interface upload your new SSH key to GitHub. Now you can push to your Git repository with

$ dotfiles push

Zsh

Install the zsh, zsh-completions, zsh-syntax-highlighting, zsh-autosuggestions and grml-zsh-config packages, which provides the same setup as the Arch ISO release. To have Fish-like syntax highlighting and autosuggestions, add these lines to your .zshrc:

source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh

To make Zsh your default shell

$ chsh -s /usr/bin/zsh

or if you are using systemd-homed

# homectl update --shell=/usr/bin/zsh tux