User:Bai-Chiang/Arch Linux installation with Bcachefs, unified kernel image (UKI), secure boot, and common setups

These notes are based on my previous ones User:Bai-Chiang/Arch Linux installation with unified kernel image (UKI), full disk encryption, secure boot, btrfs snapshots, and common setups.

Reader should already familiar the official installation guide. Also I will not give detailed explanation for each step, instead I will provide a link to corresponding wiki page for those interested.

Feature list:

bcachefs as root filesystem. (Currently I only get single drive and no encryption setup works. Other setups like multiple drives or encryption would drop to emergency shell.)
Unified kernel image boot directly from UEFI.
Secure boot with your own keys.
Optionally, systemd-homed (It could encrypts your home directory when system is suspended, but no session manager supports this feature yet, see systemd-homed#Forget key on suspend)
Optionally, SELinux for adventurous users (unofficial repository, see current status and issues)

Note: I've created an installation script and Ansible playbooks to automate this setup. The script will bootstrap a base system. Then reboot into new system, and run Ansible playbooks to finish post installation configuration.

(The automation script and Ansible playbooks are not well tested with bcachefs.)

Pre-installation

Prepare an archiso with Linux 6.7 kernel

Current latest ISO archlinux-2024.01.01-x86_64.iso only ships Linux 6.6 kernel, so we need to build our own.

On your existing Arch Linux machine install the archiso package. Then run

# mkdir -p /tmp/archiso-tmp/out
# mkarchiso -v -w /tmp/archiso-tmp -o /tmp/archiso-tmp/out /usr/share/archiso/configs/releng/

The ISO image will located /tmp/archiso-tmp/out/archlinux-2024.01.15-x86_64.iso

Put the firmware in "Setup Mode"

This step depends on BIOS implementation. First you need to enable UEFI boot only (no legacy boot). Then change Secure Boot option to Setup Mode. Usually it is under Secure Boot section, the exact name may be different, sometimes called Custom Mode. To check the Secure Boot is set to the Setup Mode, boot into the live iso. Then run

# bootctl status | grep "Secure Boot"

...
Secure Boot: disabled (setup)
...

It should have (setup).

UEFI Boot manager efibootmgr(8)

Display current boot settings

# efibootmgr

BootCurrent: 0004
BootNext: 0003
BootOrder: 0004,0000,0001,0002,0003
Timeout: 30 seconds
Boot0000* Diskette Drive(device:0)
Boot0001* CD-ROM Drive(device:FF)
Boot0002* Hard Drive(Device:80)/HD(Part1,Sig00112233)
Boot0003* PXE Boot: MAC(00D0B7C15D91)
Boot0004* Linux

Delete unused boot options. For example

# efibootmgr -b 0004 -B

will delete boot entry 0004.

Set the console keyboard layout

# loadkeys us

Check UEFI mode

If booted as UFEI mode

# ls /sys/firmware/efi/efivars

should should print the directory without error.

Check internet connection

# ping archlinux.org

Recommend using wired connection during installation. Use a USB to Ethernet dongle if necessary.

Check system clock

# timedatectl

Partition the disks

Use fdisk create three partitions:

/dev/sdX1 1GiB EFI system partition.
/dev/sdX2 root partition.
(Optionally) /dev/sdX3 swap partition. Setup swap encryption later.

Format root partition

Enable zstd compression

# bcachefs format --fs_label=ArchLinux --compression=zstd /dev/sdX2

Warning: I could not get all following examples to work. If anyone works it out please add solution to the bcachefs wiki page.

Encrypted root

# bcachefs format --fs_label=ArchLinux --encrypted /dev/sdX2

Mutiple drives RAID 0

# bcachefs format --fs_label=ArchLinux --data_replicas=1 --metadata_replicas=2 /dev/sdX2 /dev/sdY /dev/sdZ ...

Multiple drives with RAID 1 and tiered storage

# bcachefs format --fs_label=ArchLinux \
    --replicas=2 \
    --foreground_target=ssd \
    --promote_target=ssd \
    --metadata_target=ssd \
    --background_target=hdd \
    --label=ssd.ssd1 /dev/sdX2 \
    --label=ssd.ssd2 /dev/sdY \
    --label=hdd.hdd1 /dev/sdZ \
    --label=hdd.hdd2 /dev/sdA \
    ...

Mount all file systems

# mkfs.fat -n boot -F 32 /dev/sdX1
# mkswap -L swap /dev/sdX3

Single drive

# mount -t bcachefs /dev/sdX2 /mnt

Multiple drives

# mount -t bcachefs /dev/sdX2:/dev/sdY:/dev/sdZ:... /mnt
# mkdir /mnt/efi
# mount /dev/sdX1 /mnt/efi
# swapon /dev/sdX3

Installation

Add SELinux repository (optional)

Warning: Unofficial repository use at your own risk.

Enable Unofficial user repositories#selinux by adding these lines to the end of /etc/pacman.conf

[selinux]
Server = https://github.com/archlinuxhardened/selinux/releases/download/ArchLinux-SELinux
SigLevel = PackageOptional

Install essential packages

Without SELinux:

# pacstrap -K /mnt base base-devel linux linux-firmware man-db vim dosfstools e2fsprogs btrfs-progs cpu_manufacturer-ucode

With SELinux: replace base and base-devel with base-selinux and base-devel-selinux. Also add archlinux-keyring to the list:

# pacstrap -K /mnt base-selinux base-devel-selinux linux linux-firmware man-db vim dosfstools e2fsprogs btrfs-progs archlinux-keyring cpu_manufacturer-ucode

The base-selinux^AUR depends selinux-refpolicy-arch^AUR, no need to install extra policy.

Replace cpu_manufacturer-ucode with amd-ucode or intel-ucode depends on the CPU manufacturer.

Configure the system

Fstab

Generate an fstab

# genfstab -U /mnt >> /mnt/etc/fstab

Chroot

# arch-chroot /mnt
# export PS1="(chroot) ${PS1}"

Time

(chroot) # ln -sf /usr/share/zoneinfo/Region/City /etc/localtime
(chroot) # hwclock --systohc

Localization

Uncomment en_US.UTF-8 UTF-8 in /etc/locale.gen, then

(chroot) # locale-gen

Create /etc/locale.conf and /etc/vconsole.conf

(chroot) # echo "LANG=en_US.UTF-8" > /etc/locale.conf
(chroot) # echo "KEYMAP=us" > /etc/vconsole.conf

Network configuration

Hostname

(chroot) # echo archlinux > /etc/hostname

Creating the /etc/resolv.conf symlink need to be done outside of chroot environment, see systemd-resolved#DNS.

(chroot) # exit
# ln -sf /run/systemd/resolve/stub-resolv.conf /mnt/etc/resolv.conf
# arch-chroot /mnt
# export PS1="(chroot) ${PS1}"

Choose only one network manager.

systemd-networkd is my preferred choice for devices that only need wired connection. iwd works well for WPA-Personal WiFi, but not easy to configure for the WPA-Enterprise WiFi.
NetworkManager with wpa_supplicant for devices need wireless connection.

systemd-networkd

Create configuration file

/etc/systemd/network/20-ethernet.network

[Match]
Name=en*
Name=eth*

[Network]
DHCP=yes
IPv6PrivacyExtensions=yes

Enable systemd-resolved.service and systemd-networkd.service.

NetworkManager

Install networkmanager and wpa_supplicant. Then enable systemd-resolved.service, NetworkManager.service and wpa_supplicant.service.

Swap encryption

To use UUID instead of /dev/sdX3 in the /etc/crypttab, we need to create a small 1MiB sized filesystem at /dev/sdX3, see dm-crypt/Swap encryption#UUID and LABEL. First deactivate swap partition

(chroot) # swapoff /dev/sdX3

Then create a 1MiB ext2 filesystem with label cryptswap

(chroot) # mkfs.ext2 -F -F -L cryptswap /dev/sdX3 1M

Add /etc/crypttab entry

/etc/crypttab

# <name>   <device>         <password>   <options>
cryptswap  UUID=SWAP_UUID  /dev/urandom  swap,offset=2048

Get SWAP_UUID with command lsblk -dno UUID /dev/sdX3. The option offset is the start offset in 512-byte sectors. So 2048 of 512 bytes is 1MiB.

Edit the swap entry in /etc/fstab, change UUID=xxxx to /dev/mapper/cryptswap

/etc/fstab

# <filesystem>    <dir>  <type>  <options>  <dump>  <pass>
/dev/mapper/swap  none   swap    defaults   0       0

zram

Install the zram-generator package. Then create

/etc/systemd/zram-generator.conf

[zram0]
zram-size = min(ram / 2, 4 * 1024)
compression-algorithm = zstd
fs-type = swap

zram-size is a function of ram variable, your total memory size. min(ram / 2, 4 * 1024) will create a zram as a swap device with size of half of your total memory, up to 4GB maximum. See zram-generator.conf(5) § OPTIONS for more supported arithmetic operators and functions.

Since officially supported kernels have zswap enabled by default. To disable it, we need to add zswap.enabled=0 to kernel parameters in the next step.

Unified kernel image

Kernel command line

Create /etc/kernel/cmdline and /etc/kernel/cmdline_fallback file, which will contains kernel parameters for the unified kernel image. Fallback images would use cmdline_fallback as kernel parameters. You could have different kernel parameters for fallback images.

/etc/kernel/cmdline

root=UUID=ROOT_UUID modprobe.blacklist=pcspkr zswap.enabled=0

/etc/kernel/cmdline_fallback

root=UUID=ROOT_UUID modprobe.blacklist=pcspkr zswap.enabled=0

You can get ROOT_UUID with command lsblk -dno UUID /dev/sdX2. modprobe.blacklist=pcspkr disable the PC speaker (or beeper) globally, see PC speaker#Disabling the PC speaker.

Note: If you edited kernel parameters after the installation, do not forget to regenerate the initramfs. Then run sbctl sign-all to re-sign the unified kernel images.

If SELinux enabled also add lsm=landlock,lockdown,yama,integrity,selinux,bpf to the end.

Modify .preset file

Edit all .preset files for the kernels you installed, eg. linux.preset, linux-zen.preset, linux-lts.preset etc.

Add line ALL_microcode=(/boot/*-ucode.img)
Add default_uki= and fallback_uki=
Optionally, comment out default_image= and fallback_imag=
Optionally, add Arch splash screen by appending --splash to default_options=

/etc/mkinitcpio.d/linux.preset

# mkinitcpio preset file for the 'linux' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux"
ALL_microcode=(/boot/*-ucode.img)

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
#default_image="/boot/initramfs-linux.img"
default_options="--splash /usr/share/systemd/bootctl/splash-arch.bmp"
default_uki="/efi/EFI/Linux/Archlinux-linux.efi"

#fallback_config="/etc/mkinitcpio.conf"
#fallback_image="/boot/initramfs-linux-fallback.img"
fallback_options="-S autodetect --cmdline /etc/kernel/cmdline_fallback"
fallback_uki="/efi/EFI/Linux/Archlinux-linux-fallback.efi"

If installed linux-zen kernel then linux-zen.preset would looks like

/etc/mkinitcpio.d/linux-zen.preset

# mkinitcpio preset file for the 'linux-zen' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux-zen"
ALL_microcode=(/boot/*-ucode.img)

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
#default_image="/boot/initramfs-linux-zen.img"
default_options="--splash /usr/share/systemd/bootctl/splash-arch.bmp"
default_uki="/efi/EFI/Linux/Archlinux-linux-zen.efi"

#fallback_config="/etc/mkinitcpio.conf"
#fallback_image="/boot/initramfs-linux-zen-fallback.img"
fallback_options="-S autodetect --cmdline /etc/kernel/cmdline_fallback"
fallback_uki="/efi/EFI/Linux/Archlinux-linux-zen-fallback.efi"

This will generate unified kernel image Archlinux-linux.efi.

Then create the /efi/EFI/Linux directory and regenerate the initramfs

(chroot) # mkdir -p /efi/EFI/Linux
(chroot) # mkinitcpio -P

and remove any leftover initramfs-*.img from /boot or /efi.

Secure boot

Install the sbctl package.

Create keys

(chroot) # sbctl create-keys

Enroll keys

Warning: Some plug-in cards have firmware that's signed by Microsoft's keys. Such a card will not work (at least, not from the firmware) if you enable Secure Boot without the matching public key. (The card should still work once you've booted an OS. The biggest concern is for cards that must work in a pre-boot environment, such as a video card or for PXE-booting a network card.) You can add Microsoft's keys back to the environment to make such cards work, but this eliminates the advantages of not having those keys, if that's a significant factor for you.[1] Without Microsoft 3rd Party UEFI CA certificate these devices would not work before OS booted. Which could be a problem if your CPU does not have an integrated graphics card or need PXE-booting. If your CPU does not have integrated graphics card, and discrete graphics would not work before OS booted, then you won't be able to access BIOS. Similarly, network card need to work before OS for PXE-booting. For these cases you need to enroll Microsoft key.

If you want to enroll Microsoft key, or not sure about the warning message, run

(chroot) # sbctl enroll-keys --microsoft

If you are certain that you don't want to enroll Microsoft key (you may not be able to access BIOS) run

(chroot) # sbctl enroll-keys

Note:

If sbctl says You need to chattr -i files in efivarfs, first run
```
(chroot) # chattr -i /sys/firmware/efi/efivars/{PK,KEK,db}*
```
then enroll key again.
If running libvirt virtual machine, you may add --yes-this-might-brick-my-machine to force enroll keys.

Sign unified kernel image

Sign all unified kernel images

(chroot) # sbctl sign --save /efi/EFI/Linux/ArchLinux-linux.efi
(chroot) # sbctl sign --save /efi/EFI/Linux/ArchLinux-linux-fallback.efi

If installed linux-zen, you also need to sign those images

(chroot) # sbctl sign --save /efi/EFI/Linux/ArchLinux-linux-zen.efi
(chroot) # sbctl sign --save /efi/EFI/Linux/ArchLinux-linux-zen-fallback.efi

This will replace old unsigned images with new signed ones.

Note: If you install extra kernels after initial installation, do not forget to modify its .preset file and regenerate the initramfs. Then sign the unified kernel images and add UEFI boot entries. Finally run sbctl sign-all since you regenerated initramfs.

Note: sbctl will install /usr/share/libalpm/hooks/zz-sbctl.hook. This hook will sign unified kernel image when it updated.

UEFI boot entries

Install efibootmgr.

For linux kernel

(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux" --loader 'EFI\Linux\ArchLinux-linux.efi'
(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux-fallback" --loader 'EFI\Linux\ArchLinux-linux-fallback.efi'

--disk is the disk containing boot loader do not include part number, it is /dev/sdX not /dev/sdX1. The --part specify the partition number. If the EFI partition is /dev/sdX2 then it is --disk /dev/sdX --part 2.

If you install linux-zen kernel also need to add them to boot entry, for example

(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux-zen" --loader 'EFI\Linux\ArchLinux-linux-zen.efi'
(chroot) # efibootmgr --create --disk /dev/sdX --part 1 --label "ArchLinux-linux-zen-fallback" --loader 'EFI\Linux\ArchLinux-linux-zen-fallback.efi'

Warning: The last created entry will boot first. Check your current boot order with command efibootmgr.

Change the boot order, for example I want first boot entry 0003 then 0004, then 0005, run

(chroot) # efibootmgr --bootorder 0003,0004,0005

Set root password

(chroot) # passwd

Reboot into BIOS

(chroot) # exit
# umount -R /mnt
# systemctl reboot --firmware-setup

Now you can enable Secure Boot (also called User Mode) in the BIOS. Some motherboard manufacture will automatically change to User Mode if you enrolled your own key.
Then set a BIOS password.
Now finger crossed and boot into your new system.

Post-installation

SELinux#Post-installation steps

Label your filesystem

# restorecon -r /
# systemctl reboot

Check SELinux Status:

# sestatus

It should be permissive mode. To temporary switch to enforcing mode

# echo 1 > /sys/fs/selinux/enforce

Or edit /etc/selinux/config to switch permanently.

Enable restorecond.service to maintain correct context.

Add user

Run command

# EDITOR=editor visudo

and uncomment line

%wheel ALL=(ALL:ALL) ALL

to allow users in wheel group run sudo.

Then add user tux member of wheel group with default shell /bin/bash.

# useradd -m -G wheel -s /bin/bash tux
# passwd tux

systemd-homed

From Lennart Poettering's blog:

It (systemd-homed) also allows us to correct another major issue with traditional Linux systems: the way how data encryption works during system suspend. Traditionally on Linux the disk encryption credentials (e.g. LUKS passphrase) is kept in memory also when the system is suspended. This is a bad choice for security, since many (most?) of us probably never turn off their laptop but suspend it instead. But if the decryption key is always present in unencrypted form during the suspended time, then it could potentially be read from there by a sufficiently equipped attacker.

In other words, my data is still safe even if I leave my laptop suspended in a hotel. In my humble opinion, systemd-homed does not provide much benefits for server or single user desktop use cases. Also it needs extra configuration if running SSH server, see systemd-homed#SSH remote unlocking.

Note: Forgetting key on suspend feature is not supported by any session managers yet, see systemd-homed#Forget key on suspend.

Start and enable systemd-homed.service.

# homectl create tux-homed --uid=1001 --member-of=wheel --shell=/bin/bash --storage=luks --fs-type=btrfs

Will add user tux-homed with UID=1001 and a member of wheel group. Its home directory is a encrypted LUKS volume, and the filesystem is btrfs.

Disable root login

# passwd -d root
# passwd -l root

Time servers

/etc/systemd/timesyncd.conf

[Time]
NTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org
FallbackNTP=0.pool.ntp.org 1.pool.ntp.org 0.fr.pool.ntp.org

Then start and enable systemd-timesyncd.service

Pacman#Enabling parallel downloads

Uncomment ParallelDownloads in /etc/pacman.conf.

reflector

Install the reflector package.

/etc/xdg/reflector/reflector.conf

--save /etc/pacman.d/mirrorlist
--protocol https
--country us
--latest 5
--sort age

Enable reflector.service and reflector.timer.

Pacman#Cleaning the package cache

Install the pacman-contrib package. Enable paccache.timer.

Solid state drive#Periodic TRIM

Enable fstrim.timer.

Optimize AUR building

Remove any -march and -mtune CFLAGS, then add -march=native in /etc/makepkg.conf.

/etc/makepkg.conf

CFLAGS="-march=native -O2 -pipe ..."

Add -C target-cpu=native to RUSTFLAGS:

/etc/makepkg.conf

RUSTFLAGS="-C opt-level=2 -C target-cpu=native"

Enable parallel compilation

/etc/makepkg.conf

MAKEFLAGS="-j$(nproc)"

firewall

Install the firewalld package. Start and enable firewalld.service.

Restore dotfiles from a Git bare repository

Note: To setup a Git bare repository for the first time, see this

Install the git package.

Clone your dotfiles repository

$ git clone --bare dotfiles-repo-url $HOME/.dotfiles
$ alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'

Checkout the repository

$ dotfiles checkout

It may show an error of conflicting files that already exist in your home directory would be overwritten. Since this is fresh installed system, we simply delete them. Or run this command then checkout again

$ dotfiles checkout 2>&1 | egrep "\s+\." | awk {'print $1'} | xargs -I{} rm {}
$ dotfiles checkout

If using GitHub change the remote url to use SSH key

~/.dotfiles/config

...
[remote "origin"]
    url = git@github.com:username/dotfiles.git
...

Install the openssh package or openssh-selinux^AUR if using SELinux. Generate new SSH keys#Ed25519 pairs

$ ssh-keygen -t ed25519

After installed Graphical user interface upload your new SSH key to GitHub. Now you can push to your Git repository with

$ dotfiles push

Zsh

Install the zsh, zsh-completions, zsh-syntax-highlighting, zsh-autosuggestions and grml-zsh-config packages, which provides the same setup as the Arch ISO release. To have Fish-like syntax highlighting and autosuggestions, add these lines to your .zshrc:

source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh

To make Zsh your default shell

$ chsh -s /usr/bin/zsh

or if you are using systemd-homed

# homectl update --shell=/usr/bin/zsh tux