Diskless network boot NBD root
- 1 Boot from a NBD root device
- 2 Advantages over NFS
- 3 Server-Side Setup
- 4 Boot Configuration
- 5 Testing
- 6 Using a Swap Partition
- 7 Updating the client system
- 8 Known Problems
Boot from a NBD root device
This article will explain how to boot an ArchLinux Installation from a Network Block Device (NBD).
Much of the work to be done is based on the article Diskless network boot NFS root, so this will be referenced several times within the article.
Advantages over NFS
The main advantages are that NBD is faster and that you can boot from an encrypted or LVM-based NBD root device. One disadvantage is that you cannot easily update your kernel from within the running diskless client, although there is a workaround for this.
Thepackage will be needed later for the boot configuration.
Create the NBD File and Boot Directory
Create a directory that will hold the boot directory and the NBD file.
mkdir -p /nbd/boot/
Next, create the actual file that will be shared via NBD. Of course you can also use an actual block device (a hard drive) instead of creating a file on your filesystem. Just replace
/nbd/root with the block device.
In this example we are going to create a file with a size of 5GB.
dd if=/dev/zero of=/nbd/root bs=1M count=5000
Now you can create a filesystem on the file.
mkfs will show you warning about the fact that the file is no actual block device. You can ignore this and simply press Template:Keypress to continue.
Alternatively, if you want to create an encrypted NBD device:
cryptsetup luksFormat -s 256 /nbd/root cryptsetup luksOpen /nbd/root nbdcrypt mkfs.ext4 /dev/mapper/nbdcrypt
/nbd/root. If your NBD file is encrypted, replace it with
/dev/mapper/nbdcrypt, if you use an actual block device, with
Install ArchLinux on the NBD filesystem
Mount the filesystem:
mount /nbd/root /mnt
Now follow the instructions here, but be aware of three things:
- make sure you use
- you are going to have to install the AUR before recreating the kernel image (see below) AUR package from the
- the editing of
/mnt/etc/mkinitcpio.confis different for NBD (see below)
Download AUR, move the package to
/mnt and install it with:
chroot /mnt pacman -U mkinitcpio-nbd-[...].tar.xz exit
Set the following hook list in
HOOKS="base udev net nbd filesystems"
If you use an encrypted NBD device, use this:
HOOKS="base udev net nbd usbinput keymap encrypt filesystems"
Then continue with the instructions about recreating the kernel image in the NFS article.
After leaving the chroot, the kernel image will be in
/mnt/boot/. We are going to need it in
cp /mnt/boot/vmlinuz-linux /nbd/boot/ cp /mnt/boot/initramfs-linux.img /nbd/boot/
Make sure you set
NETWORK_PERSIST="yes" and your own settings in
/mnt/etc/rc.conf. You will also have to remove/disable the network daemon. The net hook will set the IP on boot, so there is no point in using it. Additionaly, if you would change your IP with the network daemon, the connection to the root device would break. If you want a static IP, use the ip kernel parameter (see Boot Configuration).
You should also edit
/mnt/etc/locale.gen and make sure your locales are enabled.
Configuring the NBD server
nbd-server is supposed to be configured with the config file
/etc/nbd-server/config, but this does not seem to work at the moment. Instead, you have to start NBD manually:
nbd-server -C " " 10809 /nbd/root
You can choose whatever port number you want, you will just have to set it right in the Boot Configuration. By default, nbd-server will start the share with read/write support. If you have more than one client you want to boot from, this will break your filesystem (see Updating the Client System for details).
You could enable read-only mode (
-r) but this is not recommended because there are just too many things that want to write to
/etc or other directories. You could partially work around this with tmpfs, but some things might still break.
Another interesting option is NBD's "copy on write"-mode (
-c). From the manual:
"When this option is provided, write-operations are not done to the exported file, but to a separate file. This separate file is removed when the connection is closed, which means that serving this way will make nbd-server slow down (especially on large block devices with lots of writes), and that after disconnecting and reconnecting the client or the server, all changes are lost."
It has not been tested yet how much this really affects performance, but if you want to boot from multiple clients, this is probably your best option.
Follow the instructions here. Just make sure you use
/nbd/boot/ instead of
/disklessroot/boot/ for the TFTP-Root.
pxelinux.0 boot file from syslinux to
/nbd/boot and create the
cp /usr/lib/syslinux/pxelinux.0 /nbd/boot/ mkdir /nbd/boot/pxelinux.cfg/
Now create and edit
/nbd/boot/pxelinux.cfg/default, which contains the boot configuration for the client. Replace the value for
nbd_server with the IP address and port your NBD server will be listening on.
See here for details about the
If your NBD device is encrypted, use the following
append line instead:
append initrd=initramfs-linux.img ip=::::::dhcp nbd_server=192.168.0.1:10809 cryptdevice=/dev/nbd0:nbdcrypt root=/dev/mapper/nbdcrypt
Before you boot your client, make sure you unmount everything on the server:
umount /mnt/proc /mnt/sys /mnt/dev /mnt
If your NBD device is encrypted, close the LUKS device as well:
cryptsetup luksClose nbdcrypt
Using a Swap Partition
Although this has not been tested yet, you should be able to do this by creating a LVM volume group on
/dev/nbd0 that contains the root and swap partitions and adding
lvm2 before the
filesystems hook in
Updating the client system
Quote from the NBD homepage:
"[...] if someone has mounted NBD read/write, you must assure that no one else will have it mounted."
In other words, if you want to be able to update from your client system, you have to make sure that the NBD device is not mounted on any other system, not even read-only. If your NBD device is encrypted, make sure to not just unmount it, but also close it with
cryptsetup luksClose. Otherwise, you may break your filesystem. If you keep that in mind, everything except kernel updates should work fine.
Alternatively, you can mount the NBD device on the server and then update it (again, make sure it is not mounted anywhere else!).
sys should only necessary for kernel upgrades, but just in case, they are included here.
mount /nbd/root /mnt mount -t proc none /mnt/proc mount -t sysfs none /mnt/sys mount -o bind /dev /mnt/dev chroot /mnt pacman -Syu exit umount /mnt/proc /mnt/sys /mnt/dev /mnt
Updating the kernel
Since the kernel the client system boots from is on the server and not in the NBD device itself, you will not be able to simply update your kernel with pacman. There are several other ways to do this:
- If you update from the client system:
/nbd/booton the server via NFS, mount it to
/bootand then update your kernel
/bootback to the server to
/nbd/bootafter the kernel update
- if you update from the server:
/nbd/bootafter the kernel update
Resume from Suspend causes I/O errors
When resuming the machine from suspend, the NBD connection drops and the root device is lost, causing I/O errors. There is currently no fix or workaround for this.