Diskless network boot NBD root
- 1 Boot from a NBD root device
- 2 Advantages/Disadvantages over NFS
- 3 Server-Side Setup
- 4 Boot Configuration
- 5 Testing
- 6 Using a Swap Partition
- 7 Updating the client system
- 8 Known Problems
Boot from a NBD root device
This article will explain how to boot an ArchLinux Installation from a Network Block Device (NBD).
Much of the work to be done is based on the article Diskless network boot NFS root, so this will be referenced several times within the article.
Advantages/Disadvantages over NFS
- updating the kernel from a client involves more work
- read/write support on multiple clients is only possible with the "copy-on-write" mode, see Configuring the NBD server for details
- resuming from suspend is currently not possible
Create the NBD File and Boot Directory
Create a directory that will hold the boot directory and the NBD file.
mkdir -p /nbd/boot/
Next, create the actual file that will be shared via NBD. Of course you can also use an actual block device (a hard drive) instead of creating a file on your filesystem. Just replace
/nbd/root with the block device.
In this example we are going to create a file with a size of 5GB.
dd if=/dev/zero of=/nbd/root bs=1M count=5000
Now you can create a filesystem on the file.
mkfs will show you warning about the fact that the file is no actual block device. You can ignore this and simply press Template:Keypress to continue.
Alternatively, if you want to create an encrypted NBD device:
cryptsetup luksFormat -s 256 /nbd/root cryptsetup luksOpen /nbd/root nbdcrypt mkfs.ext4 /dev/mapper/nbdcrypt
Install ArchLinux on the NBD filesystem
Mount the filesystem:
mount /nbd/root /mnt
Create directories for the pacman database and a few mount points that will be needed:
mkdir -p /mnt/var/lib/pacman/ mkdir /mnt/proc/ mkdir /mnt/dev/ mkdir /mnt/sys/
dev. These will be neccessary to build the kernel image in the chroot environment:
mount -t proc none /mnt/proc mount -t sysfs none /mnt/sys mount -o bind /dev /mnt/dev
Install the Arch Linux base packages into the root directory:
pacman -Sy --root /mnt --dbpath /mnt/var/lib/pacman base
Build the AUR. Then install it into the root directory as well:AUR package from the
pacman -U --root /mnt --dbpath /mnt/var/lib/pacman mkinitcpio-nbd-[...]-any.pkg.tar.xz
Configuring the Client system
The following configurations are neccessary for the client system to work properly, everything else can be configured later:
NETWORK_PERSIST="yes"and the other client settings in
/mnt/etc/rc.conf. You will also have to remove/disable the
nethook will set the IP on boot, so there is no point in using it. Additionally, if you would change your IP with the
networkdaemon, the connection to the root device would break. If you want a static IP, use the
ipkernel parameter (see Boot Configuration).
- set a mirror in
- enable your locales in
- set your DNS server in
man 5 resolv.conffor details.
- add a line for
/dev/nbd0if your root device is encrypted):
/dev/nbd0 / ext4 defaults 0 0
building the kernel image
Set the following hook list in
HOOKS="base udev net nbd filesystems"
If you use an encrypted NBD device, use this:
HOOKS="base udev net nbd usbinput keymap encrypt filesystems"
Then you can chroot into
/mnt and rebuild the kernel image. Remember that you mounted
dev earlier. This is always neccessary if you want to rebuild the kernel image.
locale-gen is only neccessary this one time to set the locales:
chroot /mnt locale-gen mkinitcpio -p linux exit umount /mnt/dev umount /mnt/proc umount /mnt/sys
After leaving the chroot, the kernel image will be in
/mnt/boot/. We are going to need it in
cp /mnt/boot/vmlinuz-linux /nbd/boot/ cp /mnt/boot/initramfs-linux.img /nbd/boot/
Configuring the NBD server
nbd-server is configured with the config file
/etc/nbd-server/config and can be started with
rc.d start nbd. You may also want to add
nbd to your DAEMONS array.
A basic config file may look like this:
[generic] [nbdroot] exportname = /nbd/root
In this example, there is only a simple export with the name
nbdroot, which makes the file
/nbd/root available via NBD. The name of the export (
nbdroot in this case, not
/nbd/root!) is important because you will have to set it in the kernel boot parameters later.
By default, nbd-server will make the export available to clients with read/write support. If you have more than one client you want to boot from, this will break your filesystem (see Updating the Client System for details). You could enable read-only by setting
readonly = true, but this is not recommended because there are just too many things that want to write to
/etc or other directories. You could partially work around this with tmpfs, but some things might still break.
Another interesting option is NBD's "copy on write"-mode (
copyonwrite = true). From
man 5 nbd-server:
"[...] any writes to this export will not be written to the master file, but to a separate file which will be removed upon disconnect. The result of using this option is that nbd-server will be somewhat slower, and that any writes will be lost upon disconnect."
It has not been tested yet how much this really affects performance, but if you want to boot from multiple clients, this is probably your best option.
For a detailed explanation of the config file, see the aforementioned man page
man 5 nbd-server.
Install here. Just make sure you use
/nbd/boot/ instead of
/disklessroot/boot/ for the TFTP-Root.
pxelinux.0 boot file from syslinux to
/nbd/boot and create the
cp /usr/lib/syslinux/pxelinux.0 /nbd/boot/ mkdir /nbd/boot/pxelinux.cfg/
Now create and edit
/nbd/boot/pxelinux.cfg/default, which contains the boot configuration for the client. Replace the value for
nbd_host with the IP address of your NBD server and
nbd_name with the export name you have specificed in
/etc/nbd-server/config. If your nbd-server is not listening on the default port (10809), you can set it with
default linux label linux kernel vmlinuz-linux append initrd=initramfs-linux.img ip=::::::dhcp nbd_host=192.168.0.1 nbd_name=nbdroot root=/dev/nbd0
See here for details about the
If your NBD device is encrypted, use the following
append line instead:
append initrd=initramfs-linux.img ip=::::::dhcp nbd_host=192.168.0.1 nbd_name=nbdroot cryptdevice=/dev/nbd0:nbdcrypt root=/dev/mapper/nbdcrypt
Before you boot your client, make sure you unmount everything on the server:
umount /mnt/proc /mnt/sys /mnt/dev /mnt
If your NBD device is encrypted, close the LUKS device as well:
cryptsetup luksClose nbdcrypt
Using a Swap Partition
Although this has not been tested yet, you should be able to do this by creating a LVM volume group on
/dev/nbd0 that contains the root and swap partitions and adding
lvm2 before the
filesystems hook in
Updating the client system
Quote from the NBD homepage:
"[...] if someone has mounted NBD read/write, you must assure that no one else will have it mounted."
In other words, if you want to be able to update from your client system, you have to make sure that the NBD device is not mounted on any other system, not even read-only. If your NBD device is encrypted, make sure to not just unmount it, but also close it with
cryptsetup luksClose. Otherwise, you may break your filesystem. If you keep that in mind, everything except kernel updates should work fine.
Alternatively, you can mount the NBD device on the server and then update it (again, make sure it is not mounted anywhere else!).
sys should only be necessary for kernel upgrades, but just in case, they are included here.
mount /nbd/root /mnt mount -t proc none /mnt/proc mount -t sysfs none /mnt/sys mount -o bind /dev /mnt/dev chroot /mnt pacman -Syu exit umount /mnt/proc /mnt/sys /mnt/dev /mnt
Updating the kernel
Since the kernel the client system boots from is on the server and not on the NBD device itself, you will not be able to simply update your kernel with pacman. There are several other ways to do this:
- If you update from the client system:
- if you update from the server:
/nbd/bootafter the kernel update
Resume from Suspend causes I/O errors
When resuming the machine from suspend, the NBD connection drops and the root device is lost, causing I/O errors.
There is currently no fix or workaround for this. See patched nbd driver edition - https://github.com/katuma/mkinitcpio-nbd - it can suspend to remote NBD disk!