Difference between revisions of "Diskless network boot NBD root"

From ArchWiki
Jump to: navigation, search
m (Boot Configuration: fix template)
(big rewrite)
Line 6: Line 6:
 
Much of the work to be done is based on the article [[Diskless network boot NFS root]], so this will be referenced several times within the article.
 
Much of the work to be done is based on the article [[Diskless network boot NFS root]], so this will be referenced several times within the article.
  
==Advantages over NFS==
+
==Advantages/Disadvantages over NFS==
The main advantages are that NBD is faster and that you can boot from an [[LUKS|encrypted]] or [[LVM]]-based NBD root device. One disadvantage is that you cannot easily update your kernel from within the running diskless client, although there is a [[Diskless network boot NBD root#Kernel Updates|workaround]] for this.
+
Advantages:
 +
* NBD is faster
 +
* allows you to boot from an [[LUKS|encrypted]] or [[LVM]]-based root device
  
==Server-Side Setup==
+
Disadvantages:
===Install syslinux===
+
* [[Diskless_network_boot_NBD_root#Updating_the_kernel|updating the kernel]] from a client involves more work
The {{Pkg|syslinux}} package will be needed later for the boot configuration.
+
* read/write support on multiple clients is only possible with the "copy-on-write" mode, see [[Diskless network boot NBD root#Configuring the NBD server|Configuring the NBD server]] for details
 +
* resuming from suspend is currently [[Diskless network boot NBD root#Resume from Suspend causes I/O errors|not possible]]
  
 +
==Server-Side Setup==
 
===Create the NBD File and Boot Directory===
 
===Create the NBD File and Boot Directory===
 
Create a directory that will hold the boot directory and the NBD file.
 
Create a directory that will hold the boot directory and the NBD file.
Line 38: Line 42:
 
  mount /nbd/root /mnt
 
  mount /nbd/root /mnt
  
Now follow the instructions [[Diskless network boot NFS root#Create Client Root Directory|here]], but be aware of three things:
+
Create directories for the [[pacman]] database and a few mount points that will be needed:
# make sure you use {{ic|/mnt}} instead of {{ic|/disklessroot}}
+
mkdir -p /mnt/var/lib/pacman/
# you are going to have to install the {{AUR|mkinitcpio-nbd}} package from the [[Arch User Repository|AUR]] before recreating the kernel image (see below)
+
mkdir /mnt/proc/
# the editing of {{ic|/mnt/etc/mkinitcpio.conf}} is different for NBD (see below)
+
mkdir /mnt/dev/
 +
mkdir /mnt/sys/
  
====Installing mkinitcpio-nbd====
+
Mount {{ic|proc}}, {{ic|sysfs}} and {{ic|dev}}. These will be neccessary to build the kernel image in the chroot environment:
Download {{AUR|mkinitcpio-nbd}} from the [[Arch User Repository|AUR]], move the package to {{ic|/mnt}} and install it with:
+
mount -t proc none /mnt/proc
chroot /mnt
+
mount -t sysfs none /mnt/sys
  pacman -U mkinitcpio-nbd-[...].tar.xz
+
mount -o bind /dev /mnt/dev
  exit
+
 
 +
Install the Arch Linux base packages into the root directory:
 +
pacman -Sy --root /mnt --dbpath /mnt/var/lib/pacman base
 +
 
 +
Build the {{AUR|mkinitcpio-nbd}} package from the [[Arch User Repository|AUR]]. Then install it into the root directory as well:
 +
  pacman -U --root /mnt --dbpath /mnt/var/lib/pacman mkinitcpio-nbd-[...]-any.pkg.tar.xz
 +
 
 +
====Configuring the Client system====
 +
 
 +
The following configurations are neccessary for the client system to work properly, everything else can be configured later:
 +
 
 +
* set {{ic|1=NETWORK_PERSIST="yes"}} and the other client settings in {{ic|/mnt/etc/[[rc.conf]]}}. You will also have to remove/disable the {{ic|network}} daemon. The {{ic|net}} hook will set the IP on boot, so there is no point in using it. Additionaly, if you would change your IP with the {{ic|network}} daemon, the connection to the root device would break. If you want a static IP, use the {{ic|ip}} kernel parameter (see [[Diskless network boot NBD root#Boot Configuration|Boot Configuration]]).
 +
* set a mirror in {{ic|/mnt/etc/pacman.d/mirrorlist}}.
 +
* enable your locales in {{ic|/mnt/etc/locale.gen}}.
 +
* set your DNS server in {{ic|/mnt/etc/resolv.conf}}, see {{ic|man 5 resolv.conf}} for details.
 +
* add a line for {{ic|/}} in {{ic|/etc/fstab}} (use {{ic|/dev/mapper/nbdcrypt}} instead of {{ic|/dev/nbd0}} if your root device is encrypted):
 +
  /dev/nbd0 / ext4 defaults 0 0
  
====Editing mkinitcpio.conf====
+
=====building the kernel image=====
 
Set the following hook list in {{ic|/mnt/etc/mkinitcpio.conf}}:
 
Set the following hook list in {{ic|/mnt/etc/mkinitcpio.conf}}:
 
  HOOKS="base udev net nbd filesystems"
 
  HOOKS="base udev net nbd filesystems"
Line 56: Line 77:
 
  HOOKS="base udev net nbd usbinput keymap encrypt filesystems"
 
  HOOKS="base udev net nbd usbinput keymap encrypt filesystems"
  
Then continue with the instructions about recreating the kernel image in the NFS article.
+
Then you can chroot into {{ic|/mnt}} and rebuild the kernel image. Remember that you mounted {{ic|proc}}, {{ic|sysfs}} and {{ic|dev}} earlier. This is always neccessary if you want to rebuild the kernel image. {{ic|locale-gen}} is only neccessary this one time to set the locales:
 +
chroot /mnt
 +
locale-gen
 +
mkinitcpio -p linux
 +
exit
 +
umount /mnt/dev
 +
umount /mnt/proc
 +
umount /mnt/sys
  
After leaving the chroot, the kernel image will be in {{ic|/mnt/boot/}}. We are going to need it in {{ic|/nbd/boot}}:
+
After leaving the chroot, the kernel image will be in {{ic|/mnt/boot/}}. We are going to need it in {{ic|/nbd/boot/}}:
 
  cp /mnt/boot/vmlinuz-linux /nbd/boot/
 
  cp /mnt/boot/vmlinuz-linux /nbd/boot/
 
  cp /mnt/boot/initramfs-linux.img /nbd/boot/
 
  cp /mnt/boot/initramfs-linux.img /nbd/boot/
 
====Editing rc.conf====
 
Make sure you set {{ic|1=NETWORK_PERSIST="yes"}} and your own settings in {{ic|/mnt/etc/[[rc.conf]]}}. You will also have to remove/disable the network daemon. The net hook will set the IP on boot, so there is no point in using it. Additionaly, if you would change your IP with the network daemon, the connection to the root device would break. If you want a static IP, use the ip kernel parameter (see [[Diskless network boot NBD root#Boot Configuration|Boot Configuration]]).
 
 
You should also edit {{ic|/mnt/etc/locale.gen}} and make sure your locales are enabled.
 
  
 
===Configuring the NBD server===
 
===Configuring the NBD server===
nbd-server is supposed to be configured with the config file {{ic|/etc/nbd-server/config}}, but [https://bbs.archlinux.org/viewtopic.php?id=121956 this does not seem to work] at the moment. Instead, you have to start NBD manually:
+
nbd-server is configured with the config file {{ic|/etc/nbd-server/config}}.  
nbd-server -C " " 10809 /nbd/root
+
 
+
You can choose whatever port number you want, you will just have to set it right in the [[Diskless network boot NBD root#Boot Configuration|Boot Configuration]].
+
By default, nbd-server will start the share with read/write support. If you have more than one client you want to boot from, this will break your filesystem (see [[Diskless network boot NBD root#Updating the Client System|Updating the Client System]] for details).
+
  
You could enable read-only mode ({{ic|-r}}) but this is not recommended because there are just too many things that want to write to {{ic|/var}}, {{ic|/etc}} or other directories. You could partially work around this with tmpfs, but some things might still break.
+
A basic config file may look like this:
 +
{{hc|/etc/nbd-server/config|<nowiki>
 +
[generic]
 +
[nbdroot]
 +
exportname = /nbd/root
 +
</nowiki>}}
 +
In this example, there is only a simple export with the name {{ic|nbdroot}}, which makes the file {{ic|/nbd/root}} available via NBD. The name of the export ({{ic|nbdroot}} in this case, not {{ic|/nbd/root}}!) is important because you will have to set it in the kernel boot parameters later.
 +
By default, nbd-server will make the export available to clients with read/write support. If you have more than one client you want to boot from, this will break your filesystem (see [[Diskless network boot NBD root#Updating the Client System|Updating the Client System]] for details). You could enable read-only by setting {{ic|1=readonly = true}}, but this is not recommended because there are just too many things that want to write to {{ic|/var}}, {{ic|/etc}} or other directories. You could partially work around this with tmpfs, but some things might still break.
  
Another interesting option is NBD's "copy on write"-mode ({{ic|-c}}). From the [http://linux.die.net/man/1/nbd-server manual]:
+
Another interesting option is NBD's "copy on write"-mode ({{ic|1=copyonwrite = true}}). From {{ic|man 5 nbd-server}}:
{{bc|"When this option is provided, write-operations are not done to the exported file, but to a separate file.  
+
{{bc|"[...] any writes to this export will not be written to the master file, but to a separate file which will be removed upon disconnect. The result of using this option is that nbd-server will be somewhat slower, and that any writes will be lost upon disconnect."}}
This separate file is removed when the connection is closed, which means that serving this way will make
+
nbd-server slow down (especially on large block devices with lots of writes), and that after disconnecting
+
and reconnecting the client or the server, all changes are lost."}}
+
  
 
It has not been tested yet how much this really affects performance, but if you want to boot from multiple clients, this is probably your best option.
 
It has not been tested yet how much this really affects performance, but if you want to boot from multiple clients, this is probably your best option.
  
 +
For a detailed explanation of the config file, see the aforementioned man page {{ic|man 5 nbd-server}}.
 
===PXE/TFTP Setup===
 
===PXE/TFTP Setup===
Follow the instructions [[Diskless network boot NFS root#PXE/TFTP Setup|here]]. Just make sure you use {{ic|/nbd/boot/}} instead of {{ic|/disklessroot/boot/}} for the TFTP-Root.
+
Install {{Pkg|syslinux}} and {{Pkg|dnsmasq}} on the server, then follow the instructions [[Diskless network boot NFS root#PXE/TFTP Setup|here]]. Just make sure you use {{ic|/nbd/boot/}} instead of {{ic|/disklessroot/boot/}} for the TFTP-Root.
  
 
==Boot Configuration==
 
==Boot Configuration==
Copy the {{ic|pxelinux.0}} boot file from syslinux to {{ic|/nbd/boot}} and create the {{ic|pxelinux.cfg}} directory:
+
Copy the {{ic|pxelinux.0}} boot file from [[syslinux]] to {{ic|/nbd/boot}} and create the {{ic|pxelinux.cfg}} directory:
 
  cp /usr/lib/syslinux/pxelinux.0 /nbd/boot/
 
  cp /usr/lib/syslinux/pxelinux.0 /nbd/boot/
 
  mkdir /nbd/boot/pxelinux.cfg/
 
  mkdir /nbd/boot/pxelinux.cfg/
  
Now create and edit {{ic|/nbd/boot/pxelinux.cfg/default}}, which contains the boot configuration for the client. Replace the value for {{ic|nbd_server}} with the IP address and port your NBD server will be listening on.
+
Now create and edit {{ic|/nbd/boot/pxelinux.cfg/default}}, which contains the boot configuration for the client. Replace the value for {{ic|nbd_host}} with the IP address of your NBD server and {{ic|nbd_name}} with the export name you have specificed in {{ic|/etc/nbd-server/config}}. If your nbd-server is not listening on the default port (10809), you can set it with {{ic|nbd_port}}.
 
{{hc|/nbd/boot/pxelinux.cfg/default|2=
 
{{hc|/nbd/boot/pxelinux.cfg/default|2=
 
default linux
 
default linux
Line 98: Line 122:
 
label linux
 
label linux
 
kernel vmlinuz-linux
 
kernel vmlinuz-linux
append initrd=initramfs-linux.img ip=::::::dhcp nbd_server=192.168.0.1:10809 root=/dev/nbd0
+
append initrd=initramfs-linux.img ip=::::::dhcp nbd_host=192.168.0.1 nbd_name=nbdroot root=/dev/nbd0
 
}}
 
}}
 
See [[Mkinitcpio#Using net|here]] for details about the {{ic|ip}} option.
 
See [[Mkinitcpio#Using net|here]] for details about the {{ic|ip}} option.
  
 
If your NBD device is encrypted, use the following {{ic|append}} line instead:
 
If your NBD device is encrypted, use the following {{ic|append}} line instead:
  append initrd=initramfs-linux.img ip=::::::dhcp nbd_server=192.168.0.1:10809 cryptdevice=/dev/nbd0:nbdcrypt root=/dev/mapper/nbdcrypt
+
  append initrd=initramfs-linux.img ip=::::::dhcp nbd_host=192.168.0.1 nbd_name=nbdroot cryptdevice=/dev/nbd0:nbdcrypt root=/dev/mapper/nbdcrypt
  
 
==Testing==
 
==Testing==
Line 135: Line 159:
  
 
===Updating the kernel===
 
===Updating the kernel===
Since the kernel the client system boots from is on the server and not in the NBD device itself, you will not be able to simply update your kernel with [[pacman]]. There are several other ways to do this:
+
Since the kernel the client system boots from is on the server and not on the NBD device itself, you will not be able to simply update your kernel with [[pacman]]. There are several other ways to do this:
 
* If you update from the client system:
 
* If you update from the client system:
**share {{ic|/nbd/boot}} on the server via NFS, mount it to {{ic|/boot}} and then update your kernel
+
**share {{ic|/nbd/boot}} on the server via NBD, [[Sshfs]] or [[NFS]], mount it to {{ic|/boot}} and then update your kernel
 
**copy {{ic|vmlinuz-linux}} and {{ic|initramfs-linux.img}} from {{ic|/boot}} back to the server to {{ic|/nbd/boot}} after the kernel update
 
**copy {{ic|vmlinuz-linux}} and {{ic|initramfs-linux.img}} from {{ic|/boot}} back to the server to {{ic|/nbd/boot}} after the kernel update
 
* if you update from the server:
 
* if you update from the server:

Revision as of 18:36, 10 April 2012

Boot from a NBD root device

This article will explain how to boot an ArchLinux Installation from a Network Block Device (NBD).

Much of the work to be done is based on the article Diskless network boot NFS root, so this will be referenced several times within the article.

Advantages/Disadvantages over NFS

Advantages:

  • NBD is faster
  • allows you to boot from an encrypted or LVM-based root device

Disadvantages:

Server-Side Setup

Create the NBD File and Boot Directory

Create a directory that will hold the boot directory and the NBD file.

mkdir -p /nbd/boot/

Next, create the actual file that will be shared via NBD. Of course you can also use an actual block device (a hard drive) instead of creating a file on your filesystem. Just replace /nbd/root with the block device. In this example we are going to create a file with a size of 5GB.

dd if=/dev/zero of=/nbd/root bs=1M count=5000

Now you can create a filesystem on the file.

mkfs.ext4 /nbd/root

mkfs will show you warning about the fact that the file is no actual block device. You can ignore this and simply press Template:Keypress to continue.

Alternatively, if you want to create an encrypted NBD device:

cryptsetup luksFormat -s 256 /nbd/root
cryptsetup luksOpen /nbd/root nbdcrypt
mkfs.ext4 /dev/mapper/nbdcrypt
Note: Be aware that the rest of the article will use /nbd/root. If your NBD file is encrypted, replace it with /dev/mapper/nbdcrypt, if you use an actual block device, with /dev/sdX.

Install ArchLinux on the NBD filesystem

Mount the filesystem:

mount /nbd/root /mnt

Create directories for the pacman database and a few mount points that will be needed:

mkdir -p /mnt/var/lib/pacman/
mkdir /mnt/proc/
mkdir /mnt/dev/
mkdir /mnt/sys/

Mount proc, sysfs and dev. These will be neccessary to build the kernel image in the chroot environment:

mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount -o bind /dev /mnt/dev

Install the Arch Linux base packages into the root directory:

pacman -Sy --root /mnt --dbpath /mnt/var/lib/pacman base

Build the mkinitcpio-nbdAUR package from the AUR. Then install it into the root directory as well:

pacman -U --root /mnt --dbpath /mnt/var/lib/pacman mkinitcpio-nbd-[...]-any.pkg.tar.xz

Configuring the Client system

The following configurations are neccessary for the client system to work properly, everything else can be configured later:

  • set NETWORK_PERSIST="yes" and the other client settings in /mnt/etc/rc.conf. You will also have to remove/disable the network daemon. The net hook will set the IP on boot, so there is no point in using it. Additionaly, if you would change your IP with the network daemon, the connection to the root device would break. If you want a static IP, use the ip kernel parameter (see Boot Configuration).
  • set a mirror in /mnt/etc/pacman.d/mirrorlist.
  • enable your locales in /mnt/etc/locale.gen.
  • set your DNS server in /mnt/etc/resolv.conf, see man 5 resolv.conf for details.
  • add a line for / in /etc/fstab (use /dev/mapper/nbdcrypt instead of /dev/nbd0 if your root device is encrypted):
/dev/nbd0	/	ext4	defaults	0	0
building the kernel image

Set the following hook list in /mnt/etc/mkinitcpio.conf:

HOOKS="base udev net nbd filesystems"

If you use an encrypted NBD device, use this:

HOOKS="base udev net nbd usbinput keymap encrypt filesystems"

Then you can chroot into /mnt and rebuild the kernel image. Remember that you mounted proc, sysfs and dev earlier. This is always neccessary if you want to rebuild the kernel image. locale-gen is only neccessary this one time to set the locales:

chroot /mnt
locale-gen
mkinitcpio -p linux
exit
umount /mnt/dev
umount /mnt/proc
umount /mnt/sys

After leaving the chroot, the kernel image will be in /mnt/boot/. We are going to need it in /nbd/boot/:

cp /mnt/boot/vmlinuz-linux /nbd/boot/
cp /mnt/boot/initramfs-linux.img /nbd/boot/

Configuring the NBD server

nbd-server is configured with the config file /etc/nbd-server/config.

A basic config file may look like this:

/etc/nbd-server/config
[generic]
[nbdroot]
	exportname = /nbd/root

In this example, there is only a simple export with the name nbdroot, which makes the file /nbd/root available via NBD. The name of the export (nbdroot in this case, not /nbd/root!) is important because you will have to set it in the kernel boot parameters later. By default, nbd-server will make the export available to clients with read/write support. If you have more than one client you want to boot from, this will break your filesystem (see Updating the Client System for details). You could enable read-only by setting readonly = true, but this is not recommended because there are just too many things that want to write to /var, /etc or other directories. You could partially work around this with tmpfs, but some things might still break.

Another interesting option is NBD's "copy on write"-mode (copyonwrite = true). From man 5 nbd-server:

"[...] any writes to this export will not be written to the master file, but to a separate file which will be removed upon disconnect. The result of using this option is that nbd-server will be somewhat slower, and that any writes will be lost upon disconnect."

It has not been tested yet how much this really affects performance, but if you want to boot from multiple clients, this is probably your best option.

For a detailed explanation of the config file, see the aforementioned man page man 5 nbd-server.

PXE/TFTP Setup

Install syslinux and dnsmasq on the server, then follow the instructions here. Just make sure you use /nbd/boot/ instead of /disklessroot/boot/ for the TFTP-Root.

Boot Configuration

Copy the pxelinux.0 boot file from syslinux to /nbd/boot and create the pxelinux.cfg directory:

cp /usr/lib/syslinux/pxelinux.0 /nbd/boot/
mkdir /nbd/boot/pxelinux.cfg/

Now create and edit /nbd/boot/pxelinux.cfg/default, which contains the boot configuration for the client. Replace the value for nbd_host with the IP address of your NBD server and nbd_name with the export name you have specificed in /etc/nbd-server/config. If your nbd-server is not listening on the default port (10809), you can set it with nbd_port.

/nbd/boot/pxelinux.cfg/default
default linux

label linux
kernel vmlinuz-linux
append initrd=initramfs-linux.img ip=::::::dhcp nbd_host=192.168.0.1 nbd_name=nbdroot root=/dev/nbd0

See here for details about the ip option.

If your NBD device is encrypted, use the following append line instead:

append initrd=initramfs-linux.img ip=::::::dhcp nbd_host=192.168.0.1 nbd_name=nbdroot cryptdevice=/dev/nbd0:nbdcrypt root=/dev/mapper/nbdcrypt

Testing

Before you boot your client, make sure you unmount everything on the server:

umount /mnt/proc /mnt/sys /mnt/dev /mnt

If your NBD device is encrypted, close the LUKS device as well:

cryptsetup luksClose nbdcrypt

Using a Swap Partition

Although this has not been tested yet, you should be able to do this by creating a LVM volume group on /dev/nbd0 that contains the root and swap partitions and adding lvm2 before the filesystems hook in /mnt/etc/mkinitcpio.conf.

Updating the client system

Quote from the NBD homepage:

"[...] if someone has mounted NBD read/write, you must assure that no one else will have it mounted."

In other words, if you want to be able to update from your client system, you have to make sure that the NBD device is not mounted on any other system, not even read-only. If your NBD device is encrypted, make sure to not just unmount it, but also close it with cryptsetup luksClose. Otherwise, you may break your filesystem. If you keep that in mind, everything except kernel updates should work fine.

Alternatively, you can mount the NBD device on the server and then update it (again, make sure it is not mounted anywhere else!). Note that proc, dev and sys should only necessary for kernel upgrades, but just in case, they are included here.

mount /nbd/root /mnt
mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount -o bind /dev /mnt/dev
chroot /mnt
pacman -Syu
exit
umount /mnt/proc /mnt/sys /mnt/dev /mnt

Updating the kernel

Since the kernel the client system boots from is on the server and not on the NBD device itself, you will not be able to simply update your kernel with pacman. There are several other ways to do this:

  • If you update from the client system:
    • share /nbd/boot on the server via NBD, Sshfs or NFS, mount it to /boot and then update your kernel
    • copy vmlinuz-linux and initramfs-linux.img from /boot back to the server to /nbd/boot after the kernel update
  • if you update from the server:
    • copy vmlinuz-linux and initramfs-linux.img from /mnt/boot to /nbd/boot after the kernel update

Known Problems

Resume from Suspend causes I/O errors

When resuming the machine from suspend, the NBD connection drops and the root device is lost, causing I/O errors. There is currently no fix or workaround for this.