Difference between revisions of "Diskless system"

From ArchWiki
Jump to: navigation, search
(Client installation: Trying to make the language clearer and more explicit)
(Additional mountpoints: VFS section is indeed redundant. systemd magic takes care of things now.)
Line 256: Line 256:
===Additional mountpoints===
===Additional mountpoints===
{{Accuracy|Might no longer be necessary}}
Add hacks to your [[fstab]] for the root filesystem and devpts.
{{hc|# vim "$root/etc/fstab"|2=
none    /          none      defaults                        0 0
none    /dev/pts    devpts    mode=0620,gid=5,nosuid,noexec  0 0}}

Revision as of 18:06, 19 December 2012

Template:Article summary start Template:Article summary text Template:Article summary heading Template:Article summary wiki Template:Article summary wiki Template:Article summary wiki Template:Article summary wiki Template:Article summary end From Wikipedia:Diskless node

A diskless node (or diskless workstation) is a workstation or personal computer without disk drives, which employs network booting to load its operating system from a server.

Server configuration

First of all, we must install the following components:

  • A DHCP server to assign IP addresses to our diskless nodes.
  • A TFTP server to transfer the boot image (a requirement of all PXE option roms).
  • A form of network storage (NFS or NBD) to export the Arch installation to the diskless node.
Note: dnsmasq is capable of simultaneously acting as both DHCP and TFTP server. For more information, see the dnsmasq article.


Install ISC dhcp.

# pacman -Syu dhcp

Configure ISC DHCP.

# vim /etc/dhcpd.conf
allow booting;
allow bootp;


option domain-name-servers;

    filename "pxelinux.0";

    subnet netmask {
        option routers;
Note: next-server should be the address of the TFTP server; everything else should be changed to match your network


# systemctl start dhcpd4.service


The TFTP server will be used to transfer the kernel, initramfs, and pxelinux to the client.

See Tftpd server for tftpd installation and configuration. You need to set ftpd server root to:

  • /srv/arch/boot if you are not going to use NBD.
  • /srv/boot if you're going to use NBD, because you'll be unable to mount your root filesystem while in use.

Network storage

The primary difference between using NFS and NBD is while with both you can in fact have multiple clients using the same installation, with NBD (by the nature of manipulating a filesystem directly) you'll need to use the copyonwrite mode to do so, which ends up discarding all writes on client disconnect. In some situations however, this might be highly desirable.


Install nfs-utils on the server.

# pacman -Syu nfs-utils

You'll need to add the root of your arch installation to your NFS exports:

# vim /etc/exports
/srv/arch *(rw,fsid=0,no_root_squash,no_subtree_check)

Next, start NFS.

# systemctl start rpc-idmapd.service rpc-mountd.service
# vim /etc/exports
/srv/arch *(rw,no_root_squash,no_subtree_check,sync)

Next, start NFSv3.

# systemctl start rpc-mountd.service rpc-statd.service
Note: If you're not worried about data loss in the event of network and/or server failure, replace sync with async--additional options can be found in the NFS article.


Install nbd.

# pacman -Syu nbd

Configure nbd.

# vim /etc/nbd-server/config
    exportname = /srv/arch.img
    copyonwrite = false
Note: Set copyonwrite to true if you want to have multiple clients using the same NBD share simultaneously; refer to man 5 nbd-server for more details.

Start nbd.

# systemctl start nbd.service

Client installation

Next we will create a full Arch Linux installation in a subdirectory on the server. During boot, the diskless client will get an IP address from the DHCP server, then boot from the host using PXE and mount this installation as its root.

Directory setup

Create a sparse file of at least 1 gigabyte, and create an ext4 filesystem on it (you can of course also use a real block device or LVM if you so desire).

# truncate -s 1G /srv/arch.img
# mkfs.ext4 /srv/arch.img
# export root=/srv/arch
# mkdir -p "$root"
# mount /srv/arch.img "$root"
Note: Creating a separate filesystem is required for NBD but optional for NFS and can be skipped/ignored.

Bootstrapping installation

Install devtools and arch-install-scripts, and run mkarchroot.

# pacman -Syu devtools arch-install-scripts
# mkarchroot -f --arch x86_64 "$root" base mkinitcpio-nfs-utils nfs-utils

Replace x86_64 with i686 as appropriate for your target hardware.

Note: In all cases mkinitcpio-nfs-utils is still required--ipconfig used in early-boot is provided only by the latter.

Now the initramfs needs to be constructed. The shortest configuration, #NFSv3, is presented as a "base" upon which all subsequent sections modify as-needed.


# vim "$root/etc/mkinitcpio.conf"
HOOKS="base udev autodetect net filesystems"

The initramfs now needs to be rebuilt; the easiest way to do this is via arch-chroot.

# arch-chroot "$root" /bin/bash
(chroot) # mkinitcpio -p linux
(chroot) # exit


Trivial modifications to the net hook are required in order for NFSv4 mounting to work (not supported by nfsmount--the default for the net hook).

# sed s/nfsmount/mount.nfs4/ "$root/usr/lib/initcpio/hooks/net" | tee "$root/usr/lib/initcpio/hooks/net_nfs4"
# cp "$root/usr/lib/initcpio/install/{net,net_nfs4}"

The copy of net is unfortunately needed so it does not get overwritten when mkinitcpio-nfs-utils is updated on the client installation.

From the base mkinitcpio.conf, replace the nfsv3 module with nfsv4, replace net with net_nfs4, and add /sbin/mount.nfs4 to BINARIES.


The mkinitcpio-ndbAUR package needs to be installed on the client.

# pacman --root "$root" --dbpath "$root/var/lib/pacman" -U mkinitcpio-nbd-0.4-1-any.pkg.tar

You will then need to append nbd to your HOOKS array after net.

Client configuration

In addition to the setup mentioned here, you should also set up your hostname, timezone, locale, and keymap, and follow any other relevant parts of the Installation Guide.



Merge-arrows-2.pngThis article or section is a candidate for merging with Syslinux.Merge-arrows-2.png

Notes: (Discuss in Talk:Diskless system#)

Install syslinux.

# pacman -Syu syslinux

Copy the pxelinux bootloader (provided by the syslinux package) to the boot directory of the client.

# cp /usr/lib/syslinux/pxelinux.0 "$root/boot"
# mkdir "$root/boot/pxelinux.cfg"

We also created the pxelinux.cfg directory, which is where pxelinux searches for configuration files by default. Because we don't want to discriminate between different host MACs, we then create the default configuration.

# vim "$root/boot/pxelinux.cfg/default"
default linux

label linux
kernel vmlinuz-linux
append initrd=initramfs-linux.img ip=:::::eth0:dhcp nfsroot=

NFSv3 mountpoints are relative to the root of the server, not fsid=0. If you're using NFSv3, you'll need to pass to nfsroot.

Or if you are using NBD, use the following append line:

append initrd=initramfs-linux.img ip=:::::eth0:dhcp nbd_host= nbd_name=arch
Note: You will need to change nbd_host and/or nfsroot, respectively, to match your network configuration (the address of the NFS/NBD server)

The pxelinux configuration syntax identical to syslinux; refer to the upstream documentation for more information.

The kernel and initramfs will be transferred via TFTP, so the paths to those are going to be relative to the TFTP root. Otherwise, the root filesystem is going to be the NFS mount itself, so those are relative to the root of the NFS server.


Merge-arrows-2.pngThis article or section is a candidate for merging with GRUB2.Merge-arrows-2.png

Notes: (Discuss in Talk:Diskless system#)

Though poorly documented, grub also supports being loaded via PXE. Because of an endianness bug in grub-core/net/tftp.c not fixed until bzr revision 4548 (grub-bios 2.00 is revision 4542), you will need to install grub-bios-bzrAUR; it makes the most sense to do this on the host installation. The Arch User Repository article describes how to build AUR packages.

Note: If you build grub-bios-bzrAUR on the host installation and your target architecture doesn't match; you'll end up needing to build it on the target installation instead
# pacman --root "$root" --dbpath "$root/var/lib/pacman" -U grub-bios-bzr-4599-1-x86_64.pkg.tar.xz

Only because we want to use the grub-bios-bzrAUR installation on the target, we arch-chroot the target installation so we can use its grub-mknetdir instead.

# arch-chroot /srv/arch
(chroot) # grub-mknetdir --net-directory=/boot/ --subdir=/grub
(chroot) # exit
Note: You will need to change the filename directive in /etc/dhcpd.conf to /grub/i386-pc/core.0 as grub-mknetdir will tell you

Now we create a trivial QnD grub configuration

# vim "$root/boot/grub/grub.cfg"
menuentry "Arch Linux" {
    linux /vmlinuz-linux ip=:::::eth0:dhcp nfsroot=
    initrd /initramfs-linux.img

GRUB dark-magic will set root=(pxe: automatically, so that the kernel and initramfs are transferred via TFTP without any additional configuration.

Note: Modify your kernel line as-necessary, refer to #Pxelinux for NBD-related options

Additional mountpoints


For a simple setup, it may not be an issue that /var is shared between all of the clients.

You could mount /var/log, for example, as tmpfs so that logs from multiple hosts don't mix unpredictably.

# vim "$root/etc/fstab"
tmpfs   /var/log    tmpfs     nodev,nosuid                    0 0

Software (BOINC and SLURMAUR, among many others) that places state information/databases and so on in /var might misbehave if multiple instances of said software are interacting with the same directory structure. Furthermore, you might also want these differences between hosts to persist.

Instead, you could create additional exports inside /var to be used by each of your nodes, one for each client (uniquely identified by hostname, for example). The mounting itself of course would occur in late-boot. This approach might prove challenging, however, if multiple copies of identical files are needlessly created, which later become inconsistent after, e.g: software upgrades. Utilizing snapshots on the underlying filesystem might prove useful in this endeavor.

Client boot


Tango-inaccurate.pngThe factual accuracy of this article or section is disputed.Tango-inaccurate.png

Reason: When using COW on the server, the clients all effectively have read-only mounts of the original filesystem; it should theoretically be safe to do a read-write mount on the NBD server (Discuss in Talk:Diskless system#)

If you're using NBD, you'll need to umount the arch.img before/while you boot your client.

This makes things particularly interesting when it comes to kernel updates. You can't have your client filesystem mounted while you're booting a client, but that also means you need to use a kernel separate from your client filesystem in order to build it.

You'll need to first copy $root/boot from the client installation to your tftp root (i.e. /srv/boot).

# cp -r "$root/boot" /srv/boot

You'll then need to umount $root before you start the client.

# umount "$root"
Note: To update the kernel in this setup, you either need to mount /srv/boot using NFS in fstab on the client (prior to doing the kernel update) or mount your client filesystem after the client has disconnected from NBD


kernel.org: Mounting the root filesystem via NFS (nfsroot)

syslinux.org: pxelinux FAQ