Diskless system

From ArchWiki
Revision as of 08:47, 26 September 2012 by Buhman (talk | contribs) (Using DHCP & XINETD: cleanup how configuration files are presented; strip tab characters)
Jump to: navigation, search

Tango-view-refresh-red.pngThis article or section is out of date.Tango-view-refresh-red.png

Reason: please use the first argument of the template to provide a brief explanation. (Discuss in Talk:Diskless system#)

Merge-arrows-2.pngThis article or section is a candidate for merging with Diskless network boot NBD root.Merge-arrows-2.png

Notes: duplicated PXE information; the only thing that changes is the root device (Discuss in Talk:Diskless system#)

Root over NFS (diskless boot)

This HOWTO will explain the step-by-step method for booting an Arch Linux installation over an NFS share. The first half will primarily cover configuration of the server, and the second will cover client-side configuration.

For more information on Network File Sharing, see NFS.

Hardware Considerations

Server: Any standard computer should work, but for performance reasons it should have fast disk access and preferably Gigabit Ethernet.

Client: For a completely diskless boot, the client computer must have a network-bootable (PXE) network card. Wireless will not work, with exception of a wireless-to-Ethernet external bridge. Many motherboards have a PXE-compatible ethernet card built in, but you will need to enable support in the BIOS.

If you do not have a PXE-compatible network card, you can set up a semi-diskless client. This involves building a flash drive or CD-based boot partition with syslinux or GRUB. The configuration is the same, but skipping the tftp steps. Creating boot-CDs or flash drives is beyond the scope of this article. Alternatively, you could check out SYSLINUX which can be used to create a bootable CD which will load a PXE-compatible boot loader and then continue booting as if you had a PXE-compatible network card.

For more Information see Etherboot/gPXE

Prebuilt gPXE images are available from http://rom-o-matic.net/

Server-Side Setup

The server will have to run a tftp daemon to share the kernel/bootsector along with an NFS share to share the root directory. PXE requires the DHCP server to announce the location of the tftp server to any clients that connect. Most routers will not support this, and this article will show how to modify an existing DNSMasq or DHCPD server. See NAT'ing_firewall_-_Share_your_broadband_connection for setting up a simple router replacement, including DHCP via DNSMasq, a good place to start for this project. If you do have a router that supports PXE, for example one running DD-WRT, then you will need to tell it to send all tftp requests to the server.

You could also use gpxe from [1] without tftp at all if you already have some web-server up and running. gPXE supports ftp, http, https and Wi-Fi booting.

At this point I will assume you have a server running with DNSMasq or DHCPd providing DHCP services, along with some space for the client's root.

First, install nfs-utils, syslinux (for the pxelinux bootloader, you can substitute for GRUB, see [2]).

Create Client Root Directory

First, create a directory that will hold the root filesystem, this can be anywhere. For this example, I will use /disklessroot.

mkdir /disklessroot

Create a directory for new pacman database, proc and dev mountpoints.

mkdir -p /disklessroot/var/lib/pacman
mkdir /disklessroot/proc
mkdir /disklessroot/dev
mkdir /disklessroot/sys

Mount the /proc and /sys filesystems to allow the installation to use the kernel-provided information within the chrooted environment, and then mount-bind the /dev filesystem.

mount -t proc none /disklessroot/proc
mount -t sysfs none /disklessroot/sys
mount -o bind /dev /disklessroot/dev

Now you need to populate that root with an installation of Arch Linux.

If you are running 64-bit server and your client is 32-bit, you want to change /etc/pacman.d/mirrorlist entry to point 32-bit architecture. Remember to switch it back after populating client's filesystem. Also you want to add --arch i686 to the following command.

pacman -Sy --root /disklessroot --dbpath /disklessroot/var/lib/pacman base

Install mkinitcpio-nfs-utils for ipconfig and nfsmount tools for NFS root support in mkinitcpio:

pacman -Sy --root /disklessroot --dbpath /disklessroot/var/lib/pacman mkinitcpio-nfs-utils
Warning: Version 0.2 of mkinitcpio-nfs-utils does not support NFS4, so make sure you use NFS3 if there is not yet a newer version of this package available.

Edit /disklessroot/etc/mkinitcpio.conf, add nfs to the MODULES section, and add net after udev under HOOKS.

HOOKS="base udev net autodetect pata scsi sata filesystems"

Now, chroot into /disklessroot and recreate the kernel image.

chroot /disklessroot
mkinitcpio -p linux
umount /disklessroot/dev
umount /disklessroot/proc
umount /disklessroot/sys

You should be back to the normal root, the exit command should leave the chroot. Check before continuing.


Using DNSMasq

Now to setup tftp and PXE.

Within DNSMasq, there is an easy-to-use tftp server, so this article will use that. First, edit /etc/dnsmasq.conf and uncomment these lines, changing them to reflect your file paths.

dhcp-range=,,12h     #change IP addresses to your network
# Set the boot filename for BOOTP. You will only need 
# this is you want to boot machines over the network and you will need
# a TFTP server; either dnsmasq's built in TFTP server or an
# external one. (See below for how to enable the TFTP server.)
dhcp-boot=pxelinux.0  #change this if you are using GRUB.

# Enable dnsmasq's built-in TFTP server

# Set the root directory for files available via FTP.

#Send options to PXELinux. Note that we need to send the options even
#though they do not appear in the parameter request list, so we need
#to use dhcp-option-force here.
#See http://syslinux.zytor.com/pxe.php#special for details.
#Magic number - needed before anything else is recognised

#Configuration file name

#Path prefix

This has the additional feature of allowing you to update the kernel within the client OS.

Make sure the firewall on your server does not block UDP port 69 (UDP ports 67-68 and 4011 are also used, but should not need to be opened).


If you already have XINETD & DHCP running then you can use them to support net booting in place of the DNSMasq method above. The xinetd service will start the TFTP program on demand as needed.

First, modify the ISC DHCPD configuration to add the following statements:

allow booting;
allow bootp;

    next-server; #change IP address to match your server
    # This is the pxe bootloader file
    filename "pxelinux.0";

    # One host block per client. This network only has one.    
    host netboot1 
        #change below to match MAC address of client network device
        hardware ethernet 08:00:27:d0:9b:97;
        #change below to IP address desired for client  

Second, create a file /etc/xinetd.d/tftp with the following lines. After the the xinetd service is restarted the tftpd program will start on demand.

service tftp
    per_source = 11
    socket_type = dgram
    protocol = udp
    user = root
    server = /usr/sbin/tftpd
    server_args = /disklessroot/boot
    wait = yes
    cps = 100 2

Be sure to restart the DHCP & XINETD services to activate the changes.

NFS Setup

Edit /etc/exports, and add this line.

/disklessroot *(rw,fsid=0,no_root_squash,no_subtree_check)

Add rpcbind nfs-common nfs-server to the server's /etc/rc.conf in the DAEMONS array, in that specific order. (Assuming dnsmasq was already installed and on the daemons list.)

Refer to NFS for any configuration/security issues that are not covered here.

Client Configuration

These files are to be edited from the server, as the client is not ready yet.

Boot Configuration

Copy pxelinux.0 to /disklessroot/boot/

cp /usr/lib/syslinux/pxelinux.0 /disklessroot/boot/
mkdir /disklessroot/boot/pxelinux.cfg

Create and edit /disklessroot/boot/pxelinux.cfg/default, replacing IP addresses and file paths as needed.

default linux

label linux
kernel vmlinuz-linux
append initrd=initramfs-linux.img rootfstype=nfs root=/dev/nfs nfsroot=,v3,rsize=16384,wsize=16384 ip=::::::dhcp

Note (one user's working setup): root=/dev/nfs should be set as root=/disklessroot in this example. nfsroot root IP address set to LAN IP address of the server e.g. nfsroot=

For more ip option look here Using net

The syntax for the pxelinux bootloader is the same as Syslinux, here[3] under "CONFIGURATION FILE". Remember that any filenames will be relative to the tftp server, check the setting in /etc/dnsmasq.conf named tftp-root=. This example places tftp-root at /disklessroot/boot/. Thus the kernel should be at /disklessroot/boot/vmlinuz-linux on the server, and will show up as /vmlinuz-linux for the first stage of client boot. Exactly like a separate /boot partition.

As there is no entry for / in the client's /etc/fstab file, the client may display the error mount: can't find / in /etc/fstab or /etc/mtab on start-up. This error can be safely ignored. However, if you want to suppress the error you can add a hack to /etc/fstab. This involves adding the following to /disklessroot/etc/fstab:

none / none

Note: If you want ssh into netbooted machine, add this line to /disklessroot/etc/fstab:

none            /dev/pts        devpts        gid=5,mode=620    0 0

Without this,you will get an error that states: "Server refused to allocate pty".

DHCP/Network Daemon Workaround

This is to prevent the client from trying to reconnect the network and killing itself. Any disconnect of the network and your client will freeze.

The following workaround involves running the dhcpcd DHCP client on start-up with the -s option to use the existing kernel DHCP auto-configured IP address instead of requesting a new one. Note: This workaround assumes that eth0 is the network interface used for DHCP.

Edit /etc/conf.d/dhcpcd and append the DHCPCD_ARGS options with -s $(ifconfig eth0 | grep -o '[0-9]*\.[0-9\.]*' | head -n1)

# Arguments to be passed to the DHCP client daemon

DHCPCD_ARGS=" -s $(ifconfig eth0 | grep -o '[0-9]*\.[0-9\.]*' | head -n1)"

To allow the shutdown process to work properly you must also set the following option in /etc/rc.conf


Another workaround is presented below which configures the DNS server manually.

DHCP/Network Daemon Workaround Number 2

Edit the client's /etc/rc.conf and disable the network daemon (/disklessroot/etc/rc.conf, or just /etc/rc.conf once you boot into the client).

DAEMONS=(syslog-ng !network ...

The main issue with this method is the fact that your /etc/resolv.conf will not auto-update (since this bypasses DHCP). To work around this, edit /etc/resolv.conf and add this line (/disklessroot/etc/resolv.conf on the server).

nameserver  #change this IP address for whatever DHCP/DNS server you have, the server's IP address usually works if you configured DNS correctly.

Testing/Debugging the Client

Now reboot the client computer, making sure the network card is first in the boot-order.

If all goes right, you should see the network card get an IP address from the server, then connect and boot the kernel. After the initial kernel messages, you should either see Arch Linux boot, or a "killed init" message.

If init was killed, check your configuration, it usually means it could not mount the NFS folder. Refer to NFS for more help.

If it freezes during a mounting operation, check the client's /etc/fstab or the /boot/pxelinux.cfg/default settings.

If it freezes duing the "Network" section, either disable it using the previously mentioned workaround, or specify IP configuration settings in /etc/rc.conf.

Performance Notes

I am currently using a diskless client as my primary Internet/development OS due to a hard drive restriction on my PC. My server is an older machine is running Arch Linux with a software-based RAID 5 array. My client can read files from the NFS share almost as fast as my server can, but there is a limitation with latency and caching. It slows down during pacman -Sy and package installs that involve many small files. Otherwise, it runs fine. It also seems that NFS runs synchronously by default, meaning that all writes have to be committed before continuing, no caching in free memory like it would on a local hard drive. This is actually a good thing, from the data integrity standpoint, as fewer files are lost during a random power outage. I have not tried the async NFS option because the nfs man page does not suggest it will work for NFSv3.

Note: NFSv3 is really old, and you should probably be using NFSv4 now.