From ArchWiki
Revision as of 17:16, 2 June 2013 by Dshub (talk | contribs) (Major rewrite of the dom0 configuration)
Jump to: navigation, search

Template:Article summary start Template:Article summary text Template:Article summary heading Template:Article summary link Template:Article summary heading Template:Article summary wiki Template:Article summary wiki Template:Article summary wiki Template:Article summary wiki Template:Article summary wiki Template:Article summary end

From Xen Overview:

Xen is an open-source type-1 or baremetal hypervisor, which makes it possible to run many instances of an operating system or indeed different operating systems in parallel on a single machine (or host). Xen is the only type-1 hypervisor that is available as open source. Xen is used as the basis for a number of different commercial and open source applications, such as: server virtualization, Infrastructure as a Service (IaaS), desktop virtualization, security applications, embedded and hardware appliances.


The Xen hypervisor is a thin layer of software which emulates a computer architecture. It is started by the boot loader of the computer it is installed on, and allows multiple operating systems to run simultaneously on top of it. Once the Xen hypervisor is loaded, it starts the "dom0" (short for "domain 0", sometimes called the host), or privileged domain, which in our case runs a Linux kernel (other possible dom0 operating systems are NetBSD and OpenSolaris). The physical hardware must, of course, be supported by this kernel to run Xen. Once the dom0 has started, one or more "domUs" (short for user domains, sometimes called VMs or guests) can be started and controlled from dom0. Xen supports both paravirtualized (PV) and hardware virtualized (HVM) domUs.

Xen.org provides a full overview

System requirements

Paravirtualized domUs require a kernel with support for Xen built in. Recent Linux kernels and some other Unix-like systems provide this support, but Microsoft Windows does not. Hardware virtualized domUs do not require kernel level support, but often run slower than PV domUs since HVM run on emulated hardware. To use HVM domUs, the dom0 must include either Intel VT-x or AMD-V (SVM) virtualization support. In order to verify this, run the following command when the Xen hypervisor is not running:

grep -E "(vmx|svm)" --color=always /proc/cpuinfo

If the above command does not produce output, then hardware virtualization support is unavailable and your hardware is unable to run HVM domUs. If you believe the CPU supports one of these features you should access the host system's BIOS configuration menu during the boot process and look if options related to virtualization support have been disabled. If such an option exists and is disabled, then enable it, boot the system and repeat the above command. The Xen hypervisor also supports PCI passthrough where PCI devices can be passed directly to the domU even in the absence of dom0 support for the device. In order to use PCI passthrough the CPU must support IOMMU/VT-d.

Install either the current stable xenAUR or the bleeding edge unstable xen-hg-unstableAUR packages available in the Arch User Repository. Both packages provide the Xen hypervisor, current xl interface and all configuration and support files, including systemd services.

Configuring dom0

The Xen hypervisor relies on a full install of the base operating system. Before attempting to install the Xen hypevisor, the host machine should have a fully operational and up-to-date install of Arch Linux. This installation can be a minimal install with only the base package and does not require a Desktop Environment or even Xorg. If you are building a new host from scratch, see the Installation Guide for instructions on installing Arch Linux. The following configuration steps are required to convert a standard installation into a working dom0 running on top of the Xen hypevisor:

  • Installation of the Xen hypervisor
  • Modification of the bootloader to boot the Xen hypervisor
  • Creation of a network bridge
  • Enable systemd services

The first step is to install the Xen hypervisor. Install either the current stable xenAUR or the bleeding edge unstable xen-hg-unstableAUR packages available in the Arch User Repository. Both of these packages require the multilib repository to be enabled.

The second step is to modify the boot loader to load a special Xen kernel (xen.gz) which is then used to boot the normal kernel. To do this a new bootloader entry is needed.

For grub users, the Xen package provides the /etc/grub.d/09_xen generator file. This file can be edited to customize the Xen boot commands, and will add the required bootloader when the following command is run:

grub-mkconfig -o /boot/grub/grub.cfg

More information is available at Grub.

The third step is to create a network bridge. Xen requires that network communications between the guest, the host (and beyond) is set up manually. The use of both DHCP and static addressing is possible, and the choice should be determined by your network topology. Complex setups are possible, see the Networking article on the Xen wiki for details and /etc/xen/scripts for scripts for various networking configurations. A basic bridged network, in which a virtual switch is created in dom0 that every domu is attached to, can be setup by modifying the example configuration files provided by Netctl in etc/netctl/examples. By default, Xen expects a bridge to exist named xenbr0. To set this up with netctl, do the following:

# cd /etc/netctl
# cp examples/bridge xenbridge-dhcp

make the following changes to xen-bridge:

Description="Xen bridge connection"
BindsToInterface=(eth0) # Use the name of the external interface found with the 'ip link' command

assuming your existing network connection is called eth0.

Start the network bridge with:

netctl start xenbridge-dhcp

when the prompt returns, check all is well

brctl show

bridge name	bridge id		STP enabled	interfaces
xenbr0		8000.001a9206c0c0	no		eth0

If the bridge is working it can be set to start automatically after rebooting with:

netctl enable xenbridge-dhcp

The final step is to enable the xenstored, xenconsoled, and xendomains system services (see Systemd for details).

Reboot your dom0 host and ensure that the Xen kernel boots correctly and that all settings survive a reboot. A properly set up dom0 should report the following when you run xl list (as root):

# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   511     2     r-----   41652.9

Of course, the Mem, VCPUs and Time columns will be different depending on machine configuration and uptime. The important thing is that dom0 is listed.

In addition to the required steps above, the current xen.org wiki has a section regarding best practices for running Xen. It includes information on allocating a fixed amount of memory dom0 and how to dedicate (pin) a CPU core for dom0 use. It also may be beneficial to create a xenfs filesystem mount point by including in /etc/fstab

 none /proc/xen xenfs defaults 0 0

Using Xen

Once the dom0 is fully operational, domUs may be created / imported. Each OS has a slightly different method of installation, see the Guest Install page of the Xen wiki for links to instructions.

Creating a Paravirtualized (PV) Arch domU

This is how to install Arch as a user domain (or VM) on an already-running Xen host. To install Arch as the Xen host (dom0), see the previous section.

To begin, download the latest install ISO from the nearest mirror: Dowload page. Place the ISO file on the dom0 host. (it is recommended that its checksum be verified, too)

Create the hard disks for the new domU. This can be done with LVM, raw hard disk partitions or image files. To create a 10GiB blank hard disk file, the following command can be used:

truncate -s 10G sda.img

This creates a sparse file, which grows (to a maximum of 10GiB) only when data is added to the image. If file IO speed is of greater importance than domain portability, using a Logical Volume or raw partition may be a better choice.

Next, loop-mount the installation ISO. To do this, ensure the directory /mnt exists and is empty, then run the following command (being sure to fill in the correct ISO path):

# mount -o loop /path/to/iso /mnt

Create the bootstrap domU configuration file:

kernel = "/mnt/arch/boot/x86_64/vmlinuz"
ramdisk = "/mnt/arch/boot/x86_64/archiso.img"
extra = "archisobasedir=arch archisolabel=ARCH_201301"
memory = 256
name = "archdomu"
disk = [ "phy:/path/to/partition,sda1,w", "file:/path/to/ISO,sdb,r" ]
vif = [ 'mac=00:16:3e:__random_three_mac_bytes__,bridge=xenbr0' ]

This file needs to tweaked for your specific use. Most importantly, the archisolabel=ARCH_201301 line must be edited to use the release year/month of the ISO being used. If you want to install 32-bit Arch, change the kernel and ramdisk paths from /x86_64/ to /i686/. The "phy:/path/to/partition,sda1,w" line must be edited to point to the partition created for the domU. If an image file is being used, the phy: needs to be changed to file:. Finally, a MAC address must be assigned. The 00:16:3e MAC block is reserved for Xen domains, do the last three digits may be randomly filled in (hex values 0-9 and a-f only). See the xl.cfg man page for more information on what the .cfg file lines do. The AUR package xen-docs will need to be installed to access the man pages.

Create the new domU:

# xl create -c /etc/xen/archdomu.cfg

The -c option will enter the new domain's console when successfully created. At this point, Arch should be installed as usual. The Installation Guide should be followed. There will be a few deviations, however. The block devices listed in the disks line of the cfg file will show up as /dev/xvd*. Use these devices when partitioning the domU. After installation and before the domU is rebooted, the following modules must be added to /etc/mkinitcpio.conf:

MODULES="xen-blkfront xen-fbfront xen-netfront xen-kbdfront"

Without these modules, the domU will not boot correctly. After saving the edit, rebuild the initramfs with the following command:

mkinitcpio -p linux

For booting, it is not necessary to install Grub. Xen has a Python-based grub emulator, so all that is needed to boot is a grub.cfg file: (It may be necessary to create the /boot/grub directory)

menuentry 'Arch GNU/Linux, with Linux core repo kernel' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-core repo kernel-true-__UUID__' {
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  __UUID__
          search --no-floppy --fs-uuid --set=root __UUID__
        echo    'Loading Linux core repo kernel ...'
        linux   /boot/vmlinuz-linux root=UUID=__UUID__ ro
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initramfs-linux.img

This file must be edited to match the UUID of the root partition. From within the domU, run the following command:

# blkid

Replace all instances of __UUID__ with the real UUID of the root partition (the one that mounts as "/").

Shutdown the domU with the poweroff command. The console will be returned to the hypervisor when the domain is fully shut down, and the domain will no longer appear in the xl domains list. Now the ISO file may be unmounted:

# umount /mnt

The domU cfg file should now be edited. Delete the "kernel = ", "ramdisk = ", and "extra = " lines and replace them with the following line:

bootloader = "pygrub"

Also remove the ISO disk from the "disk = " line.

The Arch domU is now set up. It may be started with the same line as before:

# xl create -c /etc/xen/archdomu.cfg

If the domU should be started on boot, create a symlink to the cfg file in /etc/xen/auto and ensure the xendomains service is set up correctly.

Useful xl command examples

# xl top
# xl list
# xl console domUname
# xl shutdown domUname
# xl destroy domUname

Common Errors

  • 'xl list' complains about libxl

- Either you have not booted into the Xen system, or xen modules listed in xencommons script are not installed

  • xl create fails

- check the guest's kernel is located correctly, check the pv-xxx.cfg file for spelling mistakes (like using initrd instead of ramdisk)

  • Arch linux guest hangs with a ctrl-d message

- press ctrl-d until you get back to a prompt, rebuild its initramfs described

  • Error message "failed to execute '/usr/lib/udev/socket:/org/xen/xend/udev_event' 'socket:/org/xen/xend/udev_event': No such file or directory"

- caused by /etc/udev/rules.d/xend.rules; xend is (a) deprecated and (b) not used, so it is safe to remove xend.rules