Mister Muffin Blog

Let me share how to setup xen inside qemu with kvm support and domus booting over nfs from the qemu host and connecting multiple of those instances together using vde networking. The debian installer, the debian inside qemu, the domus and debootstrap will use my local apt-cacher setup at port 3142.

This setup is based on Debian wheezy (testing at this point). To install testing, grab the latest debian installer business card image from here:

wget http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/debian-testing-amd64-businesscard.iso

Then create a disk image for qemu to use as its harddisk, create a sparse 3000mb file with dd:

dd if=/dev/zero of=disk.img bs=1 count=1 seek=3000MiB

Using the debian installer to setup the system is preferable in comparison to creating a rootfs with debootstrap as the xen hypervisor wants to be booted by grub. Qemu does not yet support booting the xen hypervisor straight way as it can boot a linux kernel with the -kernel option. Grub installation and partitioning is most easily done by just using d-i.

To automate the installation I'm using the following preseed file:

d-i debian-installer/locale string en_US
d-i console-keymaps-at/keymap select us
d-i keyboard-configuration/xkb-keymap select us
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string debian
d-i netcfg/get_domain string 
d-i mirror/country string manual
d-i mirror/http/hostname string 10.0.2.2:3142
d-i mirror/http/directory string /ftp.de.debian.org/debian
d-i mirror/suite string wheezy
d-i mirror/udeb/suite string wheezy
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password password root
d-i passwd/root-password-again password root
d-i clock-setup/utc boolean true
d-i time/zone string UTC
d-i clock-setup/ntp boolean true
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i base-installer/install-recommends boolean false
d-i base-installer/kernel/image select none
tasksel tasksel/first multiselect 
d-i pkgsel/include string xen-linux-system-amd64 xen-tools xen-utils
d-i finish-install/reboot_in_progress note

It answers all questions for debconf so that no user input is needed. It will tell d-i to use the apt-cacher setup on the qemu host (10.0.2.2 is the default ip address of the host from inside qemu when using usermode networking), will install wheezy, will set the root password to "root", will only create a single partition for / on the virtual harddrive, will not install recommends and not install the "standard" tasksel target but will install the xen hypervisor and some xen tools. I uploaded the file to http://mister-muffin.de/debian/preseed3.txt

To make qemu use virtualization features of the host cpu, one only needs to install the qemu-kvm package.

apt-get install qemu-kvm

From that point on, qemu will automagically use kvm. The speedup gained is tremendous. Instead of taking 13 minutes to boot my xen dom0 (ridiculous) the machine would boot up in only 3 minutes (workable).

Now start qemu, giving it disk.img as the harddisk and debian-testing-amd64-businesscard.iso as the boot medium in the cd drive.

qemu-system-x86_64 -m 1024 -hda disk.img -cdrom

debian-testing-amd64-businesscard.iso

The isolinux boot menu will pop up. Choose "Advanced options" and then select "Automated install" and press [TAB] to edit the boot commandline. Append the preseed url for debconf like this to the end:

preseed/url=http://mister-muffin.de/debian/preseed3.txt

After hitting enter the system will install by itself. After it is done it will automatically reboot into debian. The hypervisor will not boot by default (bug#603832) so change the grub priority from inside the virtual machine by doing:

mv /etc/grub.d/10_linux /etc/grub.d/21_linux
update-grub

You also do not want to continue using qemu in graphical mode but want to connect to the virtual machine via serial. To do so, let a tty spawn on the serial line in inittab:

echo "T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100" >> /etc/inittab

In comparison to the graphical SDL display, the advantages are easy copy and paste, using the same keyboard layout as the host, screensaver is not deactivated, proper console font, terminal, window manager integration, no grabbing and ungrabbing and working system bell.

To also get qemu, kernel and init output on serial add the following options to /etc/default/grub:

GRUB_CMDLINE_LINUX="console=ttyS0"
GRUB_TERMINAL=serial

And run update-grub again.

Now qemu can be started like this:

qemu-system-x86_64 -m 1024 -hda disk.img -nographic

It will automatically boot the xen hypervisor and attach a tty to serial when the boot is finished.

Once it is, configure xen. Activate bridged networking by uncommenting the following line in the xen config:

(network-script network-bridge)

For bridging to work, xen will need the brctl utility of the bridge-utils package (bug#648816).

apt-get install bridge-utils

Since the debian mirror is still the same as during installation, the apt-cacher setup from back then will still be used which makes installation of additional packages extremely fast.

There are then several ways to create a new domu. The easiest one is to just call:

xen-create-image --hostname=vm01 --dir=/root --dhcp --noswap --size=400Mb

This command will first run debootstrap and then configure the result of it. Since the debian mirror of the host system is chosen as the default, debootstrap will run reasonably fast.

A faster way is to have a tarball that contains the result of a debootstrap run ready and then calling xen-create-image the --install-method=tar option.

So either inside or outside qemu (outside is naturally faster) run debootstrap like this:

debootstrap --variant=minbase wheezy target-directory http://127.0.0.1:3142/ftp.de.debian.org/debian

Tar it, put it inside the virtual machine and then inside qemu:

xen-create-image --hostname=vm01 --dir=/root --dhcp --noswap --size=400Mb --install-method=tar --install-source=/root/vm01.tar

This command will unpack the tarball into a disk image and then configure it.

Instead of running xen-create-image inside qemu, you can also run it on the qemu host which will be faster but if you do not want to nfs boot but boot from the disk image it creates, dont forget to copy the xen domu config it creates inside the virtual machine.

Instead of xen-create-image you can also do all steps manually. So first run debootstrap as usual and then do some basic configuration. The most important part would be the activation of the xen tty in inittab. xen-create-image will call a number of hook scripts which do this configuration. Those hooks can also be run manually on a manually created debootstrap root directory like this:

export verbose=true
export hostname=foobar
export dhcp=true
export mirror=http://127.0.0.1:3142/ftp.de.debian.org/debian
export dist=wheezy
for script in `find /usr/lib/xen-tools/debian.d -type f ! -name '90-make-fstab' | sort -n`; do
        $script /home/josch/debian-wheezy
done

Editing fstab is not strictly needed and only hurts when using nfs boot. Dont execute the fstab hook when using nfs and generally have a look in each of the hooks to find out what they do. It is saver to run xen-create-image but to understand what it does, look into the hooks in /usr/lib/xen-tools/debian.d.

Also as a note, you can always mount the qemu image using:

mount -o loop,offset=1048576 disk.img /mnt

The offset of the first partition can be found out using fdisk on disk.img.

In our setup we want to boot the domu from a root directory which is served by the host of the virtual machine. Doing so will just require a proper xen configuration and no disk space on the hypervisor side is used.

Either create a configuration from scratch or use:

xen-create-nfs --hostname=vm01 --dhcp --nfs_server=10.0.2.2 --nfs_root=/srv/nfs/vm01 --memory=128

And then edit the result so that it looks like this:

    kernel     = '/boot/vmlinuz-3.0.0-1-amd64'
    ramdisk    = '/boot/initrd.img-3.0.0-1-amd64'
    vcpus       = '1'
    memory     = '128'
    name       = 'vm01'
    hostname   = 'vm01'
    dhcp       = 'dhcp'
    vif        = [ '' ]
    nfs_server = '10.0.2.2'
    nfs_root   = '/srv/nfs/vm01'
    root       = '/dev/nfs'
    extra      = 'boot=nfs root=/dev/nfs'

You should only have to add the 'extra' option as this is important for the initrd to boot from nfs.

On your host do:

apt-get install nfs-kernel-server

And then add a directory serving a rootfs in /etc/exports:

/srv/nfs 127.0.0.1(rw,sync,no_subtree_check,no_root_squash,insecure)

These options will only allow localhost to access it. The insecure option is necessary because of the choice of port the initrd will connect from.

The rootfs can be created using debootstrap and then running the hooks as explained above, or by taking a filesystem image that was created by xen-create-image and extracting its contents to the directory exported as an nfs share as this image will already contain the modifications needed for a proper boot.

In qemu you can now start the vm and connect to it using:

xm create /etc/xen/vm01.cfg -c

One disconnects with it using the ctrl+] escape as in telnet. To reconnect, use:

xm console vm01

To now connect multiple qemu instances, each running a hypervisor together so that they can each access the internet and talk to each other, the most convenient setup is using VDE networking. It is a network bridge implemented in userspace (no superuser priviliges required) connecting the machines together (bridging them) using socket communication. Together with the slirp module this bridge is connected to the outer world and slirp can even provide dhcp to the qemu instances connected to it.

To start vde:

vde_switch

The -daemon switch can be used to send the process into the background and the -sock switch can be used to supply a socket different from the default /tmp/vde.ctl.

To start slirp:

slirpvde -dhcp

The --daemon option will send it to the background as well while the --sock option allows to supply a custom control socket.

After the bridge is set up and slirp is connected to it, start qemu like this:

qemu-system-x86_64 -m 1024 -hda disk.img -nographic -net nic,macaddr=XX:XX:XX:XX:XX:XX -net vde,sock=/tmp/vde.ctl

Where XX:XX:XX:XX:XX:XX is a unique mac address that can be generated by doing:

printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

The ip addresses supplied by the slirp dhcp will be the same as by qemu usermode networking so the setup from above doesnt change.