Let me share how to setup xen inside qemu with kvm support and domus booting over nfs from the qemu host and connecting multiple of those instances together using vde networking. The debian installer, the debian inside qemu, the domus and debootstrap will use my local apt-cacher setup at port 3142.

This setup is based on Debian wheezy (testing at this point). To install testing, grab the latest debian installer business card image from here:

wget http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/debian-testing-amd64-businesscard.iso

Then create a disk image for qemu to use as its harddisk, create a sparse 3000mb file with dd:

dd if=/dev/zero of=disk.img bs=1 count=1 seek=3000MiB

Using the debian installer to setup the system is preferable in comparison to creating a rootfs with debootstrap as the xen hypervisor wants to be booted by grub. Qemu does not yet support booting the xen hypervisor straight way as it can boot a linux kernel with the -kernel option. Grub installation and partitioning is most easily done by just using d-i.

To automate the installation I'm using the following preseed file:

d-i debian-installer/locale string en_US
d-i console-keymaps-at/keymap select us
d-i keyboard-configuration/xkb-keymap select us
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string debian
d-i netcfg/get_domain string 
d-i mirror/country string manual
d-i mirror/http/hostname string 10.0.2.2:3142
d-i mirror/http/directory string /ftp.de.debian.org/debian
d-i mirror/suite string wheezy
d-i mirror/udeb/suite string wheezy
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password password root
d-i passwd/root-password-again password root
d-i clock-setup/utc boolean true
d-i time/zone string UTC
d-i clock-setup/ntp boolean true
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i base-installer/install-recommends boolean false
d-i base-installer/kernel/image select none
tasksel tasksel/first multiselect 
d-i pkgsel/include string xen-linux-system-amd64 xen-tools xen-utils
d-i finish-install/reboot_in_progress note

It answers all questions for debconf so that no user input is needed. It will tell d-i to use the apt-cacher setup on the qemu host (10.0.2.2 is the default ip address of the host from inside qemu when using usermode networking), will install wheezy, will set the root password to "root", will only create a single partition for / on the virtual harddrive, will not install recommends and not install the "standard" tasksel target but will install the xen hypervisor and some xen tools. I uploaded the file to http://mister-muffin.de/debian/preseed3.txt

To make qemu use virtualization features of the host cpu, one only needs to install the qemu-kvm package.

apt-get install qemu-kvm

From that point on, qemu will automagically use kvm. The speedup gained is tremendous. Instead of taking 13 minutes to boot my xen dom0 (ridiculous) the machine would boot up in only 3 minutes (workable).

Now start qemu, giving it disk.img as the harddisk and debian-testing-amd64-businesscard.iso as the boot medium in the cd drive.

qemu-system-x86_64 -m 1024 -hda disk.img -cdrom

debian-testing-amd64-businesscard.iso

The isolinux boot menu will pop up. Choose "Advanced options" and then select "Automated install" and press [TAB] to edit the boot commandline. Append the preseed url for debconf like this to the end:

preseed/url=http://mister-muffin.de/debian/preseed3.txt

After hitting enter the system will install by itself. After it is done it will automatically reboot into debian. The hypervisor will not boot by default (bug#603832) so change the grub priority from inside the virtual machine by doing:

mv /etc/grub.d/10_linux /etc/grub.d/21_linux
update-grub

You also do not want to continue using qemu in graphical mode but want to connect to the virtual machine via serial. To do so, let a tty spawn on the serial line in inittab:

echo "T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100" >> /etc/inittab

In comparison to the graphical SDL display, the advantages are easy copy and paste, using the same keyboard layout as the host, screensaver is not deactivated, proper console font, terminal, window manager integration, no grabbing and ungrabbing and working system bell.

To also get qemu, kernel and init output on serial add the following options to /etc/default/grub:

GRUB_CMDLINE_LINUX="console=ttyS0"
GRUB_TERMINAL=serial

And run update-grub again.

Now qemu can be started like this:

qemu-system-x86_64 -m 1024 -hda disk.img -nographic

It will automatically boot the xen hypervisor and attach a tty to serial when the boot is finished.

Once it is, configure xen. Activate bridged networking by uncommenting the following line in the xen config:

(network-script network-bridge)

For bridging to work, xen will need the brctl utility of the bridge-utils package (bug#648816).

apt-get install bridge-utils

Since the debian mirror is still the same as during installation, the apt-cacher setup from back then will still be used which makes installation of additional packages extremely fast.

There are then several ways to create a new domu. The easiest one is to just call:

xen-create-image --hostname=vm01 --dir=/root --dhcp --noswap --size=400Mb

This command will first run debootstrap and then configure the result of it. Since the debian mirror of the host system is chosen as the default, debootstrap will run reasonably fast.

A faster way is to have a tarball that contains the result of a debootstrap run ready and then calling xen-create-image the --install-method=tar option.

So either inside or outside qemu (outside is naturally faster) run debootstrap like this:

debootstrap --variant=minbase wheezy target-directory http://127.0.0.1:3142/ftp.de.debian.org/debian

Tar it, put it inside the virtual machine and then inside qemu:

xen-create-image --hostname=vm01 --dir=/root --dhcp --noswap --size=400Mb --install-method=tar --install-source=/root/vm01.tar

This command will unpack the tarball into a disk image and then configure it.

Instead of running xen-create-image inside qemu, you can also run it on the qemu host which will be faster but if you do not want to nfs boot but boot from the disk image it creates, dont forget to copy the xen domu config it creates inside the virtual machine.

Instead of xen-create-image you can also do all steps manually. So first run debootstrap as usual and then do some basic configuration. The most important part would be the activation of the xen tty in inittab. xen-create-image will call a number of hook scripts which do this configuration. Those hooks can also be run manually on a manually created debootstrap root directory like this:

export verbose=true
export hostname=foobar
export dhcp=true
export mirror=http://127.0.0.1:3142/ftp.de.debian.org/debian
export dist=wheezy
for script in `find /usr/lib/xen-tools/debian.d -type f ! -name '90-make-fstab' | sort -n`; do
        $script /home/josch/debian-wheezy
done

Editing fstab is not strictly needed and only hurts when using nfs boot. Dont execute the fstab hook when using nfs and generally have a look in each of the hooks to find out what they do. It is saver to run xen-create-image but to understand what it does, look into the hooks in /usr/lib/xen-tools/debian.d.

Also as a note, you can always mount the qemu image using:

mount -o loop,offset=1048576 disk.img /mnt

The offset of the first partition can be found out using fdisk on disk.img.

In our setup we want to boot the domu from a root directory which is served by the host of the virtual machine. Doing so will just require a proper xen configuration and no disk space on the hypervisor side is used.

Either create a configuration from scratch or use:

xen-create-nfs --hostname=vm01 --dhcp --nfs_server=10.0.2.2 --nfs_root=/srv/nfs/vm01 --memory=128

And then edit the result so that it looks like this:

    kernel     = '/boot/vmlinuz-3.0.0-1-amd64'
    ramdisk    = '/boot/initrd.img-3.0.0-1-amd64'
    vcpus       = '1'
    memory     = '128'
    name       = 'vm01'
    hostname   = 'vm01'
    dhcp       = 'dhcp'
    vif        = [ '' ]
    nfs_server = '10.0.2.2'
    nfs_root   = '/srv/nfs/vm01'
    root       = '/dev/nfs'
    extra      = 'boot=nfs root=/dev/nfs'

You should only have to add the 'extra' option as this is important for the initrd to boot from nfs.

On your host do:

apt-get install nfs-kernel-server

And then add a directory serving a rootfs in /etc/exports:

/srv/nfs 127.0.0.1(rw,sync,no_subtree_check,no_root_squash,insecure)

These options will only allow localhost to access it. The insecure option is necessary because of the choice of port the initrd will connect from.

The rootfs can be created using debootstrap and then running the hooks as explained above, or by taking a filesystem image that was created by xen-create-image and extracting its contents to the directory exported as an nfs share as this image will already contain the modifications needed for a proper boot.

In qemu you can now start the vm and connect to it using:

xm create /etc/xen/vm01.cfg -c

One disconnects with it using the ctrl+] escape as in telnet. To reconnect, use:

xm console vm01

To now connect multiple qemu instances, each running a hypervisor together so that they can each access the internet and talk to each other, the most convenient setup is using VDE networking. It is a network bridge implemented in userspace (no superuser priviliges required) connecting the machines together (bridging them) using socket communication. Together with the slirp module this bridge is connected to the outer world and slirp can even provide dhcp to the qemu instances connected to it.

To start vde:

vde_switch

The -daemon switch can be used to send the process into the background and the -sock switch can be used to supply a socket different from the default /tmp/vde.ctl.

To start slirp:

slirpvde -dhcp

The --daemon option will send it to the background as well while the --sock option allows to supply a custom control socket.

After the bridge is set up and slirp is connected to it, start qemu like this:

qemu-system-x86_64 -m 1024 -hda disk.img -nographic -net nic,macaddr=XX:XX:XX:XX:XX:XX -net vde,sock=/tmp/vde.ctl

Where XX:XX:XX:XX:XX:XX is a unique mac address that can be generated by doing:

printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

The ip addresses supplied by the slirp dhcp will be the same as by qemu usermode networking so the setup from above doesnt change.

Introduction

I recently put Debian Sid on my Notion Ink Adam and since I wouldnt have managed to do that alone I want to thank the guys in ##adamroot on freenode - especially RaYmAn who initially assembled a boot.img to put ubuntu on the adam and also wrote the hack to use the multitouch touchscreen as singletouch. IntuitiveNipple was also very helpful and you should check out his great personal wiki.

Quick Solution

Just download my pre-assembled boot.img

You will flash it to the adam from your linux host with a proprietary tool called nvflash that you can grab here for your specific version of adam:

• LCD WiFi 3G
• LCD WiFi
• Pixel Qi WiFi
• Pixel Qi WiFi 3G

unzip it and find out the partition number you want to flash the image to. For each invocation of nvflash you have to put your adam into apx mode recovery by holding down the volume down button and the power button until the orange charge LED and the red CPU LED light up.

./nvflash --bl bootloader.bin --getpartitiontable parts.cfg

In parts.cfg the SOS partition is the recovery partition (most likely partition 8) and LNX is the boot partition (most likely partition 9). Flashing boot.img to the recovery partition will give dual boot capabilities but destroy the recovery image. Flashing it to the boot partition has the advantage that it boots per default. I flashed it to the boot partition which was partition 9 for me.

./nvflash --bl bootloader.bin --download 9 boot.img

Once that finished the adam can be switched off an on again. The adam will now boot some /sbin/init from the first partition on an SD card.

I prepared a debian rootfs you can just extract to your SD. It comes with e17 and you best select illume in the welcome screen: debian-sid-multistrap.tar.xz

The rootfs also includes the proprietary broadcom module and firmware for the bcm4329: modules.tar.gz

The rootfs was created using this script: rootstock.sh

Modifying boot.img

If you want to make changes to the boot.img (specifically recompile the kernel) it works the following:

To extract and create the boot.img use the android BootTools (local copy)

git clone https://github.com/AndroidRoot/BootTools.git
cd BootTools
make

With this tool you can already unpack the boot.img and look what's inside. The boot.img will unpack into a ramdisk and the kernel and a textfile that specifies additional parameters like the kernel cmdline arguments. The ramdisk can be examined and changed as usual with cpio.

./BootTools/bootunpack boot.img
gunzip boot.img-ramdisk.cpio.gz
mkdir ramdisk
cd ramdisk
cpio -id < ../boot.img-ramdisk.cpio

Ramdisk

The ramdisk is good for two things: work around the setfont bug and introduce a delay before the root partition from the SD is mounted. I tried to boot without a ramdisk, using the rootdelay kernel commandline option but it didnt work. I have no idea what is going wrong due to the setfont bug. This bug means that the framebuffer console will only output anything else than the initial blank, black screen when a KDFONTOP ioctl is done to one of the ttys.

The ramdisk contains an /init that mounts the root partition from the SD card after a delay of three seconds or falls back to a shell if there is no /sbin/init on it and statically compiled busybox and setfont. Setfont is a program that will do the ioctl and load the font Lat15-Fixed16 to the tty specified by its first command line argument. The sourcecode is included in the ramdisk itself as setfont.c.gz and can also be downloaded here: setfont.c.

Cross compiling a kernel for ARM

To compile the kernel you need a cross build toolchain that is easiest installed via the emdebian repository.

apt-get install emdebian-archive-keyring
echo deb http://www.emdebian.org/debian/ squeeze main >> /etc/apt/sources.list
apt-get update
apt-get install gcc-4.4-arm-linux-gnueabi

clone the notionink kernel sources:

git clone https://github.com/notionink/adam-kernel.git

put my config into it

curl http://mister-muffin.de/adam/config > adam-kernel/.config

the following patch will change the multitouch driver into a singletouch one that is supported by the current Xorg evdev driver and will also add the accept4 system call that is needed for udev >= 168.

cd adam-kernel
curl http://mister-muffin.de/adam/kernel.diff | git apply

build the thing

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- -j5

create the boot.img from the new kernel and the ramdisk

./BootTools/mkbootimg --kernel adam-kernel/arch/arm/boot/zImage \
--ramdisk linux_boot.img-ramdisk.cpio.gz \
--cmdline 'mem=256M@0M nvmem=256M@256M mem=512M@512M vmalloc=384M video=tegra'\
'fb console=tty0,115200n8 usbcore.old_scheme_first=1 cpuid=200102 devicetype='\
'1002 tegraboot=nand mtdparts=tegra_nand:16384K@9984K(misc),16384K@26880K(rec'\
'overy),16384K@43776K(boot),204800K@60672K(system),781952K@266112K(cache) roo'\
't=/dev/mmcblk0p1 init=/sbin/init debug' --output boot_new.img

and upload it to the device as shown above

Device Specific Things

Input Devices

• /dev/input/event0 - hardware buttons
• /dev/input/event1 - capacitative buttons
• /dev/input/event2 - touchscreen
• /dev/input/event5 - accelerometer

the pixelqi display mode can be set with

/sys/devices/platform/pixel_qi_screen_ctrl/state

accelerometer is enabled with

/sys/devices/platform/accelerometer/enable

WWAN

the wwan card is a Ericsson F3307 and provides a cdc ethernet interface

chat -V ABORT "ERROR\r\n" "" \
"AT+CFUN=1" "OK\r\n" \
"AT+CGDCONT=1,"IP","$APN" "OK\r\n" \
"AT*ENAP=1,1" "OK\r\n" > /dev/ttyACM0 < /dev/ttyACM0
dhclient -v usb1

(notice that, depending on your provider you might not need to specify the APN - at least here in germany I didnt yet encounter a provider where you had to)

WLAN

wlan works as usual via wpa_supplicant

GPS

gps is /dev/ttyHS3 and outputs NMEA

Battery

/sys/devices/platform/smba10xx_battery/power_supply/battery/capacity

Not working

• alsa (no clue what's going on)
• camera (no v4l2 device in /dev)
• bluetooth (part of bcm wireless)
• fm-transceiver (part of bcm wireless)
• compass (needs memsic daemon running)
• cpu scaling (needs nvidia daemon)

As I own a Always Innovating Touchbook I wanted to make use of its touchscreen by having some physics simulation games on it. Unfortunately it's kinda hard to find such open source games as the really good ones, phun (now algodoo), zanydoodle and crayonphyiscs are proprietary. Noteable opensource games I found were numptyphysics and the powder toy

While phun and the powder toy come with their own simulation engine, the rest are using open source libraries. This got me to research a bit to what is available on open source 2D rigid body physics simulation libraries.

The one that numptyphysics and crayonphysics (and angrybirds!) rely upon is box2d. As it is also used by the torque 2d engine it is probably also used in amazing games like Insanely Twisted Shadow Planet. Starting of in 2007 it is the older library but doesnt seem to be further developed since 2010. It is implemented in C++, released under the zlib license, rendering agnostic and supports lot of features like hinges, motors, friction and springs. Its sort and prune based broadphase collision detection is supposed to work better with objects of varying size.

The newer and more fancy one is chipmunk which builds on parts of box2d, is implemented in C99 and is more high level than box2d. It is used by zany doodle and xmoto and released under MIT license. The spatial hash broad phase collision detection is supposed to make it faster for lots of objects of the same size. This is also one of the reasons why chipmunk seems to be much faster with fluids which are quite problematic for box2d because of the huge number of small circles used to simulate it.

Box2d is ported to many languages, chipmunk as many bindings. They are both cross platform, running on Windows, Mac OS, GNU/Linux, Android and Apple iOS. Chipmunk has a more modern and thought out API but that might just be because of my C++ dislike. In the end, forum posts suggest that each of them has it's strengths and weaknesses and (as usual) in the end one has to go with the one than one is more comfortable in using.

Unfortunately the most impressive engine seems to be integrated in the propriety game phun, or algodoo as it was renamed. But this might just be an impression by the immensely complex and function rich program that phun/algodoo is. Most noteably is probably its smooth handling of water including simulation of buoyancy but it also simulates specular reflection, fraction and dispersion of light. Its other features like hinges, motors, springs, friction and mass are also handled by chipmunk and box2d but there is an editor missing that is as powerful as the one of phun/algodoo.

Now the two open source games I was checking out were numptyphysics and the powder toy. I wanted to have both on my touchbook and be able to control them with the touchscreen only.

Now numptyphysics is packaged in debian but development seems to have stopped in 2008. Fortunately it's quite useable and even supports touchscreen only operation by two slide-out menus on the left and right. It took me ages to find those and i only found them by accidentally clicking on the 5 pixel border of the window which slides out the menu. With this one it can be fully controlled with the left mouse button and is hence ready for touchscreens. Unfortunately it needs some tweaks to be nicely useable. Those changes were, to increase the window size to 1024x600 (there is no way to do that without recompiling), change the selection and click tolerance to bigger values for touchscreen use (also needs recompilation), increase the area on both sides for the slide-out menus (again needs to be recompiled) and disable drawing of the rotating joint symbol as it doesnt get scaled.

You can find my changes here maybe some time I will make a better patch that also includes command line options to change all of that at runtime.

Then there is the powder toy. It comes with a fixed resolution as well which is mainly due to the fact that it builds on an online community that shares their creation and those are of fixed size. If one doesnt care about that, changing the resolution is very easy but also only possible at compile time. It involves just changing XRES, YRES, XCNTR and YCNTR in includes/defines.h. Since I wanted to run it on the armel architecture I added a new target to the makefile that built powder without sse. Also to make it run faster I didnt choose 1024x600 but 512x300 as the resolution and combined that with the scale:2 commandline parameter. Sadly scale can be only 1 or 2.

While numptyphysics is perfectly useable on the touchbook performance wise, the powder toy runs at a mere 9 fps with an empty screen. Maybe this will improve with armhf but i didnt check out how much the powder toy uses floating point arithmetics.

In the end here are some videos that are worth watching:

• algoobla
• phun
• torque2d+box2d
• Insanely Twisted Shadow Planet
• crayonphysics
• chipmunk
• numptyphysics

Since I debootstrap a lot, I download the same base packages from the debian server over and over again. This sucks as it eats up the bandwidth of the debian people and is slow for me. The solution is apt-cacher that acts as a local debian repository and totally automagically even works for foreign architectures or distributions.

After apt-getting it, the only things I did to make it run, was to edit /etc/apt-cacher/apt-cacher.conf and /etc/default/apt-cacher. In apt-cacher.conf I had to remove the asterisk from the allowed_hosts setting and also set generate_reports to 0. Then setting AUTOSTART to 1 in the default file and from that point on I just have to use the following repository mirror URL in my bootstrapping scripts:

http://localhost:3142/ftp.de.debian.org/debian

The beauty is, that I dont have to configure apt-cacher to source http://ftp.de.debian.org beforehand - it's all done by a request of the above style. Would I want apt-cacher to choose a different mirror, I would just request another url from it. This way I can transparently cache any debian mirror I want without any hassle.

Creating a new debian rootfs is fun and easy - you would just call:

sudo debootstrap --variant=minbase sid $ROOTDIR http://ftp.debian.org/debian

or

sudo cdebootstrap --flavour=minimal sid $ROOTDIR http://ftp.debian.org/debian

You could then either chroot(8) into that directory or even put the whole thing on an SD card or harddisk and boot from it (with a few additional modifications of course).

A recent addition to the family is multistrap that uses dpkg and apt to assemble the rootfs and hence benefits from the power, flexibility and reliability of these tools. For example it is possible to mix different repositories.

sudo multistrap -f /usr/share/multistrap/sid.conf -d $ROOTDIR

Now all three build a rootfs of the host's native architecture by default but there is also the option of building a rootfs for a foreign architecture. In my case I mostly build for armel from now on I will use armel as an example for a foreign architecture.

sudo debootstrap --foreign --arch=armel ...
sudo cdebootstrap --foreign --arch=armel ...
sudo multistrap -a armel ...

For all three of them there is a second stage that will require you to be able to execute the binary format of the target architecture.

cdebootstrap puts the second stage installer into $ROOTDIR/sbin/cdebootstrap-foreign and also $ROOTDIR/sbin/init links to it so that on a possible boot on the native hardware cdebootstrap-foreign would be executed to finish the bootstrapping. The real init is saved into $ROOTDIR/sbin/init.foreign.

debootstrap will be invoked with the --second-stage option on the $ROOTDIR directory on the target architecture to invoke the second stage.

multistrap requires you to run "dpkg --configure -a" in a native environment.

Now running the first stage on your host and the second stage on the device is a tedious thing, so I always ended up doing the full bootstrapping on the native hardware to avoid the hassle. Since I build for armel the target hardware is mostly quite a bit slower than my amd64 host - in terms of raw processing power and also in terms of I/O speed due to slow flash storage. Also I would prefer to use my mobile phone for actual telephony and not misuse it as a buildhost. To solve the problem I first used qemu. I set up a virtual machine using qemu-system-arm and inside that did the bootstrapping nativeley.

While the slowness issue is mostly solved by this (emulation is slow but bootstrapping benefits from the much better I/O capabilities of my host) it still remains bothersome to always have to boot a whole virtual machine which is kind of an overkill.

The problem is solved by qemu user mode emulation. Upon encountering a foreign armel ELF binary qemu-arm-static will be invoked to execute it using the kernel's binfmt mechanism. This has the advantage that one can now run code of arbitrary architectures on any amd64/i386 host without emulating a full machine but by only emulating the foreign instructions. This again is a huge increase in simplicity and execution speed.

All one has to do is to run the first stage of one of the above methods to bootstrap a debian rootfs, then copy the qemu-arm-static binary inside it to $ROOTDIR/usr/bin and then just chrooting to it.

sudo multistrap -a armel ...
sudo /usr/bin/qemu-arm-static $ROOTDIR/usr/bin
sudo chroot $ROOTDIR dpkg --configure -a
sudo chroot $ROOTDIR /bin/bash

The last two calls that invoke dpkg and bash inside the chroot environment are actually invoking armel binaries that are interpreted by qemu-arm-static on the fly. It feels as if the system would be a native one - very cool!

So at this point I thought I had found the holy grail of creating a foreign debian rootfs as I now had just one script that would invoke multistrap, copy qemu-arm-static over, configure dpkg, would do some additional foo, remove qemu-arm-static and pack everything in a tarball. But one thing still bothered me.

The end result i wanted to get out of the whole process was a tarball or maybe a squashfs image. Now both are just two files on my harddrive containing information about the contained files and directories, their permissions and so on but basically just a bunch of bytes in a special structure. So why would I need superuser rights to create this amount of bytes in my home directory?

The first stage of all three methods can be executed with fakeroot but for the second stage a call to chroot(2) is required which fakeroot doesnt implement. fakechroot comes to the rescue and imitates a chrooted environtment. It's not perfect but it works for the purpose of creating a native debootstrap.

fakeroot fakechroot chroot $ROOTDIR /bin/bash

Unfortunately if one tries the same thing on a $ROOTDIR containing non-native binaries one get an unfriendly "Unable to load interpreter" even though qemu-arm-static was copied over. Chrooting into that directory with superuser rights will work - just fakechroot will not. I did some shallow investigation but after some time gave up with a post to the debian-embedded mailinglist

I recently came back to the issue and found out some more about this error: what it says is just that it cannot find the architecture's shared libraries. The same issue occurs when outside a chroot and just running something like:

qemu-arm-static some-non-static-armel-binary

If the binary is a static one that call actually succeeds. Now it is also clearn why it worked with a real chroot and not with a fakechroot since the real chroot will take the libc from $ROOTDIR but fakechroot will take it from the host system. Luckily qemu-arm-static possesses the -L option which lets the user specify the ELF interpreter prefix. This will work:

qemu-arm-static -L $ROOTDIR some-non-static-armel-binary

Since it is not easily possible to supply the -L argument when using binfmt to detect the ELF type I planned to put the content of $ROOTDIR to the default ELF interpreter prefix location. The path given in the man page (/usr/gnemul/qemu-arm) did not work but a strace told me that the actual path is /etc/qemu-binfmt/arm/. After putting my $ROOTDIR there i could even execute shared armel binaries just right away. Hurray!

There was still one issue when using fakeroot/fakechroot, it couldnt find libfakechroot.so and libfakeroot-sysv.so which (according to another strace) it was searching for in /usr/lib/arm-linux-gnueabi/. This makes sense since an armel binary will of course need armel versions of the fakeroot and fakechroot shared libraries as well. After grabbing the armel .debs and extracting the .so files to that directory it finally worked and i could do this for my armel rootfs:

fakeroot fakechroot chroot $ROOTDIR /bin/bash

Some experimentation also showed that i didnt need to put a full $ROOTDIR to /etc/qemu-binfmt/arm/ but ld-2.11.2.so, ld-linux.so.3, libc-2.11.2.so and libc.so.6 in /etc/qemu-binfmt/arm/lib/ where enough.

For future repeatability the setup is:

wget http://ftp.debian.org/debian/pool/main/f/fakechroot/fakechroot_2.14-1_armel.deb
wget http://ftp.debian.org/debian/pool/main/f/fakeroot/fakeroot_1.14.5-1_armel.deb
wget http://ftp.debian.org/debian/pool/main/e/eglibc/libc6_2.11.2-11_armel.deb
sudo mkdir -p /etc/qemu-binfmt/arm/
sudo mkdir -p /usr/lib/arm-linux-gnueabi/
sudo dpkg -x libc6_2.11.2-11_armel.deb /etc/qemu-binfmt/arm/
dpkg-deb --fsys-tarfile fakeroot_1.14.5-1_armel.deb \
| sudo tar -xf - --strip-components=4 -C /usr/lib/arm-linux-gnueabi/ \
./usr/lib/libfakeroot/libfakeroot-sysv.so
dpkg-deb --fsys-tarfile fakechroot_2.14-1_armel.deb \
| sudo tar -xf - --strip-components=4 -C /usr/lib/arm-linux-gnueabi/ \
./usr/lib/fakechroot/libfakechroot.so
rm fakechroot_2.14-1_armel.deb fakeroot_1.14.5-1_armel.deb libc6_2.11.2-11_armel.deb

Should you at some point get errors like "error while loading shared libraries: 4ò@: invalid mode for dlopen(): Invalid argument" then it is just telling you that the libc version in /etc/qemu-binfmt/arm/ does not match the one needed by your fakechroot and you should change it accordingly - probably update to the latest.