Mister Muffin Blog

Creating a new debian rootfs is fun and easy - you would just call:

sudo debootstrap --variant=minbase sid $ROOTDIR http://ftp.debian.org/debian

or

sudo cdebootstrap --flavour=minimal sid $ROOTDIR http://ftp.debian.org/debian

You could then either chroot(8) into that directory or even put the whole thing on an SD card or harddisk and boot from it (with a few additional modifications of course).

A recent addition to the family is multistrap that uses dpkg and apt to assemble the rootfs and hence benefits from the power, flexibility and reliability of these tools. For example it is possible to mix different repositories.

sudo multistrap -f /usr/share/multistrap/sid.conf -d $ROOTDIR

Now all three build a rootfs of the host's native architecture by default but there is also the option of building a rootfs for a foreign architecture. In my case I mostly build for armel from now on I will use armel as an example for a foreign architecture.

sudo debootstrap --foreign --arch=armel ...
sudo cdebootstrap --foreign --arch=armel ...
sudo multistrap -a armel ...

For all three of them there is a second stage that will require you to be able to execute the binary format of the target architecture.

cdebootstrap puts the second stage installer into $ROOTDIR/sbin/cdebootstrap-foreign and also $ROOTDIR/sbin/init links to it so that on a possible boot on the native hardware cdebootstrap-foreign would be executed to finish the bootstrapping. The real init is saved into $ROOTDIR/sbin/init.foreign.

debootstrap will be invoked with the --second-stage option on the $ROOTDIR directory on the target architecture to invoke the second stage.

multistrap requires you to run "dpkg --configure -a" in a native environment.

Now running the first stage on your host and the second stage on the device is a tedious thing, so I always ended up doing the full bootstrapping on the native hardware to avoid the hassle. Since I build for armel the target hardware is mostly quite a bit slower than my amd64 host - in terms of raw processing power and also in terms of I/O speed due to slow flash storage. Also I would prefer to use my mobile phone for actual telephony and not misuse it as a buildhost. To solve the problem I first used qemu. I set up a virtual machine using qemu-system-arm and inside that did the bootstrapping nativeley.

While the slowness issue is mostly solved by this (emulation is slow but bootstrapping benefits from the much better I/O capabilities of my host) it still remains bothersome to always have to boot a whole virtual machine which is kind of an overkill.

The problem is solved by qemu user mode emulation. Upon encountering a foreign armel ELF binary qemu-arm-static will be invoked to execute it using the kernel's binfmt mechanism. This has the advantage that one can now run code of arbitrary architectures on any amd64/i386 host without emulating a full machine but by only emulating the foreign instructions. This again is a huge increase in simplicity and execution speed.

All one has to do is to run the first stage of one of the above methods to bootstrap a debian rootfs, then copy the qemu-arm-static binary inside it to $ROOTDIR/usr/bin and then just chrooting to it.

sudo multistrap -a armel ...
sudo /usr/bin/qemu-arm-static $ROOTDIR/usr/bin
sudo chroot $ROOTDIR dpkg --configure -a
sudo chroot $ROOTDIR /bin/bash

The last two calls that invoke dpkg and bash inside the chroot environment are actually invoking armel binaries that are interpreted by qemu-arm-static on the fly. It feels as if the system would be a native one - very cool!

So at this point I thought I had found the holy grail of creating a foreign debian rootfs as I now had just one script that would invoke multistrap, copy qemu-arm-static over, configure dpkg, would do some additional foo, remove qemu-arm-static and pack everything in a tarball. But one thing still bothered me.

The end result i wanted to get out of the whole process was a tarball or maybe a squashfs image. Now both are just two files on my harddrive containing information about the contained files and directories, their permissions and so on but basically just a bunch of bytes in a special structure. So why would I need superuser rights to create this amount of bytes in my home directory?

The first stage of all three methods can be executed with fakeroot but for the second stage a call to chroot(2) is required which fakeroot doesnt implement. fakechroot comes to the rescue and imitates a chrooted environtment. It's not perfect but it works for the purpose of creating a native debootstrap.

fakeroot fakechroot chroot $ROOTDIR /bin/bash

Unfortunately if one tries the same thing on a $ROOTDIR containing non-native binaries one get an unfriendly "Unable to load interpreter" even though qemu-arm-static was copied over. Chrooting into that directory with superuser rights will work - just fakechroot will not. I did some shallow investigation but after some time gave up with a post to the debian-embedded mailinglist

I recently came back to the issue and found out some more about this error: what it says is just that it cannot find the architecture's shared libraries. The same issue occurs when outside a chroot and just running something like:

qemu-arm-static some-non-static-armel-binary

If the binary is a static one that call actually succeeds. Now it is also clearn why it worked with a real chroot and not with a fakechroot since the real chroot will take the libc from $ROOTDIR but fakechroot will take it from the host system. Luckily qemu-arm-static possesses the -L option which lets the user specify the ELF interpreter prefix. This will work:

qemu-arm-static -L $ROOTDIR some-non-static-armel-binary

Since it is not easily possible to supply the -L argument when using binfmt to detect the ELF type I planned to put the content of $ROOTDIR to the default ELF interpreter prefix location. The path given in the man page (/usr/gnemul/qemu-arm) did not work but a strace told me that the actual path is /etc/qemu-binfmt/arm/. After putting my $ROOTDIR there i could even execute shared armel binaries just right away. Hurray!

There was still one issue when using fakeroot/fakechroot, it couldnt find libfakechroot.so and libfakeroot-sysv.so which (according to another strace) it was searching for in /usr/lib/arm-linux-gnueabi/. This makes sense since an armel binary will of course need armel versions of the fakeroot and fakechroot shared libraries as well. After grabbing the armel .debs and extracting the .so files to that directory it finally worked and i could do this for my armel rootfs:

fakeroot fakechroot chroot $ROOTDIR /bin/bash

Some experimentation also showed that i didnt need to put a full $ROOTDIR to /etc/qemu-binfmt/arm/ but ld-2.11.2.so, ld-linux.so.3, libc-2.11.2.so and libc.so.6 in /etc/qemu-binfmt/arm/lib/ where enough.

For future repeatability the setup is:

wget http://ftp.debian.org/debian/pool/main/f/fakechroot/fakechroot_2.14-1_armel.deb
wget http://ftp.debian.org/debian/pool/main/f/fakeroot/fakeroot_1.14.5-1_armel.deb
wget http://ftp.debian.org/debian/pool/main/e/eglibc/libc6_2.11.2-11_armel.deb
sudo mkdir -p /etc/qemu-binfmt/arm/
sudo mkdir -p /usr/lib/arm-linux-gnueabi/
sudo dpkg -x libc6_2.11.2-11_armel.deb /etc/qemu-binfmt/arm/
dpkg-deb --fsys-tarfile fakeroot_1.14.5-1_armel.deb \
| sudo tar -xf - --strip-components=4 -C /usr/lib/arm-linux-gnueabi/ \
./usr/lib/libfakeroot/libfakeroot-sysv.so
dpkg-deb --fsys-tarfile fakechroot_2.14-1_armel.deb \
| sudo tar -xf - --strip-components=4 -C /usr/lib/arm-linux-gnueabi/ \
./usr/lib/fakechroot/libfakechroot.so
rm fakechroot_2.14-1_armel.deb fakeroot_1.14.5-1_armel.deb libc6_2.11.2-11_armel.deb

Should you at some point get errors like "error while loading shared libraries: 4ò@: invalid mode for dlopen(): Invalid argument" then it is just telling you that the libc version in /etc/qemu-binfmt/arm/ does not match the one needed by your fakechroot and you should change it accordingly - probably update to the latest.