Mister Muffin Blog

Since I debootstrap a lot, I download the same base packages from the debian server over and over again. This sucks as it eats up the bandwidth of the debian people and is slow for me. The solution is apt-cacher that acts as a local debian repository and totally automagically even works for foreign architectures or distributions.

After apt-getting it, the only things I did to make it run, was to edit /etc/apt-cacher/apt-cacher.conf and /etc/default/apt-cacher. In apt-cacher.conf I had to remove the asterisk from the allowed_hosts setting and also set generate_reports to 0. Then setting AUTOSTART to 1 in the default file and from that point on I just have to use the following repository mirror URL in my bootstrapping scripts:

http://localhost:3142/ftp.de.debian.org/debian

The beauty is, that I dont have to configure apt-cacher to source http://ftp.de.debian.org beforehand - it's all done by a request of the above style. Would I want apt-cacher to choose a different mirror, I would just request another url from it. This way I can transparently cache any debian mirror I want without any hassle.

I use GNU screen a lot and most importantly on my servers so that a session is not lost after logging out. One of the screen features I use is the tiling feature. By pressing "C-a S" the terminal is split vertically into two regions. Unfortunately once come back after having detached from the session I manually have to split again and put my terminals on the upper and lower region. This is a whole lot of unnecessary keystrokes each time I connect to my server. Screen has no (and will not have) functionality to keep track of the tiling for each session so a hack to still preserve it, is to run a nested screen where the innermost screen is tiled but one only connects or detaches to the outer screen. This again has of course the disadvantage of longer keyboard shortcuts once one wants to manipulate the innermost screen. There are of course solutions like a different screen config for the outer screen and such but it still feels ugly.

So since that really botherd me and I heard of tmux which promised more functionality, better code, easier handling (modifiable keyboard shortcuts) and most importantly: it offered the functionality that I was looking for as explained above. Long story short: it works like a charm and I guess I have constantly switched now.

One thing to notice for users of the 1.3 version of tmux in Debian based operating systems: if you want to use the supplied screen configuration so that you have screen keyboard shortcuts in tmux (I do that) then after copying over /usr/share/doc/tmux/examples/screen-keys.conf to /etc/tmux.conf or ~/.tmux.conf, replace the down-pane command on line 96 with select-pane -U. You might also want to delete the entries for BTab.

A friend of mine got also saved by tmux - he was complaining that when using GNU screens multi-display-mode (screen -x) then if the one user changes terminals, this change is not done for the other. That sucks a lot if you want to help someone over a screen session. With tmux - this works out of the box :)

One drawback nonetheless is, that this hack to display a libnotify message on my laptop screen from a remote host over a ssh connection using the urxvt print-pipe doesnt work anymore. The problem is that tmux doesnt understand the \033[5i escape. NicM from #tmux said that printing something like "\033Ptmux;\033\033[5i\033\" with tmux HEAD would work but i didnt try it out. Since this approach doesnt work anymore I now switched to simple taskbar highlighting for notifying me that stuff is going on in one of my remote hosts. For this to work i just had to set "URxvt.urgentOnBell: true" in my ~/.Xdefaults. Whenever the bell character "\a" is printed the window will now flash. In my jabber client mcabber i also have to set "set beep_on_message = 1" and comment the "#set events_command" line that was putting the print escapes to stderr before. Using awesome wm, switching to the workspace that requires attention is now even quicker by using the Modkey+U shortcut.

Creating a new debian rootfs is fun and easy - you would just call:

sudo debootstrap --variant=minbase sid $ROOTDIR http://ftp.debian.org/debian

or

sudo cdebootstrap --flavour=minimal sid $ROOTDIR http://ftp.debian.org/debian

You could then either chroot(8) into that directory or even put the whole thing on an SD card or harddisk and boot from it (with a few additional modifications of course).

A recent addition to the family is multistrap that uses dpkg and apt to assemble the rootfs and hence benefits from the power, flexibility and reliability of these tools. For example it is possible to mix different repositories.

sudo multistrap -f /usr/share/multistrap/sid.conf -d $ROOTDIR

Now all three build a rootfs of the host's native architecture by default but there is also the option of building a rootfs for a foreign architecture. In my case I mostly build for armel from now on I will use armel as an example for a foreign architecture.

sudo debootstrap --foreign --arch=armel ...
sudo cdebootstrap --foreign --arch=armel ...
sudo multistrap -a armel ...

For all three of them there is a second stage that will require you to be able to execute the binary format of the target architecture.

cdebootstrap puts the second stage installer into $ROOTDIR/sbin/cdebootstrap-foreign and also $ROOTDIR/sbin/init links to it so that on a possible boot on the native hardware cdebootstrap-foreign would be executed to finish the bootstrapping. The real init is saved into $ROOTDIR/sbin/init.foreign.

debootstrap will be invoked with the --second-stage option on the $ROOTDIR directory on the target architecture to invoke the second stage.

multistrap requires you to run "dpkg --configure -a" in a native environment.

Now running the first stage on your host and the second stage on the device is a tedious thing, so I always ended up doing the full bootstrapping on the native hardware to avoid the hassle. Since I build for armel the target hardware is mostly quite a bit slower than my amd64 host - in terms of raw processing power and also in terms of I/O speed due to slow flash storage. Also I would prefer to use my mobile phone for actual telephony and not misuse it as a buildhost. To solve the problem I first used qemu. I set up a virtual machine using qemu-system-arm and inside that did the bootstrapping nativeley.

While the slowness issue is mostly solved by this (emulation is slow but bootstrapping benefits from the much better I/O capabilities of my host) it still remains bothersome to always have to boot a whole virtual machine which is kind of an overkill.

The problem is solved by qemu user mode emulation. Upon encountering a foreign armel ELF binary qemu-arm-static will be invoked to execute it using the kernel's binfmt mechanism. This has the advantage that one can now run code of arbitrary architectures on any amd64/i386 host without emulating a full machine but by only emulating the foreign instructions. This again is a huge increase in simplicity and execution speed.

All one has to do is to run the first stage of one of the above methods to bootstrap a debian rootfs, then copy the qemu-arm-static binary inside it to $ROOTDIR/usr/bin and then just chrooting to it.

sudo multistrap -a armel ...
sudo /usr/bin/qemu-arm-static $ROOTDIR/usr/bin
sudo chroot $ROOTDIR dpkg --configure -a
sudo chroot $ROOTDIR /bin/bash

The last two calls that invoke dpkg and bash inside the chroot environment are actually invoking armel binaries that are interpreted by qemu-arm-static on the fly. It feels as if the system would be a native one - very cool!

So at this point I thought I had found the holy grail of creating a foreign debian rootfs as I now had just one script that would invoke multistrap, copy qemu-arm-static over, configure dpkg, would do some additional foo, remove qemu-arm-static and pack everything in a tarball. But one thing still bothered me.

The end result i wanted to get out of the whole process was a tarball or maybe a squashfs image. Now both are just two files on my harddrive containing information about the contained files and directories, their permissions and so on but basically just a bunch of bytes in a special structure. So why would I need superuser rights to create this amount of bytes in my home directory?

The first stage of all three methods can be executed with fakeroot but for the second stage a call to chroot(2) is required which fakeroot doesnt implement. fakechroot comes to the rescue and imitates a chrooted environtment. It's not perfect but it works for the purpose of creating a native debootstrap.

fakeroot fakechroot chroot $ROOTDIR /bin/bash

Unfortunately if one tries the same thing on a $ROOTDIR containing non-native binaries one get an unfriendly "Unable to load interpreter" even though qemu-arm-static was copied over. Chrooting into that directory with superuser rights will work - just fakechroot will not. I did some shallow investigation but after some time gave up with a post to the debian-embedded mailinglist

I recently came back to the issue and found out some more about this error: what it says is just that it cannot find the architecture's shared libraries. The same issue occurs when outside a chroot and just running something like:

qemu-arm-static some-non-static-armel-binary

If the binary is a static one that call actually succeeds. Now it is also clearn why it worked with a real chroot and not with a fakechroot since the real chroot will take the libc from $ROOTDIR but fakechroot will take it from the host system. Luckily qemu-arm-static possesses the -L option which lets the user specify the ELF interpreter prefix. This will work:

qemu-arm-static -L $ROOTDIR some-non-static-armel-binary

Since it is not easily possible to supply the -L argument when using binfmt to detect the ELF type I planned to put the content of $ROOTDIR to the default ELF interpreter prefix location. The path given in the man page (/usr/gnemul/qemu-arm) did not work but a strace told me that the actual path is /etc/qemu-binfmt/arm/. After putting my $ROOTDIR there i could even execute shared armel binaries just right away. Hurray!

There was still one issue when using fakeroot/fakechroot, it couldnt find libfakechroot.so and libfakeroot-sysv.so which (according to another strace) it was searching for in /usr/lib/arm-linux-gnueabi/. This makes sense since an armel binary will of course need armel versions of the fakeroot and fakechroot shared libraries as well. After grabbing the armel .debs and extracting the .so files to that directory it finally worked and i could do this for my armel rootfs:

fakeroot fakechroot chroot $ROOTDIR /bin/bash

Some experimentation also showed that i didnt need to put a full $ROOTDIR to /etc/qemu-binfmt/arm/ but ld-2.11.2.so, ld-linux.so.3, libc-2.11.2.so and libc.so.6 in /etc/qemu-binfmt/arm/lib/ where enough.

For future repeatability the setup is:

wget http://ftp.debian.org/debian/pool/main/f/fakechroot/fakechroot_2.14-1_armel.deb
wget http://ftp.debian.org/debian/pool/main/f/fakeroot/fakeroot_1.14.5-1_armel.deb
wget http://ftp.debian.org/debian/pool/main/e/eglibc/libc6_2.11.2-11_armel.deb
sudo mkdir -p /etc/qemu-binfmt/arm/
sudo mkdir -p /usr/lib/arm-linux-gnueabi/
sudo dpkg -x libc6_2.11.2-11_armel.deb /etc/qemu-binfmt/arm/
dpkg-deb --fsys-tarfile fakeroot_1.14.5-1_armel.deb \
| sudo tar -xf - --strip-components=4 -C /usr/lib/arm-linux-gnueabi/ \
./usr/lib/libfakeroot/libfakeroot-sysv.so
dpkg-deb --fsys-tarfile fakechroot_2.14-1_armel.deb \
| sudo tar -xf - --strip-components=4 -C /usr/lib/arm-linux-gnueabi/ \
./usr/lib/fakechroot/libfakechroot.so
rm fakechroot_2.14-1_armel.deb fakeroot_1.14.5-1_armel.deb libc6_2.11.2-11_armel.deb

Should you at some point get errors like "error while loading shared libraries: 4ò@: invalid mode for dlopen(): Invalid argument" then it is just telling you that the libc version in /etc/qemu-binfmt/arm/ does not match the one needed by your fakechroot and you should change it accordingly - probably update to the latest.