minimal qemu-binfmt root with multistrap

Wed, 22 Jun 2011 00:02 categories: code

In this earlier post I listed the following manual method to fill /etc/qemu-binfmt/arm/ with shared libraries of the foreign architecture:

wget http://ftp.debian.org/debian/pool/main/e/eglibc/libc6_2.11.2-11_armel.deb
sudo mkdir -p /etc/qemu-binfmt/arm/
sudo dpkg -x libc6_2.11.2-11_armel.deb /etc/qemu-binfmt/arm/
rm libc6_2.11.2-11_armel.deb

This approach has two disadvantages. Firstly when some other library is required one again has to manually look up the url to the latest deb of the package that contains it and manually extract it to the target directory. Secondly, whenever one wants to upgrade the libraries because of incompatibilities, one has to repeat the whole process. The downloading is hard to script because version numbers and thus, urls to packages change. Fortunately apt can be used to to automatically get the latest binary packages for any architecture and suite. This is the same mechanism that multistrap uses to setup a rootfs. Without further thinking I hacked together a small script that basically does, was the core of what multistrap does:

#!/bin/sh -ex

usage() {
  echo "Usage: $0 arch suite rootdir [mirror]" }

MIRROR="http://127.0.0.1:3142/ftp.de.debian.org/debian"

[ "$#" -ne 3 ] && [ "$#" -ne 4 ] && { usage; exit; }

ARCH="$1"
SUITE="$2"
ROOTDIR="$3"
MIRROR=${4:-$MIRROR}

[ -e "$ROOTDIR" ] && { echo "root directory still exists"; exit; }

mkdir "$ROOTDIR"

ROOTDIR=`realpath "$ROOTDIR"`

# apt options
APT_OPTS="-y"
APT_OPTS=$APT_OPTS" -o Apt::Architecture=$ARCH"
APT_OPTS=$APT_OPTS" -o Dir::Etc::TrustedParts=$ROOTDIR/etc/apt/trusted.gpg.d"
APT_OPTS=$APT_OPTS" -o Dir::Etc::Trusted=$ROOTDIR/etc/apt/trusted.gpg"
APT_OPTS=$APT_OPTS" -o Apt::Get::AllowUnauthenticated=true"
APT_OPTS=$APT_OPTS" -o Apt::Get::Download-Only=true"
APT_OPTS=$APT_OPTS" -o Apt::Install-Recommends=false"
APT_OPTS=$APT_OPTS" -o Dir=$ROOTDIR/"
APT_OPTS=$APT_OPTS" -o Dir::Etc=$ROOTDIR/etc/apt/"
APT_OPTS=$APT_OPTS" -o Dir::Etc::SourceList=$ROOTDIR/etc/apt/sources.list"
APT_OPTS=$APT_OPTS" -o Dir::State=$ROOTDIR/var/lib/apt/"
APT_OPTS=$APT_OPTS" -o Dir::State::Status=$ROOTDIR/var/lib/dpkg/status"
APT_OPTS=$APT_OPTS" -o Dir::Cache=$ROOTDIR/var/cache/apt/"

# initial setup for apt and dpkg to work properly
mkdir -p $ROOTDIR
mkdir -p $ROOTDIR/etc/apt/
mkdir -p $ROOTDIR/etc/apt/sources.list.d/
mkdir -p $ROOTDIR/etc/apt/preferences.d/
mkdir -p $ROOTDIR/var/lib/apt/
mkdir -p $ROOTDIR/var/lib/apt/lists/partial/
mkdir -p $ROOTDIR/var/lib/dpkg/
mkdir -p $ROOTDIR/var/cache/apt/
touch $ROOTDIR/var/lib/dpkg/status

# fill sources.list
echo deb $MIRROR $SUITE main > $ROOTDIR/etc/apt/sources.list

# update and install git and ruby
apt-get $APT_OPTS update
apt-get $APT_OPTS install libc6 libselinux1 libacl1 man-db libstdc++6

# unpack downloaded archives
for deb in $ROOTDIR/var/cache/apt/archives/*.deb; do
        dpkg -x $deb $ROOTDIR
done

# cleanup
rm -rf $ROOTDIR/var/lib/apt/lists
rm -rf $ROOTDIR/var/cache/apt/

One would invoke it like that:

./create-binfmt-tree.sh armel sid binfmt-root

I did something like that before (documented here) so it seemed straight forward to do it like that. It was only after I was done and everything was working when I realized that multistrap has the omitrequired option with which one can do exactly what I wanted: not build a whole rootfs but just get some packages with apt and extract them to a directory. This is the much simpler multistrap config that does the same as the script above:

[General]
arch=
directory=
cleanup=true
unpack=true
noauth=true
aptsources=Debian
bootstrap=Debian
allowrecommends=false
addimportant=false
omitrequired=true

[Debian]
packages=libc6 libselinux1 libacl1 man-db libstdc++6
source=http://ftp.de.debian.org/debian
suite=sid
omitdebsrc=true

Invoking it like that will produce a rootfs I can put into /etc/qemu-binfmt/arm right away:

multistrap -a armel -d binfmt-root -f multistrap-binfmt.conf
View Comments

new toy

Mon, 06 Jun 2011 16:41 categories: blog

Finally my new toy arrived: Seagate FreeAgent GoFlex Net

seagate goflex net

  • 1.2 GHz Marvell Kirkwood 88F6281
  • 128MB RAM
  • 256MB NAND
  • 1x GbEthernet
  • 1x USB 2.0
  • 2x SATA
  • 2.3 W power consumption

I acquired it for 40 EUR which is quite a bargain for a computer with these specs.

I already found the pin headers for RX and TX so hacking it can start right after I got myself a new serial usb adapter.

In the photo above, there is a 10-pin header. In the upper row, the three pins from the left are GND, TX and RX. With screen I can then use:

screen /dev/ttyUSB0 115200
View Comments

latex with blogofile

Sat, 04 Jun 2011 23:29 categories: blog

I can now finally add latex formulas to this blog :)

I am using this filter.

To test, have some maxwell:

\begin{aligned} \nabla \cdot \mathbf{D} &= \rho_f \\ \nabla \cdot \mathbf{B} &= 0 \\ \nabla \times \mathbf{E} &= -\frac{\partial \mathbf{B}} {\partial t} \\ \nabla \times \mathbf{H} &= \mathbf{J}_f + \frac{\partial \mathbf{D}} {\partial t} \\ \end{aligned}
View Comments

generate silent wav

Sat, 04 Jun 2011 23:14 categories: code

I wanted a few seconds of complete silent audio. Since I already knew how audio is encoded using LPCM I thought it would be simple enough to write a small snippet that also creates the RIFF file structure around it for a complete *.wav file. And indeed it is. This is how to generate a *.wav file containing some seconds of silence.

#!/usr/bin/python

from struct import pack
from sys import stdout

duration = 1    # seconds of silence
channels = 1    # number of channels
bps = 16        # bits per sample
sample = 44100  # sample rate
ExtraParamSize = 0
Subchunk1Size = 16+2+ExtraParamSize
Subchunk2Size = duration*sample*channels*bps/8
ChunkSize = 4 + (8 + Subchunk1Size) + (8 + Subchunk2Size)

stdout.write("".join([
    'RIFF',                                # ChunkID (magic)      # 0x00
    pack('<I', ChunkSize),                 # ChunkSize            # 0x04
    'WAVE',                                # Format               # 0x08
    'fmt ',                                # Subchunk1ID          # 0x0c
    pack('<I', Subchunk1Size),             # Subchunk1Size        # 0x10
    pack('<H', 1),                         # AudioFormat (1=PCM)  # 0x14
    pack('<H', channels),                  # NumChannels          # 0x16
    pack('<I', sample),                    # SampleRate           # 0x18
    pack('<I', bps/8 * channels * sample), # ByteRate             # 0x1c
    pack('<H', bps/8 * channels),          # BlockAlign           # 0x20
    pack('<H', bps),                       # BitsPerSample        # 0x22
    pack('<H', ExtraParamSize),            # ExtraParamSize       # 0x22
    'data',                                # Subchunk2ID          # 0x24
    pack('<I', Subchunk2Size),             # Subchunk2Size        # 0x28
    '\0'*Subchunk2Size
]))

And because it was fun, the whole thing in shell:

#!/bin/sh

pack_int(){ printf "%08X\n" $1 | sed 's/\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)/\\\\\\x\4\\\\\\x\3\\\\\\x\2\\\\\\x\1/I' | xargs printf; }

pack_short(){ printf "%04X\n" $1 | sed 's/\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)/\\\\\\x\2\\\\\\x\1/I' | xargs printf; }

duration=1
channels=1
bps=16
sample=44100
Subchunk1Size=18
Subchunk2Size=$(($duration*$sample*$channels*$bps/8))
ChunkSize=$((20 + $Subchunk1Size + $Subchunk2Size))

echo -n RIFF
pack_int $ChunkSize
echo -n "WAVEfmt "
pack_int $Subchunk1Size
pack_short 1
pack_short $channels
pack_int $sample
pack_int $((bps/8 * channels * sample))
pack_short $((bps/8 * channels))
pack_short $bps
pack_short 0
echo -n data
pack_int $Subchunk2Size

dd if=/dev/zero bs=1 count=$Subchunk2Size 2>/dev/null
View Comments

shell commands in parallel

Sat, 04 Jun 2011 19:58 categories: code

I needed a way to execute a list of commands in parallel. Existing tools like parallel from the moreutils debian package and pexec only allowed to pass the arguments by commandline. This becomes a problem when there are more commands than exec() can handle. You find out that limit with getconf NCARGS. Another issue with them is that they allow only a list of arguments that they append to a given command, not a list of commands to be run in parallel. Also the number of arguments that they can give to that command is limited to one. They also can only execute one command and not a chain of commands separated by semicolon.

What I needed was a program that would sequentially read commands or multiple commands separated by semicolons from a file. One command or chain of them per line and execute them when the overall number of currently executing processes is below a threshold.

The following script reads a file from stdin and does exactly what I want:

#!/bin/sh

NUM=0
QUEUE=""
MAX_NPROC=6

while read CMD; do
    sh -c "$CMD" &

    PID=$!
    QUEUE="$QUEUE $PID"
    NUM=$(($NUM+1))

    # if enough processes were created
    while [ $NUM -ge $MAX_NPROC ]; do
        # check whether any process finished
        for PID in $QUEUE; do
            if [ ! -d /proc/$PID ]; then
                TMPQUEUE=$QUEUE
                QUEUE=""
                NUM=0
                # rebuild new queue from processes
                # that are still alive
                for PID in $TMPQUEUE; do
                    if [ -d /proc/$PID  ]; then
                        QUEUE="$QUEUE $PID"
                        NUM=$(($NUM+1))
                    fi
                done
                break
            fi
        done
        sleep 0.5
    done
done
wait

EDIT: way too late I figured out that what I wanted to do is much easier by just using xargs like this:

cat command_list | xargs --delimiter='\n' --max-args=1 --max-procs=4 sh -c

where -P executes sh in parallel.

View Comments
« Older Entries