minimal qemu-binfmt root with multistrap

Wed, 22 Jun 2011 00:02 categories: code

In this earlier post I listed the following manual method to fill /etc/qemu-binfmt/arm/ with shared libraries of the foreign architecture:

wget http://ftp.debian.org/debian/pool/main/e/eglibc/libc6_2.11.2-11_armel.deb
sudo mkdir -p /etc/qemu-binfmt/arm/
sudo dpkg -x libc6_2.11.2-11_armel.deb /etc/qemu-binfmt/arm/
rm libc6_2.11.2-11_armel.deb

This approach has two disadvantages. Firstly when some other library is required one again has to manually look up the url to the latest deb of the package that contains it and manually extract it to the target directory. Secondly, whenever one wants to upgrade the libraries because of incompatibilities, one has to repeat the whole process. The downloading is hard to script because version numbers and thus, urls to packages change. Fortunately apt can be used to to automatically get the latest binary packages for any architecture and suite. This is the same mechanism that multistrap uses to setup a rootfs. Without further thinking I hacked together a small script that basically does, was the core of what multistrap does:

#!/bin/sh -ex

usage() {
  echo "Usage: $0 arch suite rootdir [mirror]" }

MIRROR="http://127.0.0.1:3142/ftp.de.debian.org/debian"

[ "$#" -ne 3 ] && [ "$#" -ne 4 ] && { usage; exit; }

ARCH="$1"
SUITE="$2"
ROOTDIR="$3"
MIRROR=${4:-$MIRROR}

[ -e "$ROOTDIR" ] && { echo "root directory still exists"; exit; }

mkdir "$ROOTDIR"

ROOTDIR=`realpath "$ROOTDIR"`

# apt options
APT_OPTS="-y"
APT_OPTS=$APT_OPTS" -o Apt::Architecture=$ARCH"
APT_OPTS=$APT_OPTS" -o Dir::Etc::TrustedParts=$ROOTDIR/etc/apt/trusted.gpg.d"
APT_OPTS=$APT_OPTS" -o Dir::Etc::Trusted=$ROOTDIR/etc/apt/trusted.gpg"
APT_OPTS=$APT_OPTS" -o Apt::Get::AllowUnauthenticated=true"
APT_OPTS=$APT_OPTS" -o Apt::Get::Download-Only=true"
APT_OPTS=$APT_OPTS" -o Apt::Install-Recommends=false"
APT_OPTS=$APT_OPTS" -o Dir=$ROOTDIR/"
APT_OPTS=$APT_OPTS" -o Dir::Etc=$ROOTDIR/etc/apt/"
APT_OPTS=$APT_OPTS" -o Dir::Etc::SourceList=$ROOTDIR/etc/apt/sources.list"
APT_OPTS=$APT_OPTS" -o Dir::State=$ROOTDIR/var/lib/apt/"
APT_OPTS=$APT_OPTS" -o Dir::State::Status=$ROOTDIR/var/lib/dpkg/status"
APT_OPTS=$APT_OPTS" -o Dir::Cache=$ROOTDIR/var/cache/apt/"

# initial setup for apt and dpkg to work properly
mkdir -p $ROOTDIR
mkdir -p $ROOTDIR/etc/apt/
mkdir -p $ROOTDIR/etc/apt/sources.list.d/
mkdir -p $ROOTDIR/etc/apt/preferences.d/
mkdir -p $ROOTDIR/var/lib/apt/
mkdir -p $ROOTDIR/var/lib/apt/lists/partial/
mkdir -p $ROOTDIR/var/lib/dpkg/
mkdir -p $ROOTDIR/var/cache/apt/
touch $ROOTDIR/var/lib/dpkg/status

# fill sources.list
echo deb $MIRROR $SUITE main > $ROOTDIR/etc/apt/sources.list

# update and install git and ruby
apt-get $APT_OPTS update
apt-get $APT_OPTS install libc6 libselinux1 libacl1 man-db libstdc++6

# unpack downloaded archives
for deb in $ROOTDIR/var/cache/apt/archives/*.deb; do
        dpkg -x $deb $ROOTDIR
done

# cleanup
rm -rf $ROOTDIR/var/lib/apt/lists
rm -rf $ROOTDIR/var/cache/apt/

One would invoke it like that:

./create-binfmt-tree.sh armel sid binfmt-root

I did something like that before (documented here) so it seemed straight forward to do it like that. It was only after I was done and everything was working when I realized that multistrap has the omitrequired option with which one can do exactly what I wanted: not build a whole rootfs but just get some packages with apt and extract them to a directory. This is the much simpler multistrap config that does the same as the script above:

[General]
arch=
directory=
cleanup=true
unpack=true
noauth=true
aptsources=Debian
bootstrap=Debian
allowrecommends=false
addimportant=false
omitrequired=true

[Debian]
packages=libc6 libselinux1 libacl1 man-db libstdc++6
source=http://ftp.de.debian.org/debian
suite=sid
omitdebsrc=true

Invoking it like that will produce a rootfs I can put into /etc/qemu-binfmt/arm right away:

multistrap -a armel -d binfmt-root -f multistrap-binfmt.conf
View Comments

generate silent wav

Sat, 04 Jun 2011 23:14 categories: code

I wanted a few seconds of complete silent audio. Since I already knew how audio is encoded using LPCM I thought it would be simple enough to write a small snippet that also creates the RIFF file structure around it for a complete *.wav file. And indeed it is. This is how to generate a *.wav file containing some seconds of silence.

#!/usr/bin/python

from struct import pack
from sys import stdout

duration = 1    # seconds of silence
channels = 1    # number of channels
bps = 16        # bits per sample
sample = 44100  # sample rate
ExtraParamSize = 0
Subchunk1Size = 16+2+ExtraParamSize
Subchunk2Size = duration*sample*channels*bps/8
ChunkSize = 4 + (8 + Subchunk1Size) + (8 + Subchunk2Size)

stdout.write("".join([
    'RIFF',                                # ChunkID (magic)      # 0x00
    pack('<I', ChunkSize),                 # ChunkSize            # 0x04
    'WAVE',                                # Format               # 0x08
    'fmt ',                                # Subchunk1ID          # 0x0c
    pack('<I', Subchunk1Size),             # Subchunk1Size        # 0x10
    pack('<H', 1),                         # AudioFormat (1=PCM)  # 0x14
    pack('<H', channels),                  # NumChannels          # 0x16
    pack('<I', sample),                    # SampleRate           # 0x18
    pack('<I', bps/8 * channels * sample), # ByteRate             # 0x1c
    pack('<H', bps/8 * channels),          # BlockAlign           # 0x20
    pack('<H', bps),                       # BitsPerSample        # 0x22
    pack('<H', ExtraParamSize),            # ExtraParamSize       # 0x22
    'data',                                # Subchunk2ID          # 0x24
    pack('<I', Subchunk2Size),             # Subchunk2Size        # 0x28
    '\0'*Subchunk2Size
]))

And because it was fun, the whole thing in shell:

#!/bin/sh

pack_int(){ printf "%08X\n" $1 | sed 's/\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)/\\\\\\x\4\\\\\\x\3\\\\\\x\2\\\\\\x\1/I' | xargs printf; }

pack_short(){ printf "%04X\n" $1 | sed 's/\([0-9A-F]\{2\}\)\([0-9A-F]\{2\}\)/\\\\\\x\2\\\\\\x\1/I' | xargs printf; }

duration=1
channels=1
bps=16
sample=44100
Subchunk1Size=18
Subchunk2Size=$(($duration*$sample*$channels*$bps/8))
ChunkSize=$((20 + $Subchunk1Size + $Subchunk2Size))

echo -n RIFF
pack_int $ChunkSize
echo -n "WAVEfmt "
pack_int $Subchunk1Size
pack_short 1
pack_short $channels
pack_int $sample
pack_int $((bps/8 * channels * sample))
pack_short $((bps/8 * channels))
pack_short $bps
pack_short 0
echo -n data
pack_int $Subchunk2Size

dd if=/dev/zero bs=1 count=$Subchunk2Size 2>/dev/null
View Comments

shell commands in parallel

Sat, 04 Jun 2011 19:58 categories: code

I needed a way to execute a list of commands in parallel. Existing tools like parallel from the moreutils debian package and pexec only allowed to pass the arguments by commandline. This becomes a problem when there are more commands than exec() can handle. You find out that limit with getconf NCARGS. Another issue with them is that they allow only a list of arguments that they append to a given command, not a list of commands to be run in parallel. Also the number of arguments that they can give to that command is limited to one. They also can only execute one command and not a chain of commands separated by semicolon.

What I needed was a program that would sequentially read commands or multiple commands separated by semicolons from a file. One command or chain of them per line and execute them when the overall number of currently executing processes is below a threshold.

The following script reads a file from stdin and does exactly what I want:

#!/bin/sh

NUM=0
QUEUE=""
MAX_NPROC=6

while read CMD; do
    sh -c "$CMD" &

    PID=$!
    QUEUE="$QUEUE $PID"
    NUM=$(($NUM+1))

    # if enough processes were created
    while [ $NUM -ge $MAX_NPROC ]; do
        # check whether any process finished
        for PID in $QUEUE; do
            if [ ! -d /proc/$PID ]; then
                TMPQUEUE=$QUEUE
                QUEUE=""
                NUM=0
                # rebuild new queue from processes
                # that are still alive
                for PID in $TMPQUEUE; do
                    if [ -d /proc/$PID  ]; then
                        QUEUE="$QUEUE $PID"
                        NUM=$(($NUM+1))
                    fi
                done
                break
            fi
        done
        sleep 0.5
    done
done
wait

EDIT: way too late I figured out that what I wanted to do is much easier by just using xargs like this:

cat command_list | xargs --delimiter='\n' --max-args=1 --max-procs=4 sh -c

where -P executes sh in parallel.

View Comments

eject

Fri, 03 Jun 2011 23:21 categories: code

A friend of mine recently gave me her Huawei E1550 umts modem and as many others it first presents itself to the operating system as a usb cdrom drive. It will only switch to serial modem mode when the cdrom is ejected. For this to happen you can fire the eject command and you are done. Since I was curious how linux would eject a cdrom and how eject really works I investigated and found that I can replicate the eject behavior with this C snippet.

#include <sys/ioctl.h>
#include <fcntl.h>
#include <stdlib.h>
#include <errno.h>
#include <stdio.h>
#include <linux/cdrom.h>

int main() {
    int ret, fd;

    fd = open("/dev/sr0", O_NONBLOCK);
    if (fd == -1) {
        perror("open");
        return EXIT_FAILURE;
    }

    ret = ioctl(fd, CDROMEJECT, 0);
    if (ret == -1) {
        perror("ioctl");
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

or alternatively with this python oneliner:

python -c "import fcntl, os; fcntl.ioctl(os.open('/dev/sr0', os.O_NONBLOCK), 0x5309, 0)"
View Comments

youtube video download

Thu, 26 May 2011 10:29 categories: code

Being intimidated by youtube-dl with its over 3000 lines of code, I thought there must be a simpler way than that and wrote a little shell script that now does all I want: download youtube videos.

#!/bin/sh -e

if [ "$#" -ne "1" ]; then
  echo specify the youtube id as the first argument
  exit 1
fi

code="$1"

urldecode() { echo -n $1 | sed 's/%\([0-9A-F]\{2\}\)/\\\\\\\x\1/gI' | xargs printf; }

cookiejar=`mktemp`
baseurl="http://www.youtube.com/get_video_info?video_id=$code&el=detailpage"
data=`curl --silent --cookie-jar "$cookiejar" "$baseurl"`

highestfmt=0
highesturl=""
title=""
for part in `echo $data | tr '&' ' '`; do
  key=`echo $part | cut -d"=" -f1`
  value=`echo $part | cut -d"=" -f2`
  if [ "$value" != "" ]; then
    value=`urldecode "$value"`
  fi    
  case "$key" in
  "fmt_url_map") 
    for format in `echo $value | tr ',' ' '`; do
      fmt=`echo $format | cut -d"|" -f1`
      url=`echo $format | cut -d"|" -f2`
      if [ "$fmt" = "18" ] \
      || [ "$fmt" = "22" ] \
      || [ "$fmt" = "37" ] \
      || [ "$fmt" = "43" ] \
      || [ "$fmt" = "45" ] ; then
        if [ "$fmt" -gt "$highestfmt" ]; then
          highestfmt=$fmt
          highesturl=$url
        fi
      fi
    done ;;
  "title") title="$value" ;;
  esac
done

echo writing output to "${title}_${code}.mp4"
curl --location --cookie "$cookiejar" "$highesturl" > "${title}_${code}.mp4"
rm $cookiejar

and in python because that was so much fun

#!/usr/bin/env python

import cookielib, urllib2, shutil, urlparse, sys

cookie_processor = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
urllib2.install_opener(urllib2.build_opener(cookie_processor))

if len(sys.argv) != 2:
    print "specify the youtube id as the first argument"
    exit(1)

code = sys.argv[1]
baseurl = "http://www.youtube.com/get_video_info?video_id=%s&el=detailpage"%code
data = urllib2.urlopen(baseurl).read()

data = urlparse.parse_qs(data)
title = data["title"][0]
url = dict(part.split('|', 1) for part in data["fmt_url_map"][0].split(','))
url = url.get("37", url.get("22", url.get("18")))

print "writing output to %s_%s.mp4"%(title,code)
data = urllib2.urlopen(url)
with open("%s_%s.mp4"%(title,code), 'wb') as fp:
    shutil.copyfileobj(data, fp, 16*1024)

The shell script can also easily turned into something that will deliver you the video remotely. This is useful if you have the server in the US and get annoyed by all the "This video contains content from ****. It is not available in your country." messages when accessing content e.g. from Germany.

Just change the top part into this:

#!/bin/sh -e

read request

while /bin/true; do
  read header
  [ "$header" = "`printf '\r'`" ] && break
done

code="${request#GET /}"
code="${code% HTTP/*}"

and the bottom part into this:

url=`curl --silent --head --output /dev/null --write-out %{redirect_url} --cookie "$cookiejar" "$highesturl"`
while [ "$url" != "" ]; do
        highesturl=$url
  url=`curl --silent --head --output /dev/null --write-out %{redirect_url} --cookie "$cookiejar" "$highesturl"`
done
curl --silent --include --cookie "$cookiejar" "$highesturl"
rm $cookiejar

then you can run the script like this:

while true; do netcat -l -p 80 -e youtube.sh; done

or by using inetd:

www stream tcp nowait nobody /usr/local/bin/youtube youtube

And better chroot the whole thing.

View Comments
« Older Entries -- Newer Entries »