Mister Muffin Blog

Sitting in a lonely text file somewhere on my harddrive let me finally write down some useful commands that allowed me to use my bluetoooth devices with bluez 4.

apt-get install bluez bluez-gstreamer bluez-alsa

discovery

hcitool scan
python /usr/share/doc/bluez/examples/test-discovery

listing adapters

hciconfig
python /usr/share/doc/bluez/examples/list-devices
dbus-send --system --dest=org.bluez --print-reply / org.bluez.Manager.ListAdapters
dbus-send --system --dest=org.bluez --print-reply $path org.bluez.Adapter.GetProperties

$path will be something along the lines of /org/bluez/2199/hci0 and it's printed by the first dbus-send.

listing device details

dbus-send --system --dest=org.bluez --print-reply $devpath org.bluez.Device.GetProperties
dbus-send --system --dest=org.bluez --print-reply $devpath/node org.bluez.Node.GetProperties

where $devpath is an entry of the Devices array of the Adapter.GetProperties call before and looks like: /org/bluez/2199/hci0/dev_00_11_22_AA_BB_CC

connecting a hid device

dbus-send --system --dest=org.bluez --print-reply $devpath org.bluez.Input.Connect

$devpath as above.

pairing a device

python /usr/share/doc/bluez/examples/simple-agent hci0 $address

$address was discovered by the first step above and is the device address like: 00:11:22:AA:BB:CC

removing a device

python /usr/share/doc/bluez/examples/simple-agent hci0 $address remove

trusting a device

python /usr/share/doc/bluez/examples/test-device trusted $address yes

playing sound

gst-launch-0.10 filesrc location=recit.mp3 ! mad ! audioconvert ! sbcenc ! a2dpsink device=$address
gconftool -t string -s /system/gstreamer/0.10/default/musicaudiosink "sbcenc ! a2dpsink device=$address"

Being intimidated by youtube-dl with its over 3000 lines of code, I thought there must be a simpler way than that and wrote a little shell script that now does all I want: download youtube videos.

#!/bin/sh -e

if [ "$#" -ne "1" ]; then
  echo specify the youtube id as the first argument
  exit 1
fi

code="$1"

urldecode() { echo -n $1 | sed 's/%\([0-9A-F]\{2\}\)/\\\\\\\x\1/gI' | xargs printf; }

cookiejar=`mktemp`
baseurl="http://www.youtube.com/get_video_info?video_id=$code&el=detailpage"
data=`curl --silent --cookie-jar "$cookiejar" "$baseurl"`

highestfmt=0
highesturl=""
title=""
for part in `echo $data | tr '&' ' '`; do
  key=`echo $part | cut -d"=" -f1`
  value=`echo $part | cut -d"=" -f2`
  if [ "$value" != "" ]; then
    value=`urldecode "$value"`
  fi    
  case "$key" in
  "fmt_url_map") 
    for format in `echo $value | tr ',' ' '`; do
      fmt=`echo $format | cut -d"|" -f1`
      url=`echo $format | cut -d"|" -f2`
      if [ "$fmt" = "18" ] \
      || [ "$fmt" = "22" ] \
      || [ "$fmt" = "37" ] \
      || [ "$fmt" = "43" ] \
      || [ "$fmt" = "45" ] ; then
        if [ "$fmt" -gt "$highestfmt" ]; then
          highestfmt=$fmt
          highesturl=$url
        fi
      fi
    done ;;
  "title") title="$value" ;;
  esac
done

echo writing output to "${title}_${code}.mp4"
curl --location --cookie "$cookiejar" "$highesturl" > "${title}_${code}.mp4"
rm $cookiejar

and in python because that was so much fun

#!/usr/bin/env python

import cookielib, urllib2, shutil, urlparse, sys

cookie_processor = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
urllib2.install_opener(urllib2.build_opener(cookie_processor))

if len(sys.argv) != 2:
    print "specify the youtube id as the first argument"
    exit(1)

code = sys.argv[1]
baseurl = "http://www.youtube.com/get_video_info?video_id=%s&el=detailpage"%code
data = urllib2.urlopen(baseurl).read()

data = urlparse.parse_qs(data)
title = data["title"][0]
url = dict(part.split('|', 1) for part in data["fmt_url_map"][0].split(','))
url = url.get("37", url.get("22", url.get("18")))

print "writing output to %s_%s.mp4"%(title,code)
data = urllib2.urlopen(url)
with open("%s_%s.mp4"%(title,code), 'wb') as fp:
    shutil.copyfileobj(data, fp, 16*1024)

The shell script can also easily turned into something that will deliver you the video remotely. This is useful if you have the server in the US and get annoyed by all the "This video contains content from ****. It is not available in your country." messages when accessing content e.g. from Germany.

Just change the top part into this:

#!/bin/sh -e

read request

while /bin/true; do
  read header
  [ "$header" = "`printf '\r'`" ] && break
done

code="${request#GET /}"
code="${code% HTTP/*}"

and the bottom part into this:

url=`curl --silent --head --output /dev/null --write-out %{redirect_url} --cookie "$cookiejar" "$highesturl"`
while [ "$url" != "" ]; do
        highesturl=$url
  url=`curl --silent --head --output /dev/null --write-out %{redirect_url} --cookie "$cookiejar" "$highesturl"`
done
curl --silent --include --cookie "$cookiejar" "$highesturl"
rm $cookiejar

then you can run the script like this:

while true; do netcat -l -p 80 -e youtube.sh; done

or by using inetd:

www stream tcp nowait nobody /usr/local/bin/youtube youtube

And better chroot the whole thing.

I'm planning to go biking and swimming again in the summer. What bugs me about it, is that during both activities I can't do much more than just that. I can't work on my projects or read a book. What I could do is listen to music or to a audio book. But what I would really like to do most is to continue reading my Perry Rhodan books. There are audio recordings of them but unfortunately those are edited and not a 1:1 copy of the books. But since my goal was to really read all of the books without leaving anything out this was unacceptable.

My idea was to see how far we are in terms of text to speech synthesizers. To increase the requirements I was looking for a good German synthesizer since the Perry Rhodan novels are written in German. After some search I found out that apparently a company called ivona is currently providing the best possible solution for my problem.

But of course there were additional issues. First, this company of course doesn't release their product for linux so I had two options: installing windows in qemu and running their software there or seeing if I could find out how I could find a new purpose for their online demo application.

While I was setting up an emulated Windows XP in the background I tried to figure out how the webfrontend of ivona.com was working. After a few hours of trial and error I got the solution which in contrast to my initial believe did not even require any reverse engineering of flash binaries.

string="Wenn die Testpiloten und Spezialisten des Linearkommandos von dem
neuartigen Kompensationskonverter zur Errichtung eines aus Sechsdimensional
übergeordneten Feldlinien bestehenden Kompensatorfeldes sprachen, dann gab sich
niemand mehr die Mühe, die zungenbrecherischen Begriffe exakt auszusprechen.";
mplayer `curl --data-urlencode "tresc=$string" --data synth=1
--data chk=\`echo -n $string | perl -pe 'use MIME::Base64;
$_=MIME::Base64::encode($_);tr/\+\/=/-_./;s/\n//g'\` --data voice=20
--header "X-Requested-With: XMLHttpRequest"
http://www.ivona.com/index.php | sed "s/.*loadAudioFile('\(.*\)').*/\1/"`

The X-Requested-With header is required and instead of 20, the voice POST parameter can become 21 for a female German voice. Other numbers are for other languages.

Interestingly enough, the 250 character limit seems to be only enforced on the client side. As one can see with the request above it is no problem at all to convert phrases with more characters as well. Instead the server will just stop converting at a random point. It will be around 23-26 seconds of speech or 180-200 KB. But maybe this is useful anyways for somebody.

In my quest to minimize the external third party services I rely on in my daily life, I stumbled over dudle which is an online poll system like doodle only better :)

The problems with giving your availability for a meeting or preferences over a subject to third party services like doodle are obviously the entailed privacy issues. Since I thought that doing stuff like a simple poll can also be handled by software running on my machine I firstly just wanted something like doodle running on one of my servers. This way, whenever me or anybody else would vote, they would no longer be depending on trusting someone like doodle but they would only have to trust me, who I own the server. (and hopefully they trust me more than a unknown money making instance)

But one can do even better than that! Benjamin Kellermann implemented an online poll platform where the availability or preferences of the participants are not even known by the server itself (nor by the other participants) and only when everybody voted a sum of all the votes can be calculated to decide for a timeslot or choice of subject. Hence, the user doesn't even have to trust the server he uses that runs dudle. All the server and the other participants get to see are encrypted availability vectors from which nobody can infer the original choice of options of the user. Only when combining all of them, a sum can be calculated which represents the overall availability of all users at a given time.

By its usage dudle is as simple as doodle. The only thing that might sound tedious is the private key each user has to take care of after his registration (which of course works without giving away any private detail like email) but this is greatly solved by using a bookmarklet to enter his private key into a poll field. I was showing the setup and what its advantages are to some non-CS people and was very pleased by the positive responses I got. A big kudos to Benjamin for his great work on this piece of software!

I am running thttpd on mister-muffin.de and by design it chroots into the www directory to increase security. A result of this is, that the only cgi scripts that can be run are statically compiled executables. Dudle is written in ruby and uses git as the database backend. Hence I had to setup a minimal chroot environment inside my www directory so that dynamically linked executables like git and ruby would work. I could've bothered with compiling both statically but was not up for the trouble (yet). A requirement of this environment was of course that it was very small and only included stuff that was necessary for git and ruby to run: dynamic libraries and ruby modules. Another requirement was that there were no setuid programs that could be used by an attacker to break out of the chroot environment by becoming root.

One way to do that would be to do a normal debootstrap including git and ruby and then manually removing everything that was not needed. It turned out that a normal debootstrap creates lots of overhead in retrieving lots of things that are not needed anyways and result in also having to delete lots of things afterwards, which is not worth the hassle.

My idea was to retrieve the git and ruby debian packages and all its dependencies and all the dependencies of those recursively and then just extract those packages into a directory. Since I didnt want to do the dependency resolution manually I let myself be inspired by multistrap and used apt to do that. Using apt for this task (as well as for multistrap) is possible because one can specify a custom target directory for apt in the commandline.

Since my final setup did not contain /bin/sh which ruby needed to call git, I had to patch dudle. I also proposed a way to get rid of the htdigest dependency to Benjamin and he included that and my /bin/sh patch into dudle. ☺

#!/bin/sh -ex

# check for fakeroot
if [ "$LOGNAME" = "root" ] \
|| [ "$USER" = "root" ] \
|| [ "$USERNAME" = "root" ] \
|| [ "$SUDO_COMMAND" != "" ] \
|| [ "$SUDO_USER" != "" ] \
|| [ "$SUDO_UID" != "" ] \
|| [ "$SUDO_GID" != "" ]; then
        echo "don't run this script as root - there is no need to"
        exit
fi

# modify these
ARCH="amd64"
DIST="squeeze"
MIRROR="http://127.0.0.1:3142/ftp.de.debian.org/debian"
DIRECTORY="`pwd`/debian-$DIST-$ARCH-ministrap"

# re-execute script in fakeroot
if [ "$FAKEROOTKEY" = "" ]; then
        echo "re-executing script inside fakeroot"
        fakeroot $0;
        rsync -Phaze ssh $DIRECTORY/ mister-muffin.de:/var/www/
        ssh mister-muffin.de "chown -R www-data:www-data /var/www/dudle.mister-muffin.de/"
        exit
fi

# apt options
APT_OPTS="-y"
APT_OPTS=$APT_OPTS" -o Apt::Architecture=$ARCH"
APT_OPTS=$APT_OPTS" -o Dir::Etc::TrustedParts=$DIRECTORY/etc/apt/trusted.gpg.d"
APT_OPTS=$APT_OPTS" -o Dir::Etc::Trusted=$DIRECTORY/etc/apt/trusted.gpg"
APT_OPTS=$APT_OPTS" -o Apt::Get::AllowUnauthenticated=true"
APT_OPTS=$APT_OPTS" -o Apt::Get::Download-Only=true"
APT_OPTS=$APT_OPTS" -o Apt::Install-Recommends=false"
APT_OPTS=$APT_OPTS" -o Dir=$DIRECTORY/"
APT_OPTS=$APT_OPTS" -o Dir::Etc=$DIRECTORY/etc/apt/"
APT_OPTS=$APT_OPTS" -o Dir::Etc::SourceList=$DIRECTORY/etc/apt/sources.list"
APT_OPTS=$APT_OPTS" -o Dir::State=$DIRECTORY/var/lib/apt/"
APT_OPTS=$APT_OPTS" -o Dir::State::Status=$DIRECTORY/var/lib/dpkg/status"
APT_OPTS=$APT_OPTS" -o Dir::Cache=$DIRECTORY/var/cache/apt/"

# clean root directory
rm -rf $DIRECTORY

# initial setup for apt to work properly
mkdir -p $DIRECTORY
mkdir -p $DIRECTORY/etc/apt/
mkdir -p $DIRECTORY/etc/apt/sources.list.d/
mkdir -p $DIRECTORY/etc/apt/preferences.d/
mkdir -p $DIRECTORY/var/lib/apt/
mkdir -p $DIRECTORY/var/lib/apt/lists/partial/
mkdir -p $DIRECTORY/var/lib/dpkg/
mkdir -p $DIRECTORY/var/cache/apt/
# apt somehow needs this file to be present
touch $DIRECTORY/var/lib/dpkg/status

# fill sources.list
echo deb $MIRROR $DIST main > $DIRECTORY/etc/apt/sources.list

# update and install git and ruby
apt-get $APT_OPTS update
apt-get $APT_OPTS install ruby git-core libgettext-ruby1.8 libjson-ruby1.8

# unpack downloaded archives
for deb in $DIRECTORY/var/cache/apt/archives/*.deb; do
  dpkg -x $deb $DIRECTORY
done

# delete obsolete directories
rm -rf $DIRECTORY/usr/share/
rm -rf $DIRECTORY/usr/lib/perl/
rm -rf $DIRECTORY/usr/lib/gconv/
rm -rf $DIRECTORY/usr/lib/git-core/
rm -rf $DIRECTORY/usr/sbin/
rm -rf $DIRECTORY/var/
rm -rf $DIRECTORY/bin/
rm -rf $DIRECTORY/sbin/
rm -rf $DIRECTORY/selinux/
rm -rf $DIRECTORY/etc/*

# delete all setuid programs
find $DIRECTORY -perm -4000 -delete

# delete all binaries except for "git" and "ruby"
find $DIRECTORY/usr/bin/ -type f -o -type l | egrep -v "ruby|git$" | xargs rm -rf

# git needs /etc/passwd otherwise git says: "You dont't exist, go away!"
cat > $DIRECTORY/etc/passwd << __END__
www-data:x:33:33:www-data:/var/www:/bin/sh
__END__

# dont forget to create /tmp directory for dudle
mkdir -m 777 $DIRECTORY/tmp

# get latest dudle
bzr branch https://dudle.inf.tu-dresden.de/unstable/ $DIRECTORY/dudle.mister-muffin.de
( cd $DIRECTORY/dudle.mister-muffin.de; make; )
bzr branch https://dudle.inf.tu-dresden.de/unstable/extensions/dc-net/ $DIRECTORY/dudle.mister-muffin.de/extensions/dc-net/
( cd $DIRECTORY/dudle.mister-muffin.de/extensions/dc-net/; make; )

# fix shebang
find $DIRECTORY/dudle.mister-muffin.de/ -type f -regex ".*\.cgi\|.*\.rb" \
    | xargs sed -i 's/#!\/usr\/bin\/env ruby/#!\/usr\/bin\/ruby/'

The above code will compile a minimal chroot environment, delete everything that is not needed, fetch dudle and deploy it to my server. The comments in the code should explain everything.

It is simple to change the font size of rxvt-unicode via an entry in ~/.Xdefaults:

URxvt*font:             xft:DejaVu Sans Mono-8

But it is also possible to change the font on the fly via this escape sequence:

printf '\33]50;%s\007' "xft:DejaVu Sans Mono-8"

While playing around with that I also found out that it is very simple to change the font of any other terminal you know the PID of, by just doing something like:

printf '\33]50;%s\007' "xft:DejaVu Sans Mono-8" > /proc/$pid/fd/0

where $pid is the process id of the shell you are running.