Mister Muffin Blog

I needed a way to execute a list of commands in parallel. Existing tools like parallel from the moreutils debian package and pexec only allowed to pass the arguments by commandline. This becomes a problem when there are more commands than exec() can handle. You find out that limit with getconf NCARGS. Another issue with them is that they allow only a list of arguments that they append to a given command, not a list of commands to be run in parallel. Also the number of arguments that they can give to that command is limited to one. They also can only execute one command and not a chain of commands separated by semicolon.

What I needed was a program that would sequentially read commands or multiple commands separated by semicolons from a file. One command or chain of them per line and execute them when the overall number of currently executing processes is below a threshold.

The following script reads a file from stdin and does exactly what I want:

#!/bin/sh

NUM=0
QUEUE=""
MAX_NPROC=6

while read CMD; do
    sh -c "$CMD" &

    PID=$!
    QUEUE="$QUEUE $PID"
    NUM=$(($NUM+1))

    # if enough processes were created
    while [ $NUM -ge $MAX_NPROC ]; do
        # check whether any process finished
        for PID in $QUEUE; do
            if [ ! -d /proc/$PID ]; then
                TMPQUEUE=$QUEUE
                QUEUE=""
                NUM=0
                # rebuild new queue from processes
                # that are still alive
                for PID in $TMPQUEUE; do
                    if [ -d /proc/$PID  ]; then
                        QUEUE="$QUEUE $PID"
                        NUM=$(($NUM+1))
                    fi
                done
                break
            fi
        done
        sleep 0.5
    done
done
wait

EDIT: way too late I figured out that what I wanted to do is much easier by just using xargs like this:

cat command_list | xargs --delimiter='\n' --max-args=1 --max-procs=4 sh -c

where -P executes sh in parallel.

A friend of mine recently gave me her Huawei E1550 umts modem and as many others it first presents itself to the operating system as a usb cdrom drive. It will only switch to serial modem mode when the cdrom is ejected. For this to happen you can fire the eject command and you are done. Since I was curious how linux would eject a cdrom and how eject really works I investigated and found that I can replicate the eject behavior with this C snippet.

#include <sys/ioctl.h>
#include <fcntl.h>
#include <stdlib.h>
#include <errno.h>
#include <stdio.h>
#include <linux/cdrom.h>

int main() {
    int ret, fd;

    fd = open("/dev/sr0", O_NONBLOCK);
    if (fd == -1) {
        perror("open");
        return EXIT_FAILURE;
    }

    ret = ioctl(fd, CDROMEJECT, 0);
    if (ret == -1) {
        perror("ioctl");
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

or alternatively with this python oneliner:

python -c "import fcntl, os; fcntl.ioctl(os.open('/dev/sr0', os.O_NONBLOCK), 0x5309, 0)"

When I learned about css media queries from Debian Developer Vincent Bernat's blog I made a few changes to the css of this site as well.

I increased the content width from 600px to 700px and made use of this construct throughout the css:

media screen and (max-width: 700px) {#content {width:100%;}}

This piece will overwrite the width property of the content div from static 700px to variable 100% of the window size and by that consuming whatever the sub-700px-width screen size leaves for the main content. The positive effect of this measure is that on screens below a width of 700px, no horizontal scrolling will be needed to read the main content. It will be cramped in whatever is offered. This is of course most usable on mobile devices. Horizontal scrolling will only be needed to access the main menu. I could put the menu at the bottom but decided that scrolling sidewards will be less hassle than scrolling to the bottom of a page.

I not only changed the content width for screens below 700px width but also adapted margins and paddings from 1em to barely noticeable 1 pixel.

Try to resize your browser window to below 700px to see the effect - it's pretty nifty and works without any javascript on browsers that understand that css property.

Sitting in a lonely text file somewhere on my harddrive let me finally write down some useful commands that allowed me to use my bluetoooth devices with bluez 4.

apt-get install bluez bluez-gstreamer bluez-alsa

discovery

hcitool scan
python /usr/share/doc/bluez/examples/test-discovery

listing adapters

hciconfig
python /usr/share/doc/bluez/examples/list-devices
dbus-send --system --dest=org.bluez --print-reply / org.bluez.Manager.ListAdapters
dbus-send --system --dest=org.bluez --print-reply $path org.bluez.Adapter.GetProperties

$path will be something along the lines of /org/bluez/2199/hci0 and it's printed by the first dbus-send.

listing device details

dbus-send --system --dest=org.bluez --print-reply $devpath org.bluez.Device.GetProperties
dbus-send --system --dest=org.bluez --print-reply $devpath/node org.bluez.Node.GetProperties

where $devpath is an entry of the Devices array of the Adapter.GetProperties call before and looks like: /org/bluez/2199/hci0/dev_00_11_22_AA_BB_CC

connecting a hid device

dbus-send --system --dest=org.bluez --print-reply $devpath org.bluez.Input.Connect

$devpath as above.

pairing a device

python /usr/share/doc/bluez/examples/simple-agent hci0 $address

$address was discovered by the first step above and is the device address like: 00:11:22:AA:BB:CC

removing a device

python /usr/share/doc/bluez/examples/simple-agent hci0 $address remove

trusting a device

python /usr/share/doc/bluez/examples/test-device trusted $address yes

playing sound

gst-launch-0.10 filesrc location=recit.mp3 ! mad ! audioconvert ! sbcenc ! a2dpsink device=$address
gconftool -t string -s /system/gstreamer/0.10/default/musicaudiosink "sbcenc ! a2dpsink device=$address"

Being intimidated by youtube-dl with its over 3000 lines of code, I thought there must be a simpler way than that and wrote a little shell script that now does all I want: download youtube videos.

#!/bin/sh -e

if [ "$#" -ne "1" ]; then
  echo specify the youtube id as the first argument
  exit 1
fi

code="$1"

urldecode() { echo -n $1 | sed 's/%\([0-9A-F]\{2\}\)/\\\\\\\x\1/gI' | xargs printf; }

cookiejar=`mktemp`
baseurl="http://www.youtube.com/get_video_info?video_id=$code&el=detailpage"
data=`curl --silent --cookie-jar "$cookiejar" "$baseurl"`

highestfmt=0
highesturl=""
title=""
for part in `echo $data | tr '&' ' '`; do
  key=`echo $part | cut -d"=" -f1`
  value=`echo $part | cut -d"=" -f2`
  if [ "$value" != "" ]; then
    value=`urldecode "$value"`
  fi    
  case "$key" in
  "fmt_url_map") 
    for format in `echo $value | tr ',' ' '`; do
      fmt=`echo $format | cut -d"|" -f1`
      url=`echo $format | cut -d"|" -f2`
      if [ "$fmt" = "18" ] \
      || [ "$fmt" = "22" ] \
      || [ "$fmt" = "37" ] \
      || [ "$fmt" = "43" ] \
      || [ "$fmt" = "45" ] ; then
        if [ "$fmt" -gt "$highestfmt" ]; then
          highestfmt=$fmt
          highesturl=$url
        fi
      fi
    done ;;
  "title") title="$value" ;;
  esac
done

echo writing output to "${title}_${code}.mp4"
curl --location --cookie "$cookiejar" "$highesturl" > "${title}_${code}.mp4"
rm $cookiejar

and in python because that was so much fun

#!/usr/bin/env python

import cookielib, urllib2, shutil, urlparse, sys

cookie_processor = urllib2.HTTPCookieProcessor(cookielib.CookieJar())
urllib2.install_opener(urllib2.build_opener(cookie_processor))

if len(sys.argv) != 2:
    print "specify the youtube id as the first argument"
    exit(1)

code = sys.argv[1]
baseurl = "http://www.youtube.com/get_video_info?video_id=%s&el=detailpage"%code
data = urllib2.urlopen(baseurl).read()

data = urlparse.parse_qs(data)
title = data["title"][0]
url = dict(part.split('|', 1) for part in data["fmt_url_map"][0].split(','))
url = url.get("37", url.get("22", url.get("18")))

print "writing output to %s_%s.mp4"%(title,code)
data = urllib2.urlopen(url)
with open("%s_%s.mp4"%(title,code), 'wb') as fp:
    shutil.copyfileobj(data, fp, 16*1024)

The shell script can also easily turned into something that will deliver you the video remotely. This is useful if you have the server in the US and get annoyed by all the "This video contains content from ****. It is not available in your country." messages when accessing content e.g. from Germany.

Just change the top part into this:

#!/bin/sh -e

read request

while /bin/true; do
  read header
  [ "$header" = "`printf '\r'`" ] && break
done

code="${request#GET /}"
code="${code% HTTP/*}"

and the bottom part into this:

url=`curl --silent --head --output /dev/null --write-out %{redirect_url} --cookie "$cookiejar" "$highesturl"`
while [ "$url" != "" ]; do
        highesturl=$url
  url=`curl --silent --head --output /dev/null --write-out %{redirect_url} --cookie "$cookiejar" "$highesturl"`
done
curl --silent --include --cookie "$cookiejar" "$highesturl"
rm $cookiejar

then you can run the script like this:

while true; do netcat -l -p 80 -e youtube.sh; done

or by using inetd:

www stream tcp nowait nobody /usr/local/bin/youtube youtube

And better chroot the whole thing.