oneliner on Mister Muffin Blog

transposing csv for gnuplot

Wed, 23 Nov 2011 11:49:00 +0000

I recently got a csv that was exported from openoffice spreadsheet with data arranged in rows and not columns as gnuplot likes it. It seems that gnuplot (intentionally) lacks the ability to parse data in rows instead of columns. Hence I had to switch rows and columns (transpose) my input csv such that gnuplot likes it.

Transposing whitespace delimetered text can be done with awk but csv is a bit more complex as it allows quotes and escapes. So a solution had to be found which understood how to read csv.

This turned out to be so simple and minimalistic that I had to post the resulting oneliner that did the job for me:

python -c'import csv,sys;csv.writer(sys.stdout,csv.excel_tab).writerows(map(None,*list(csv.reader(sys.stdin))))'

This will read input csv from stdin and output the transpose to stdout. The transpose is done by using:

map(None, *thelist)

Another way to do the transpose in python is by using:

zip(*thelist)

But this solution doesnt handle rows of different length well.

In addition the solution above will output the csv tab delimetered instead of using commas as gnuplot likes it by using the excel_tab dialect in the csv.writer.

The solution above is problematic when some of the input values inbetween are empty. It is not problematic because the csv would be transposed incorrectly but because gnuplot collapses several whitespaces into one. There are several solutions to that problem. Either, instead of an empty cell, insert “-” in the output:

python -c'import csv,sys; csv.writer(sys.stdout, csv.excel_tab).writerows(map(lambda *x:map(lambda x:x or "-",x),*list(csv.reader(sys.stdin))))'

Or output a comma delimetered cvs and tell gnupot that the input is comma delimetered:

python -c'import csv,sys;csv.writer(sys.stdout).writerows(map(None,*list(csv.reader(sys.stdin))))'

And then in gnuplot:

set datafile separator ","

converting filename charset

Thu, 08 Sep 2011 06:41:00 +0000

So I was copying files on a vfat-formatted usb stick when this error popped up:

Invalid or incomplete multibyte or wide character

Same issue with cp, rsync and tar. So I just thought this would be an issue with stupid vfat so I mkfs.ntfs the drive and tried again - same issue.

So while I thought that at least NTFS would allow any character except NULL I was proven wrong. Looking it up in wikipedia indeed yielded that ntfs file names are limited to UTF-16 code points - in contrast to ext2/3/4, reiserfs, Btrfs, XFS and surely even more which allow any byte except NULL.

So what my files contained was a number of iso 8859-1 or latin1 characters like 0xe8 (è), 0xe0 (à), 0xf9 (ù) or 0xec (ì). It might also have been Windows-1252 but apparently the bytes between 0x80 and 0x9F didnt appear.

This is how to finally batch-convert character sets in filenames:

convmv -f latin1 -t utf8 --notest *.flac

audio conversion, splitting, tagging

Wed, 07 Sep 2011 15:44:00 +0000

I had some operas lying around amongst others Wagner’s “Ring Des Nibelungen”, Puccini’s “La Boheme” and Verdi’s “La Traviata”. Sadly, some of them were not stored as flac but as monkey’s audio (extension .ape) which is a non-free lossless codec but fortunately ffmpeg has a GPL’d decoder included. Additionally some CD’s were stored as a single file with a cue sheet next to them.

So my task was: convert ape to flac, split audio by information from cue files and tag everything correctly.

These were the lines I used to split the audio:

ffmpeg -i CD1.ape CD1.wav
shnsplit -o flac -f CD1.cue -t "%a %n %t" CD1.wav
rm *pregap.flac CD1.wav CD1.ape
cuetag CD1.cue *.flac

First ape is converted to wav so that shnsplit can read it. Shnsplit will then take the timing information from the cue sheet and split the wav into chunks which it conveniently also converts to flac on the fly. Sadly it doesnt do tagging of those files so this is done by cuetag afterward with information from the cue sheet. You need the shntool and cuetools packages.

Other data was already existing as separate tracks and only had to be converted from ape to flac. Sadly flac offers no means for batch conversion but using a for loop and basename the same effect is created. Conveniently ffmpeg will copy the tags from the ape files to the new flac files as well.

for f in *.ape; do b=`basename $f .ape`; ffmpeg -i "$b".ape "$b".flac; done

The resulting flac files were about 3-5% larger than the ape files which is a totally acceptable tradeoff for being able to ditch an unfree format.

thomasg pointed out to me, that using the zsh there is an even neater way to do this loop without basename(1) and without some shell-loop keywords:

for f in *.ape; ffmpeg -i $f $f:r.flac

converting base64 on the commandline

Sat, 03 Sep 2011 20:11:00 +0000

There is an interesting amount of ways to do that. First line is encoding, second line is decoding.

using python:

python -m base64 -e < infile > outfile
python -m base64 -d < infile > outfile

using openssl (-a is short for -base64):

openssl enc -a -e < infile > outfile
openssl enc -a -d < infile > outfile

using base64 (part of the coreutils package):

base64 < infile > outfile
base64 -d < infile > outfile

using perl:

perl -MMIME::Base64 -ne 'print encode_base64($_)' < infile > outfile
perl -MMIME::Base64 -ne 'print encode_base64($_)' < infile > outfile

using uuencode (but adds a header and footer):

uuencode -m - < infile > outfile
uudecode < infile > outfile

convert mailman archive to mbox

Tue, 26 Jul 2011 11:36:00 +0000

The mailman mailing list manager allows to download monthly archives in “Gzip’d Text” format. This format is not mbox but can easily be turned into it by the following simple line of sed:

sed 's/^\(From:\? .*\) \(at\|en\) /\1@/'

This makes it much easier to browse paste emails of an archive eg. using mutt or reply to past emails with a proper Message-ID as to not break threads.

text to speech

Sat, 21 May 2011 02:44:00 +0000

I’m planning to go biking and swimming again in the summer. What bugs me about it, is that during both activities I can’t do much more than just that. I can’t work on my projects or read a book. What I could do is listen to music or to a audio book. But what I would really like to do most is to continue reading my Perry Rhodan books. There are audio recordings of them but unfortunately those are edited and not a 1:1 copy of the books. But since my goal was to really read all of the books without leaving anything out this was unacceptable.

My idea was to see how far we are in terms of text to speech synthesizers. To increase the requirements I was looking for a good German synthesizer since the Perry Rhodan novels are written in German. After some search I found out that apparently a company called ivona is currently providing the best possible solution for my problem.

But of course there were additional issues. First, this company of course doesn’t release their product for linux so I had two options: installing windows in qemu and running their software there or seeing if I could find out how I could find a new purpose for their online demo application.

While I was setting up an emulated Windows XP in the background I tried to figure out how the webfrontend of ivona.com was working. After a few hours of trial and error I got the solution which in contrast to my initial believe did not even require any reverse engineering of flash binaries.

string="Wenn die Testpiloten und Spezialisten des Linearkommandos von dem
neuartigen Kompensationskonverter zur Errichtung eines aus Sechsdimensional
übergeordneten Feldlinien bestehenden Kompensatorfeldes sprachen, dann gab sich
niemand mehr die Mühe, die zungenbrecherischen Begriffe exakt auszusprechen.";
mplayer `curl --data-urlencode "tresc=$string" --data synth=1
--data chk=\`echo -n $string | perl -pe	'use MIME::Base64;
$_=MIME::Base64::encode($_);tr/\+\/=/-_./;s/\n//g'\` --data voice=20
--header "X-Requested-With: XMLHttpRequest"
http://www.ivona.com/index.php | sed "s/.*loadAudioFile('\(.*\)').*/\1/"`

The X-Requested-With header is required and instead of 20, the voice POST parameter can become 21 for a female German voice. Other numbers are for other languages.

Interestingly enough, the 250 character limit seems to be only enforced on the client side. As one can see with the request above it is no problem at all to convert phrases with more characters as well. Instead the server will just stop converting at a random point. It will be around 23-26 seconds of speech or 180-200 KB. But maybe this is useful anyways for somebody.

silent ssh-keygen

Fri, 25 Mar 2011 22:01:00 +0000

I sometimes have to tell friends to connect to me via ssh to get some files or similar things but they mostly did not yet generate a private/public keypair for themselves.

So for me to remember the command to generate that in the standard place without requiring any user input that would just confuse them here is the command:

ssh-keygen -q -t rsa -f ~/.ssh/id_rsa -N ""

download from loombo

Thu, 24 Mar 2011 07:58:00 +0000

I currently use http://watchseries.eu to check out some series I later might watch in full in better quality. It provides links to several offerings from video streaming web services of a whole lot of series. One of them is megaupload and a howto download from there will follow but this article is about a quick one-liner to download from “loombo.com” which has the advantage that in contrast to megavideo there is no daily limit imposed. The sleep 6 is important as the webserver seems to use that delay to make sure a human is clicking a button after that timeout and not a machine as doing it. The id variable comes from the url of the loombo video you want.

id=0ioy51cxit1b; wget `curl --silent --data "op=download2&id=$id&rand=\`curl \
--silent http://loombo.com/$id | sed -ne 's/<input type=\"hidden\" \
name=\"rand\" value=\"\(\w\+\)\">/\1/p'; sleep 6\`" http://loombo.com/$id | \
sed -ne "s/s1.addVariable('file','\(\S\+\)');/\1/p"`