Mister Muffin Blog

I recently stumbled over some nasty problems with respect to python scopes in for loops and nested functions therein.

Firstly, try out this snippet:

>>> a = []
>>> for i in range(10):
...     a.append(lambda: i)
>>> for f in a: f()

While I expected it to print 0-9 it printed 9 ten times. The reason is twofold.

Firstly it might not be very straightforward but the following should not be surprising:

>>> for i in range(10): pass
>>> i
9

This is, the loop variable is not local to the for loop. There is no new scope created for loops in python. There is only scope for classes and functions.

Secondly, python does late binding with function or lambda calls. The following might be a bit more surprising:

>>> i = 0
>>> f = lambda: i
>>> i = 1
>>> f()
1

Since there is no scope for loops, the following will also print 81 nine times:

>>> a = []
>>> for i in range(10):
...    j = i**2
...    a.append(lambda: j)
>>> for f in a: f()

The problem of course presented itself to me in a much weirder manner which made it take quite some time until I figuered out the root cause of my problem. My problem was, that I indeed expected the first example to print the numbers 0-9 which it doesnt for reasons explained above.

What I was struggeling with, were gtk and dbus callbacks. My code looked like this:

for iface in interfaces:
    def on_succes_cb(msg):
        print iface
    iface.MyDBusMethod(reply_handler=on_success_cb)

This of course printed the last value iface had in this loop on every invocation of the reply_handler. On a sidenote is is surprising to see how sparsely documented the use of reply_handler and error_handler in dbus python is and how seldomly it seems to be used.

Another piece of the same code looked like this:

for func in ["RequestScan", "EnableTechnology", "DisableTechnology"]:
    button = gtk.Button(func)
    def button_onclick(button, event):
        print func
    button.connect("button_press_event", button_onclick)
    hbox.pack_start(button, False, False, 0)

And of course every time the differently named buttons where clicked it would print "DisableTechnology".

So how to fix it?

Lets see how to fix the first example:

>>> lst = []
>>> for i in range(10):
...     lst.append(lambda j=i: j)
>>> for f in lst: f()

What's the difference? The lambda now has it's own local variable j and in contrast to i, the scope of j is local to the lambda. This will now successfully print 0-9.

But this doesnt help me with my above dbus and gtk callback problems as I can't freely change the function signature for the callbacks. So what to do?

The solution Michael 'emdete' Dietrich pointed me to, was to just use a wrapper function around my code which gets the loop variables as its arguments. By doing so, the loop variable gets copied into the function scope and will not be changed there by subsequent loop iterations.

>>> lst = []
>>> for i in range(10):
...     def bind(j):
...             lst.append(lambda: j)
...     bind(i)
>>> for f in lst: f()

or

>>> lst = []
>>> def bind(j):
...     lst.append(lambda: j)
>>> for i in range(10):
...     bind(i)
>>> for f in lst: f()

I agreed with emdete that the second variant looks cleaner. The first variant would have made sense if loops had their own scope but hey, they havent. Using the second variant also avoids confusing with variable usage etc.

So now my code looks like this:

def bind(iface):
    def on_succes_cb(msg):
        print iface
    iface.MyDBusMethod(reply_handler=on_success_cb)
for iface in interfaces:
    bind(iface)

and this:

def bind(func):
    button = gtk.Button(func)
    def button_onclick(button, event):
        print func
    button.connect("button_press_event", button_onclick)
    return button
for func in ["RequestScan", "EnableTechnology", "DisableTechnology"]:
    button = bind(func)
    hbox.pack_start(button, False, False, 0)

To me it always appeared unintuitive that scope is limited to functions and not extended to code blocks. But on the other hand it mimics other languages with side effects:

int i;
for(i=0; i<10; i++);
printf("%d\n", i);

There was a big discussion (64 mails) with several possible solutions on the python-ideas list three years ago: http://mail.python.org/pipermail/python-ideas/2008-October/002109.html

But it doesnt seem as if anything caught on. Hence, for the time being one has to create just another function for new scope. Works for me. Still, even though understanding why and how it works I have trouble finding the first example easily understandable. I would still expect it to work differently.