Python’s Mutable Parameter Trap

I spent an embarrassing amount of time today diagnosing what appeared to be a Heisenbug in Python. I was debugging a method, and each time I added a debug statement to peek at a value, the value itself would be returned and then… just disappear. It was as if observation changed the results. First the problem, then read on for the solution.

Here were the debug messages that illustrate the problem (part of some DHCP-related code):

print "packet.dhcp_options: %s" % (packet.dhcp_options)
print "packet.get_option('router'): %s" % (packet.get_option('router'))
print "packet.dhcp_options: %s" % (packet.dhcp_options)

The output from this code was:

packet.dhcp_options: {'subnet_mask': [255, 255, 255, 0], 'router': [10, 0, 0, 1]}
packet.get_option('router'): 10.0.0.1
packet.dhcp_options: {'subnet_mask': [255, 255, 255, 0], 'router': []}

So obviously something in the get_option method must be overwriting that value, right? Let’s have a look:

def get_option(self, name):
    """ Get DHCP options"""

    option_num = self.get_option_number(name)
    option_type = dhcp.DHCP_OPTION_TYPES[option_num]['type']
    return bytelist_to_value(option_type, self.dhcp_options[name])

OK, nothing obvious that would cause reassignment of dhcp_options[‘router’], let’s have a look at the bytelist_to_value() method:

def bytelist_to_value(optype, bytelist):
    new_value = ''
    bytelist.reverse()
    while len(bytelist) > 3:
        if new_value:  # if there is already at least 1 IP,
            new_value += ', '  # append a comma between addresses
        for i in range(0,3):
            new_value += str(bytelist.pop()) + "."
        new_value += str(bytelist.pop())
    return new_value

At first glance, this looks OK, we take the input ‘bytelist’ and treat it like a local variable, doing a bit of manipulation on it before returning the formatted value.  So the question is: why was my value being overwritten each time I retrieved the value? Finally it dawned on me.

The problem happens when I call bytelist.reverse() and bytelist.pop(). I was popping all the members off the list, and it was being modified in place. Python passes the actual object in a method call, not just the value(s) of the object. If I perform manipulation on a list, I’m actually changing the actual list itself and not a local copy. This behavior doesn’t manifest with immutable types in Python, only with mutable types. This is easily demonstrated:

>>> def foo(x):
...   x += 5
...   print x
... 
>>> x = 5
>>> foo(x)
10
>>> x
5

>>> def bar(y):
...   print y.pop()
... 
>>> y = [1,2,3]
>>> bar(y)
3
>>> y
[1, 2]

In Python you can efficiently make a copy of list L with ‘L[:]’. The [:] corresponds to all members of the list, so instead of acting on the actual list we are making modifications to a new object with the same values.

Changing my code in bytelist_to_value() to make a copy of the list solved the problem:

def bytelist_to_value(optype, bytelist):
    new_value = ''
    __bytelist = bytelist[:]
    __bytelist.reverse()
    while len(__bytelist) > 3:
        if new_value:  # if there is already at least 1 IP,
            new_value += ', '  # append a comma between addresses
        for i in range(0,3):
            new_value += str(__bytelist.pop()) + "."
        new_value += str(__bytelist.pop())
    return new_value