035 Some Common Problems

035 Some Common Problems#

COM6018

Copyright © 2023, 2024 Jon Barker, University of Sheffield. All rights reserved.

1. Introducing#

This notebook considers some common Python and NumPy programming issues that you may encounter. The list is not exhaustive, but is based on personal experience. If you have any suggestions for other common problems to include, please let me know.

Below we will import NumPy which will be used in some of the examples that follow.

import numpy as np

2. Comparing NumPy arrays#

How do we check whether two arrays are equal? The obvious answer is to use the == operator, but this does not work as expected:

x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
if x == y:
    print("X and Y are equal")
else:
    print("X and Y are not equal")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 3
      1 x = np.array([1, 2, 3])
      2 y = np.array([4, 5, 6])
----> 3 if x == y:
      4     print("X and Y are equal")
      5 else:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

The reason is that the == operator works on the individual elements of the array, returning an array of booleans:

x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
print(x == y)

[False False False]

The if statement strictly requires a single Boolean value and not an array of Booleans.

The NumPy methods all() and any() can be used to reduce an array of Booleans to a single Boolean value. i.e., we can write x.all() or x.any(). As the names suggest, all() will only return True if all the elements in the array x are True, whereas any() returns True if any of the elements are True.

So to compare a pair of arrays, we can write code like the following:

x = np.array([1, 2, 3])
y = np.array([1, 5, 6])

if (x == y).all():
    print("X and Y are equal")
else:
    print("X and Y are not equal")

if (x == y).any():
    print("Some elements of X and Y are equal")
else:
    print("No elements of X and Y are equal")

X and Y are not equal
Some elements of X and Y are equal

Note, any() and all() can be used in many contexts; e.g., see the example below which checks if elements in one array are greater than the corresponding elements in another array:

x = np.array([4, 5, 1])
y = np.array([1, 2, 3])

if (x > y).all():
    print("Print all elements of X are greater than Y")
elif (x > y).any():
    print("Print some elements of X are greater than Y")
else:
    print("Print no elements of X are greater than Y")

Print some elements of X are greater than Y

3. Mutating a function’s argument#

If we want to keep our programmes simple and easy to debug, we should strive to write ‘pure functions’ where possible.

Pure functions have the following properties:

  • The function return values should depend deterministically on its input values (i.e., given input values will always lead to the same output).

  • The function must have no ‘side effects’ (i.e. it should not change external variables).

It is, however, very easy to write functions that accidentally break this second rule… and this can lead to subtle bugs and unexpected behaviour.

For example, consider the following trivial function that simply multiplies its input by 2 and returns a result.

def times_2(x):
    x*= 2
    return x

This looks fine at first glance, and it seems to be behaving as expected:

x = 10

print("x before function call:", x)
y = times_2(x)
print("y after function call:", y)
print("x after function call:", x)

x before function call: 10
y after function call: 20
x after function call: 10

But now consider what happens if we pass a NumPy array to the function:

x = np.array([10, 20, 30])

print("x before function call:", x)
y = times_2(x)
print("y after function call:", y)
print("x after function call:", x)

x before function call: [10 20 30]
y after function call: [20 40 60]
x after function call: [20 40 60]

Notice how the value of x has changed after the function call. This is because the function has mutated the value of x. This is a side effect and breaks the second rule of pure functions. Someone using this function would not have expected that x had been changed and might be using x again later in their code, expecting it to still contain [10, 20, 30].

There are several solutions. The simplest fix to this particular example is just to rewrite the function as,

def times_2(x):
    y = x * 2  # y will be a new array, the orignal x is left unchanged
    return y

This is a good solution, but it is not always possible to avoid mutating the input arguments. For example, consider the following function that adds a new element to a list:

def append_to_list(x, y):
    x.append(y)
    return x

We will now use the function to append 40 to the list [10, 20, 30]:

x = [10, 20, 30]

print("x before function call:", x)
y = append_to_list(x, 40)
print("y after function call:", y)
print("x after function call:", x)

x before function call: [10, 20, 30]
y after function call: [10, 20, 30, 40]
x after function call: [10, 20, 30, 40]

Notice again that the value of x has changed after the function call.

In an example list this, if we want to ensure that append_to_list is a pure function, we need to change it so that it makes an explicit copy of the input list before appending the new element. This can be done using the copy module:

import copy
def append_to_list(x, y):
    x_copy = copy.copy(x)
    x_copy.append(y)
    return x_copy

The function will now behave as expected:

x = [10, 20, 30]

print("x before function call:", x)
y = append_to_list(x, 40)
print("y after function call:", y)
print("x after function call:", x)

x before function call: [10, 20, 30]
y after function call: [10, 20, 30, 40]
x after function call: [10, 20, 30]

The downside of this approach is that it is inefficient as it requires the creation of a new copy of the list. This is not a problem for small lists, but for large lists it could be a problem.

If we were concerned about efficiency then we might allow the function to mutate the input list, but we would need to document this behaviour carefully. The user could then make a copy of the argument before using the function if they needed to keep the original list unchanged, i.e. you might end up with code like this.

def append_to_list(x, y):
    """Note, this function mutates the input list x"""
    x.append(y)
    return x

import copy
x = [10, 20, 30]
print("x before function call:", x)
x_copy = copy.copy(x) # Make a copy of x that we can pass to the function
y = append_to_list(x_copy, 40)
print("y after function call:", y)
print("x after function call:", x)

x before function call: [10, 20, 30]
y after function call: [10, 20, 30, 40]
x after function call: [10, 20, 30]

4. Confusing Append and Extend#

This is a very simple issue, but I have included it because it is the source of many beginner errors.

The append method adds a single element to the end of a list, so, for example, let us say that we want to attach the number 4 to the end of the list [1, 2, 3]:

my_list = [1, 2, 3]
my_list.append(4)
print(my_list)

[1, 2, 3, 4]

This is fine and works exactly as expected. But now say I want to add the numbers [4, 5, 6] to the list. Beginners often try to do this using the append method:

my_list = [1, 2, 3]
my_list.append([4, 5, 6])

The code will run and everything seems to go fine. But let us now look at the result,

print(my_list)

[1, 2, 3, [4, 5, 6]]

…oops. This is not quite what we wanted. The numbers 4, 5 and 6 have not been added to the list, instead the list [4, 5, 6] has been added as a single element to the end of the list. i.e., rather than ending up with a list of 6 elements, we have ended up with a list of 4 elements, where the last element is itself a list of 3 elements. This is because the append method just adds its parameter as a single element to the list.

If you wanted to add several elements, then you should have used the extend method instead. The correct code looks like this:

my_list = [1, 2, 3]
my_list.extend([4, 5, 6])
print(my_list)

[1, 2, 3, 4, 5, 6]

So, just be aware that append and extend exist and that they do different things. Understand the difference, and you will avoid a lot of frustration. :angry:

5. Misunderstanding how default arguments work#

This last issue is more subtle and can catch out even experienced programmers. It is related to the way Python handles default arguments.

First of all, we will write a simple function to remember what a default argument is. Let us say we want to write a function called ‘apply_scaling’ that simply applies a scaling factor to a number. Let us say that in our application, the most common use is to scale things by a factor of 2, so we can make this the default value for the scaling factor:

def apply_scaling(x, scale=2):
    return x * scale

x = np.array([10,20,30])

# Scale x by 10...
print(apply_scaling(x, 10))

# ... or we can just use the default scaling factor of 2
print(apply_scaling(x))

[100 200 300]
[20 40 60]

This works as expected and there are no issues with the above code.

The problems begin when we have a default value that is a mutable object, such as a list or a NumPy array. Note that a mutable object is a value that can be changed after it has been created.

Let us say that you want to write a function that can append a number to a list, if the user does not supply a list then it will default to adding the number to an empty list. You might write something like this,

def append_to_list2(x, y=[]):
    y.append(x)
    return y

This looks fine, but let us see what happens when we use the function:

print(append_to_list2(10, [1,2,3,4,5]))

print(append_to_list2(20))

[1, 2, 3, 4, 5, 10]
[20]

Both of the above provide the expected result. But now let us call the function a second time with the default value

print(append_to_list2(30))

[20, 30]

Now, rather than getting [30] as we would expect, we get [20, 30].

The first call changed the value of the default argument from [] to [20]. This happens because the default argument is only evaluated when the function is defined. i.e., it is set to [] when the function is defined but does not get reset every time the function is called. So, if a previous call has changed its value then it will stay changed.

If we need a default value to be a mutuable type, e.g., such as a list, then we need to use the following pattern,

def append_to_list3(x, y=None):
    if y is None:
        y = []
    y.append(x)
    return y

This will ensure that the parameter y is reset to the empty list every time the function is called without a value for y being supplied.

We can test this using the same sequence of calls as before:

print(append_to_list3(10, [1,2,3,4,5]))
print(append_to_list3(20))
print(append_to_list3(30))

[1, 2, 3, 4, 5, 10]
[20]
[30]

It now works as expected.

6. Submit your own#

If you have any suggestions for other common problems to include, please let me know.