Generators and Decorators

Helsinki PyLadies

2013-09-25

Generators

What is a generator?

A generator is a function that contains (at least) a yield:

def even_numbers(limit):
    for n in range(limit):
        if n%2 == 0:
            yield n

Unlike regular functions, generators return a generator object:

>>> even_numbers(10)
<generator object even_numbers at 0xb7243694>

Generator objects are iterators, this means that they can be used in a number of places, including for loops and functions that accept iterables.

What is an iterable? What is an iterator?

An iterable is an object capable of returning its members one at a time. Everything that you can use in a for loop is an iterable: lists, strings, files, dictionaries, etc.. Objects that are iterables have an __iter__ method that returns an iterator.

An iterator is an object with a __next__ method. Iterators are used to keep track of the position during the iteration -- calling next(it) will return the next element or raise StopIteration if there are no more elements.

How do I use a generator?

After you defined your generator function, you can call it and iterate on the returned generator object:

>>> for x in even_numbers(10):
...   print(x)
...
0
2
4
6
8

Note that you can iterate on a generator object only once:

>>> e = even_numbers(10)
>>> list(e)
[0, 2, 4, 6, 8]
>>> list(e)
[]

How does a generator work?

Generator functions have (at least) a yield. Calling the generator function doesn't execute the code inside the function, but gives you a generator object (so if you call even_numbers() you won't get any even number yet).

When you start iterating on the generator object the code is executed until the yield, then the execution is paused at that position and a value is returned. The execution is resumed as soon as the next element is requested and paused again when another yield is found.

Basic example

>>> def simple_gen():
...     print('start')
...     print('generate a value and pause')
...     yield 0
...     print('resume execution, generate a value, and pause again')
...     yield 1
...     print('resume execution, generate a value, and pause again')
...     yield 2
...     print('finish')
...
>>> g = simple_gen()
>>> g
<generator object simple_gen at 0xb713b7fc>
>>> next(g)
start
generate a value and pause
0
>>> next(g)
resume execution, generate a value, and pause again
1
>>> next(g)
resume execution, generate a value, and pause again
2
>>> next(g)
finish
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Basic example

Note that:

to create a generator function is enough to have (at least) a yield
calling the generator function gives you a generator object
no code is executed when you create the generator object
iterating (or calling next()) on the generator object executes all the code until the first yield
once the yield is reached, the execution is paused and a value is returned
when an other value is requested the execution is resumed and the code executed until the next yield
once the end of the function is reached, a StopIteration is raised to signal that there are no more values, and the iteration is interrupted

Generator expressions

A generator expression is an expression that gives you a generator object. Its syntax is similar to a list comprehension, except that parentheses are used instead of square brackets.

>>> g = (x for x in range(10) if x%2 == 0)
>>> g
<generator object <genexpr> at 0xb6a86c5c>
>>> list(g)
[0, 2, 4, 6, 8]

If used inside a function call, the parentheses can be omitted:

>>> sum(x for x in range(10) if x%2 == 0)
20

Why should I use generators?

Generators are lazy -- no code is executed until a value is requested
Generators are efficient -- they only keep a single value in memory instead of building huge sequences of values in memory
Generators can be infinite -- a generator can generate an infinite amount of values
Generators are flexible -- they can be combined together easily

Decorators

A decorator is a function that accepts a function and returns a function:

def deco(func):
    return func

Decorators are used on other functions to add behavior or replace them with a different function:

def foo():
    print("hello world")
foo = deco(foo)

Python supports a specific syntax that makes applying decorators easier and more readable:

@deco
def foo():
    print("hello world")

Both are equivalent, the @ form is just syntactic sugar.

Basic example

Here's an example of defining and using a simple decorator:

>>> def repl_func():
...     print("I'm the replacement function")
...
>>> def deco(func):
...     return repl_func
...
>>> @deco
... def orig_func():
...     print("I'm the original function")
...
>>> orig_func()
I'm the replacement function
>>> orig_func
<function repl_func at 0xb71f7344>

(continues...)

Basic example

Note that:

the decorator is just a regular function that receives the decorated function
the decorator is applied on orig_func by using the @deco syntax
the decorator is called just once -- at function definition time (when orig_func is defined
the decorator returns the replacement function repl_func
repl_func replaces orig_func
the original orig_func is no longer accessible

(continues...)

Basic example

This is equivalent to:

>>> def repl_func():
...     print("I'm the replacement function")
...
>>> def deco(func):
...     return repl_func
...
>>> def orig_func():
...     print("I'm the original function")
...
>>> orig_func = deco(orig_func)
>>> orig_func()
I'm the replacement function
>>> orig_func
<function repl_func at 0xb71f7344>

What can I use decorators for?

Decorators can be used for many different purposes:

memoization
logging
input/output validation
benchmarking/profiling
setup/teardown
...

Let's see a few examples.

Example 1

The simplest decorators return the same function that they receive.

For example, if you don't want to add functions names to __all__ manually, you can use a decorator that will do it for you:

__all__ = []

def add_to_all(func):
    """Add the decorated function to __all__."""
    __all__.append(func.__name__)
    return func

@add_to_all
def foo():
    print("I'm foo and I will get imported")

@add_to_all
def bar():
    print("I'm bar and I will get imported")

def baz():
    print("I'm baz and I will not get imported")

Note how the decorator returns the same function (continues...)

Example 1

Now only the decorated functions will be imported:

>>> from deco_ex1 import *
>>> foo()
I'm foo and I will get imported
>>> bar()
I'm bar and I will get imported
>>> baz()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'baz' is not defined

Example 2

For example, this other decorator shows you when a function is being defined:

import time

def log_definition(func):
    print(time.strftime("[%Y-%m-%d %H:%M:%S]"),
          func.__name__, "has been defined")
    return func

@log_definition
def func1():
    pass

@log_definition
def func2():
    pass

time.sleep(3)

@log_definition
def func3():
    pass

Note that I added a sleep between the last two functions (continues...)

Example 2

If we run the script we will see when each function has been imported:

$ python3 deco_ex2.py
[2013-09-22 16:54:17] func1 has been defined
[2013-09-22 16:54:17] func2 has been defined
[2013-09-22 16:54:20] func3 has been defined

Example 3

Most of the times decorators are used to replace the decorated function with another function. This is usually done by defining an inner function inside the decorator.

def call_twice(func):
    def inner():
        func()
        func()
    return inner

@call_twice
def hello():
    print("hello world")

>>> from deco_ex3 import hello
>>> hello()
hello world
hello world
>>>

Example 4

The following decorator calculates and prints the time necessary to execute the function.

import time
import random

def calc_time(func):
    def inner(*args, **kwargs):
        start = time.time()
        res = func(*args, **kwargs)
        end = time.time()
        print('The function took', end-start, 'seconds.')
        return res
    return inner

@calc_time
def stupid_sort(input_seq):
    seq = list(input_seq)
    while seq != sorted(seq):
        random.shuffle(seq)
    return seq

@calc_time
def smart_sort(seq):
    return sorted(seq)

(continues...)

Example 4

>>> from deco_ex4 import *
>>> res = stupid_sort(range(10, 0, -1))
The function took 34.63149285316467 seconds.
>>> res = smart_sort(range(10, 0, -1))
The function took 1.9788742065429688e-05 seconds.

Example 5

The following decorator saves the input and output of the function, and if the function is called again with the same input it returns the result without calling it again.

def memoize(func):
    cache = {}
    def inner(*args):
        if args in cache:
            print('returning memoized result')
            return cache[args]
        res = func(*args)
        cache[args] = res
        return res
    return inner

@memoize
def add(a, b):
    print('adding', a, 'and', b)
    return a + b

(continues...)

Example 5

>>> from deco_ex5 import add
>>> add(2, 3)
adding 2 and 3
5
>>> add(2, 3)
returning memoized result
5
>>> add(2, 5)
adding 2 and 5
7
>>> add(2, 5)
returning memoized result
7
>>> add(2, 3)
returning memoized result
5

Decorator factories

A decorator factory is a function that returns a decorator.

Decorator factories are used when you want to pass additional argument to the decorator, and they are commonly implemented by nesting 3 functions.

The outer function is the decorator factory, the middle one is the decorator, and the inner one is the function that will replace the decorated function.

Decorator factories -- example 6

This decorator factory returns a decorator that repeats a function N times.

def repeat_for(times):
    def deco(func):
        def inner():
            for x in range(times):
                func()
        return inner
    return deco

@repeat_for(3)
def hello():
    print("hello")

@repeat_for(2)
def bye():
    print("bye")

# same as bye = repeat_for(2)(bye)

Decorator factories -- example 6

>>> from deco_ex6 import *
>>> hello()
hello
hello
hello
>>> bye()
bye
bye
>>>