Decorators what are they and how to use them#
Python decorators are an extremely useful and powerful python construct that makes it very easy to extend functions and methods and avoid a lot of code-duplications
Learning outcomes#
In this notebook we are going to learn about:
The basic concept of a decorators, what is a decorator?
Functions as first-class objects
Passing functions as arguments to functions
Returning functions from functions
Defining functions within a function
How to write your own decorators
Write a decorator that logs function calls and arguments
Introduction to the logging module
Requirements#
Python 3
numpy
logging (part of the python standard library)
functools (part of the python standard library)
import numpy as np
Decorators explained#
So what are decorators?#
“A decorator in Python is any callable Python object that is used to modify a function or a class” from [1]
Wait… What?#
Probably the most common decorator that you will encounter is the inbuild property decorator.
Most of you would have seen something like the following:
class rectangle(object):
def __init__(self, x, y):
self.x = x
self.y = y
@property
def area(self):
return self.x*self.y
a = rectangle(2., 3.)
a.y = 5
print(a.area)
10.0
An aside on functions#
To understand how decorators work we have take first take a detour and look at functions in Python.
In python functions are first-class objects, what this means is that functions can be assigned to other variables e.g. like so:
def func(x):
print("Hello {}".format(x))
func("Nick")
gunc = func
gunc("Binbin")
Hello Nick
Hello Binbin
They can be passed as parameters to other functions
def otherfunc(f):
f("Jochen")
otherfunc(func)
Hello Jochen
They can also be returned from other functions:
def anotherfunc(f):
print("Passed function {}".format(f.__name__))
return f
g = anotherfunc(func)
g("Nick")
Passed function func
Hello Nick
In the above example we also demonstrated that the name of the function is held in the __name__ attribute
Importantly unlike e.g. in C we can define functions inside another function:
def outsidefunc(greet):
def greeter(name):
print("{} {}".format(greet, name))
return greeter
gg = outsidefunc("Hej")
gg("Binbin")
gg("Nick")
Hej Binbin
Hej Nick
How does this relate to decorators#
Decorators are essentially functions (or classes) that modify other functions. Let’s look at a basic example.
def add(x,y):
return x+y
def decexample(func):
def wrapper(x, y):
print("Calling function {}".format(func.__name__))
return func(x,y)
return wrapper
newadd = decexample(add)
newadd(2,4.)
Calling function add
6.0
The pythonic syntax for decorator is using the @ sign.
@decexample
def add(x,y):
return x+y
add(2,5.)
Calling function add
7.0
To use decorators more generally it is better to add the ability for arbitary number of arguments and keyword arguments
def decexample(func):
def wrapper(*args, **kwargs):
print("Calling function {}".format(func.__name__))
print("Number of arguments {}".format(len(args)))
print("Number of keyword arguments: {}".format(len(kwargs)))
return func(*args, **kwargs)
return wrapper
@decexample
def polynomial(x, a, b, c=0):
return a*x**2 + b*x + c
x=np.linspace(0, 10, 1000)
y = polynomial(x, 1, 2, c=1)
Calling function polynomial
Number of arguments 3
Number of keyword arguments: 1
Logging decorator#
When running experiments it is often useful to have a log of the different instrument calls, processing functions and keeping track of errors etc.. Instead of putting different logging calls into each function we can use a decorator, which gives a much cleaner interface and much less code duplication.
You should also note that while in the examples here we add decorators to functions one can also add them to methods, e.g. for a instrument class.
In the example we will also itroduce the logging module and show how to implement decorators with arguments.
import logging
from functools import wraps
logging.basicConfig(level=logging.DEBUG)
In the above code we import the logging and functools modules (more on the second later) and initialize the logger to log all events with a severity above DEBUG level (default is WARNING). By default the logging modules logs to the standard output, if you want to log to a file use the filename keyword.
def log_event(func): # the decorator function
@wraps(func)
def logwrapper(*args, **kwargs): # the function that is returned by the decorator and performs the logging
frmt = "call of {} function with arguments: " + ', '.join(["{}"]*len(args)) + " and keyword arguments " + ", ".join(["{}"]*len(kwargs))
message = frmt.format(func.__name__, *args, *["{}:{}".format(k,v) for k,v in kwargs.items()])
logging.debug(message)
return func(*args, **kwargs)
return logwrapper
@log_event
def add(x, y):
return x+y
@log_event
def operation(x, y, op="sub"):
if op == "sup":
return x-y
elif op == "add":
return x+y
add(1,2)
operation(1,3, op="add")
DEBUG:root:call of add function with arguments: 1, 2 and keyword arguments
DEBUG:root:call of operation function with arguments: 1, 3 and keyword arguments op:add
4
The @wraps decorator#
Let’s have a closer look what the @wraps decorator does. For this we should first look at a decorator without this
def do_nothing(func):
def nothingwrapper(*args, **kwargs):
print("I'm not doing anything")
return func(*args, **kwargs)
return nothingwrapper
@do_nothing
def add(x, y):
return x+y
add(1,2) #works as expected
I'm not doing anything
3
# if we want to look at the function
print(add)
<function do_nothing.<locals>.nothingwrapper at 0x798c85715ab0>
Ok but our function should be called add?
#this is where the wraps decorator comes in
def do_nothing(func):
@wraps(func)
def nothingwrapper(*args, **kwargs):
print("I'm not doing anything")
return func(*args, **kwargs)
return nothingwrapper
@do_nothing
def add(x, y):
return x+y
print(add) # ok all good
<function add at 0x798c857157e0>
Back to logging#
Often we want to format the log messages differently in particular it is good to have a timestamp (note we have to restart the kernel because the logging module keeps its state).
import logging
from functools import wraps
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(message)s') # this specifies a logging format of "time message"
def log_event(func):
@wraps(func)
def logwrapper(*args, **kwargs):
frmt = "call of {} function with arguments: " + ', '.join(["{}"]*len(args)) + " and keyword arguments: " + ", ".join(["{}"]*len(kwargs))
message = frmt.format(func.__name__, *args, *["{}:{}".format(k,v) for k,v in kwargs.items()])
logging.debug(message)
return func(*args, **kwargs)
return logwrapper
@log_event
def add(x, y):
return x+y
@log_event
def operation(x, y, op="sub"):
if op == "sup":
return x-y
elif op == "add":
return x+y
add(1,2)
operation(1,3, op="add")
DEBUG:root:call of add function with arguments: 1, 2 and keyword arguments:
DEBUG:root:call of operation function with arguments: 1, 3 and keyword arguments: op:add
4
Decorator arguments#
Often it would be useful to pass arguments to the decorator, such as a custom message or a different log level. For this we need to encapsulate the decorator function in another function that takes arguments. In the following example the log_event decorator takes a custom format string and a log level. Note that the format string must be for the correct number of arguments.
def logger(log_level=logging.DEBUG, msg_frmt=None): # another outer function which takes a log_level and a msg_frmt
def decorator_log(func): # the same logger decorator function as before
@wraps(func)
def wrapper_logger(*args, **kwargs): # the actual wrapper
nonlocal msg_frmt
if msg_frmt is None:
msg_frmt_out = "call of {} function with arguments: " + ', '.join(["{}"]*len(args)) + " and keyword arguments: " + ", ".join(["{}"]*len(kwargs))
message = msg_frmt_out.format(func.__name__, *args, *["{}:{}".format(k,v) for k,v in kwargs.items()])
else:
message = msg_frmt.format(*args, *["{}:{}".format(k,v) for k,v in kwargs.items()])
logging.log(log_level, message)
return func(*args, **kwargs)
return wrapper_logger
return decorator_log
@logger(logging.INFO, "addition of {} and {}")
def add(x, y):
return x+y
@logger(logging.INFO)
def operation(x, y, op="sub"):
if op == "sup":
return x-y
elif op == "add":
return x+y
add(1,2)
operation(1,3, op="add")
INFO:root:addition of 1 and 2
INFO:root:call of operation function with arguments: 1, 3 and keyword arguments: op:add
4
But#
@logger
def add(x, y):
return x+y
add(1,2)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[23], line 5
1 @logger
2 def add(x, y):
3 return x+y
----> 5 add(1,2)
TypeError: logger.<locals>.decorator_log() takes 1 positional argument but 2 were given
The problem is that the logger function does not has not been called with an argument. To fix that we need to allow the logger function to recognise if it has or hasn’t been called with an argument.
def logger(_func = None, *, log_level=logging.DEBUG, msg_frmt=None ): # another outer function which takes the keywords (the * means catch all non-kwargs)
print("_func = {}".format(_func))
def decorator_log(func): # the same logger decoratr function as before
@wraps(func)
def wrapper_logger(*args, **kwargs): # the actual wrapper that does the logging
nonlocal msg_frmt
if msg_frmt is None:
msg_frmt_out = "call of {} function with arguments: " + ', '.join(["{}"]*len(args)) + " and keyword arguments: " + ", ".join(["{}"]*len(kwargs))
message = msg_frmt_out.format(func.__name__, *args, *["{}:{}".format(k,v) for k,v in kwargs.items()])
else:
message = msg_frmt.format(*args, *["{}:{}".format(k,v) for k,v in kwargs.items()])
logging.log(log_level, message)
return func(*args, **kwargs)
return wrapper_logger
# the magic is here
if _func is None:
return decorator_log
else:
return decorator_log(_func)
@logger(log_level=logging.INFO, msg_frmt="addition of {} and {}")
def add(x, y):
return x+y
@logger
def operation(x, y, op="sub"):
if op == "sup":
return x-y
elif op == "add":
return x+y
add(1,2)
operation(1,3, op="add")
Example: retry decorator#
Some instruments are buggy, i.e. they don’t respond all the time, or are very slow to respond, causing timeouts. If we know these instruments we probably want to retry the command if we receive a timeout.
Solution:
Put a for loop into each of our methods/functions for that instrument
lots of code duplication
use a decorator
Solution#
import numpy as np
def retry_on_fail(_func=None, *, max_retries=3, exception_type=Exception): # Exception type is configurable
def retry_decorator(func):
@wraps(func)
def retry_wrapper(*args, **kwargs):
nonlocal max_retries
for i in range(max_retries):
try:
return func(*args, **kwargs)
except exception_type:
print("An exception occurred retrying") # we could log the event here
continue
raise IOError("Retried {} times, but still get an {} Error".format(max_retries, exception_type))
return retry_wrapper
if _func is None:
return retry_decorator
else:
return retry_decorator(_func)
@retry_on_fail(max_retries=3)
def failing_function(): # this is very much a dummy function
i = np.random.choice([True, False])
print(i)
if i:
print("I finished")
else:
raise Exception("Error")
for i in range(10):
print(i)
failing_function()
Nesting#
One great feature is that it’s possible to nest decorators. So we can use our two together.
Important: Decorator order matters:
@retry_on_fail(max_retries=3, exception_type=Exception)
@logger(log_level=logging.DEBUG, msg_frmt=None)
def failing_function(): # this is very much a dummy function
i = np.random.choice([True, False])
print("choice: {}".format(i))
if i:
print("I finished")
else:
raise Exception("Error")
for i in range(10):
print(i)
failing_function()
@logger(log_level=logging.DEBUG, msg_frmt=None)
@retry_on_fail(max_retries=3, exception_type=Exception)
def failing_function(): # this is very much a dummy function
i = np.random.choice([True, False])
print("choice: {}".format(i))
if i:
print("I finished")
else:
raise Exception("Error")
for i in range(10):
print(i)
failing_function()