Xmas decoration, part 1

Binary trees are trees too

Summary

Every Python dev will go through the moment where they need to learn about those damn @decorator.

And truth is, they are mostly syntactic sugar for this:

>>> def decorator(the_callback):
...     def wrapper():
...         print('Behavior to add')
...         the_callback()
...     return wrapper
...
>>> def func(): ...
>>> func = decorator(func)

But my experience is that there are a few steps before getting to understand the code above.

@ a glance

You cannot use Python for long before eventually stumbling upon some weird @ shenanigans. E.G, if you use flask, the hello world contains:

@app.route("/") # what's with the @ ?
def hello_world():

Those are decorators, a design pattern that has been integrated into the Python language syntax. It's not something that's obvious at first glance, because if you open any book about decorators, they will show you the object-oriented version, but Python implement a functional one.

In this first article, we will talk about how to use them, then how to create your own, or at least a basic version of it.

@ your service

So how do you use @?

There are 3 ways.

First, there is the @ operator between 2 objects. E.G, if you use numpy:

>>> import numpy as np
...
... a = np.array([[1, 2], [3, 4]])
... b = np.array([[5, 6], [7, 8]])
...
... a @ b
array([[19, 22],
       [43, 50]])

This is called the "matmul" operator, for matrix multiplication, and has nothing to do with decorators. So for this article, you can just forget about it.

Then you can use @ above a class name. E.G, when declaring a data class:

>>> from dataclasses import dataclass
>>> @dataclass
... class XMasMovie:
...     title: str
...     director: str
...     xmas_factor: int

That's is called unsurprisingly a "class decorator", and it's an advanced use case we will see, but in a later article.

What we will focus on today, are "function decorators", like we have seen with the flask example.

To use them, you have to use @, then usually a name, then sometimes parenthesis. It must be above a function or a method.

Let's see a few examples from the Python standard library:

A decorator on a method, with parentheses:

import unittest
from unittest.mock import patch
import json

class MyTestCase(unittest.TestCase):
    @patch('json.load')
    def test_get_user_info(self):
        ...

For the duration of test_get_user_info, json.load is replaced by a mock.

A decorator on a method, no parenthesis:

>>> class Computer:
...     @classmethod
...     def the_answer_to_the_universe_and_everything(cls):
...         return 42
...

Turns the_answer_to_the_universe_and_everything into a class method.

A decorator on a function, with parenthesis:

from functools import lru_cache

@lru_cache()
def partially_cached_fib(n):
    if n < 2:
        return n
    return partially_cached_fib(n-1) + partially_cached_fib(n-2)

This caches the last 128 results of the function.

A decorator on a function, no parenthesis (python 3.9):

from functools import cache

@cache
def cached_fib(n):
    if n < 2:
        return n
    return cached_fib(n-1) + cached_fib(n-2)

This caches all the results of the function.

As you can see there is no way to know what a decorator is going to do just by looking at it. Just like with regular functions, you need to read the doc, see an example or ask ChatGPT. Sometimes the name helps.

@ odds

The syntax of decorators can be a bit tricky.

First, how do you know when to use parentheses or not?

Again, ask ChatGPT. Seriously. There is no way to know unless you read the doc, look at the code or find an example.

Second, it accepts any attribute access, so you can do:

import functools

@functools.cache
def cached_fib(n):
    if n < 2:
        return n
    return cached_fib(n-1) + cached_fib(n-2)

You can also stack them:

import functools

@functools.cache
@functools.singledispatch
def oh_oh_oh(arg):
    raise NotImplementedError("You have been naughty")

And since arguments and eval can be used, you can do awful things like:

_ = lambda x: x

@_(presents[0].unwrap().price_tag)
def milk():
    ...

@eval("presents[0].unwrap().price_tag")
def cookies():
    ...

There is even a PEP that hopes to make those sins more official.

Because of all those possibilities, a good copy/paste of a working example is going to make your life easier.

And read this series of articles to understand how they work so that you know what you just introduced to your code base.

@ the helm

Let's gain back control, and dive into what makes decorators tick.

There is a prerequisite for this, and it's to be comfortable with function references. This is why I wrote an article on that a few days ago.

In short:

• Functions are objects.

• You can put them in variables and collections.

• You can pass them as parameters.

• You can define them dynamically.

• You can return them from another function.

If this is not VERY clear to you, go read the article, because we are going to make heavy use of this.

In fact, we are going to mix all those concepts, by writing a function that:

• Takes another function as a callback.

• Creates a function dynamically.

• Calls the callback in the dynamically created function.

• Returns the dynamically created function.

That's a lot. Ready?

>>> def a_function_that_expects_a_callback(the_callback):
...
...     print('The function that expects a callback starts!')
...
...     def a_dynamically_defined_function():
...         print('The dynamically defined function starts!')
...         the_callback()
...         print('The dynamically defined function stops!')
...
...     print('The function that expects a callback stops!')
...
...     return a_dynamically_defined_function
...

Take your time, there is a much to unpack here.

a_function_that_expects_a_callback has one parameters, and this parameter is another function.

In its belly, it creates on the fly a_dynamically_defined_function, BUT it doesn't use the_callback.

Instead a_dynamically_defined_function is the one that uses the_callback.

BUT there is another twist!

a_dynamically_defined_function is never executed, it is returned.

What does this beast look like when we use it?

Let's create a function and call a_function_that_expects_a_callback with it as a callback:

>>> def a_normal_function():
...     print("I'm a super normal function. I'm nothing special")
...
... mystery = a_function_that_expects_a_callback(a_normal_function)
The function that expects a callback starts!
The function that expects a callback stops!

When we use it, several things happen:

• We pass to a_function_that_expects_a_callback the function a_normal_function as a parameter. We use the syntax a_normal_function, not a_normal_function(), because we pass the reference to the function object, not the result of the function call.

• When a_function_that_expects_a_callback runs, we get some prints. It tells us a_function_that_expects_a_callback has run correctly from start to end.

• But we don't see any print from a_dynamically_defined_function or a_normal_function. They didn't run at all!

The most important thing, what is the mysterious result?

>>> mystery
<function a_function_that_expects_a_callback.<locals>.a_dynamically_defined_function at 0x7ff61d6385e0>

The result is a a_dynamically_defined_function. The variable mystery contains a freshly baked function, ready to be used!

Let's use it:

>>> mystery()
The dynamically defined function starts!
I'm a super normal function. I'm nothing special
The dynamically defined function stops!

Now we see the prints from a_dynamically_defined_function and a_normal_function. In fact, the code of a_normal_function is wrapped, it is executed in between the beginning and the end of a_dynamically_defined_function.

We have created a new function that is made of a_normal_function plus some additional behavior.

So in summary:

• a_function_that_expects_a_callback takes a callback, and creates a new function out of it.

• the new function contains the code of the callback, plus some additional behavior.

@ a crossroad

Congratulations, you have created your first decorator.

I know, it's not obvious yet, so let's add a twist. We will assign the result of calling a_function_that_expects_a_callback so that it overrides the original function:

>>> def dumb_function():
...     print("I don't know anything about decorators I swear!")
...
>>> dumb_function()
I don't know anything about decorators I swear!
>>> dumb_function = a_function_that_expects_a_callback(dumb_function)
The function that expects a callback starts!
The function that expects a callback stops!

The original dumb_function is no more. It's been erased and replaced by the newly dynamically created function:

>>> dumb_function()
The dynamically defined function starts!
I don't know anything about decorators I swear!
The dynamically defined function stops!

This patterns allows us to take any function, and add any code we want to it, before or after.

We "decorate" the original code with some additional features.

It just happens that Python has some syntactic sugar for this:

@a_function_that_expects_a_callback # that was a decorator all along!
def dumb_function():
    print("I don't know anything about decorators I swear!")

Yep, that's right, the syntax above is the equivalent to:

def dumb_function():
    print("I don't know anything about decorators I swear!")
dumb_function = a_function_that_expects_a_callback(dumb_function)

It's just a shorthand.

a_function_that_expects_a_callback is a decorator.

@ a snail's pace

There is a lot to say about decorators, so we are going to take several articles to do so.

Today was about giving you a rundown of the general principle.

In the next few posts, we will see:

• How to deal with parameters.

• What the hell you are supposed to use decorators for, and typical use cases for them, with examples.

• Class decorators.

• Best practices with decorators, and libraries that can help you with them.