Xmas decoration, part 2

Curing the hungover

Summary

If you tried the code from our intro on decorator, you may have noticed big shortcomings. That's because it was a bird view, but there are details to iron out.

We need to deal with parameters, return value and metadata.

In short, the result looks like this:

from functools import wraps
def function_that_creates_a_decorator(upper_case=False):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            if upper_case:
                print('BEHAVIOR TO ADD BEFORE')
                result = func(*args, **kwargs)
                print('BEHAVIOR TO ADD AFTER')
            else:
                print('Behavior to add before')
                result = func(*args, **kwargs)
                print('Behavior to add after')
            return result
        return wrapper
    return decorator

But it will take a bit of explanation to understand what this code means.

I get it, but I don't get it

In the last article about Python decorators, we gave a general presentation of the concept and how it works. However, if you start trying to create your own, you will hit some very practical matters:

• How do you deal with decorated functions that expect parameters and return values?

• How do you pass parameters to your decorator?

• Why help() and repr() stop working on decorated function?

So today we are going to address those points.

Decorated function with parameters

So, remember, we said a decorator is something like this:

>>> def decorator(func):
...     def wrapper():
...         print('Behavior to add before')
...         func()
...         print('Behavior to add after')
...     return wrapper

And then you use it like this:

... @decorator
... def decorated_function():
...     print('ta da!')
...
>>> decorated_function()
Behavior to add
ta da!
Behavior to add before

That's all fine and dandy, but most functions don't just print. In fact, most functions shouldn't, since good design is to have a sizeable chunk of your code free of side effects.

So realistically, you will have something like this:

... @decorator
... def normalize_user_input(user_input):
...     return str(user_input).strip().casefold()

But if you try to call it, it will fail:

>>> normalize_user_input("   HeLlO i'M uSer   ")
TypeError: decorator.<locals>.wrapper() takes 0 positional arguments but 1 was given

Not only you don't get what you want, but the error message is as clear as the moral of the story of The binding of Isaac. The game, not the Bible story. Actually, either works.

That's because the decorator we coded creates a wrapper function (you know, the dynamically created one) that declares no parameter. Let's fix this:

>>> def decorator(func):
...     def wrapper(user_input): # expect a parameter
...         print('Behavior to add before')
...         func(user_input) # pass it as argument
...         print('Behavior to add after')
...     return wrapper
... @decorator
... def normalize_user_input(user_input):
...     return str(user_input).strip().casefold()

This works:

>>> normalize_user_input("   HeLlO i'M uSer   ")
Behavior to add before
Behavior to add after

But you'll notice we don't get the result value. That's because again, our wrapper doesn't deal with this. Let's fix it:

>>> def decorator(func):
...     def wrapper(user_input):
...         print('Behavior to add before')
...         result = func(user_input) # save the value
...         print('Behavior to add after')
...         return result
...     return wrapper
... @decorator
... def normalize_user_input(user_input):
...     return str(user_input).strip().casefold()

OK, now it works:

>>> result = normalize_user_input("   HeLlO i'M uSer   ")
Behavior to add before
Behavior to add after
>>> print(result)
hello i'm user

But what happens if I add another parameter?

... @decorator
... def normalize_user_input(user_input, default=None):
...     return str(user_input).strip().casefold() or default

It fails again:

>>> result = normalize_user_input("   HeLlO i'M uSer   ", "No, I'm doesn't")
TypeError: decorator.<locals>.wrapper() takes 1 positional argument but 2 were given

That's because our decorator only expects a single parameter. We need something dynamic, and this is where * and ** comes in handy:

>>> def decorator(func):
...     def wrapper(*positional_params, **keyword_params):
...         print('Behavior to add before')
...         result = func(*positional_params, **keyword_params)
...         print('Behavior to add after')
...         return result
...     return wrapper
... @decorator
... def normalize_user_input(user_input, default=None):
...     return str(user_input).strip().casefold() or default

And now it will work nicely, no matter the number of params we pass:

>>> result = normalize_user_input("   HeLlO i'M uSer   ")
Behavior to add before
Behavior to add after
>>> result = normalize_user_input("   HeLlO i'M uSer   ", "No, I'm doesn't")
Behavior to add before
Behavior to add after

In fact, this is one of those rare occasions where I will encourage you to name those parameters args and kwargs:

>>> def decorator(func):
...     def wrapper(*args, **kwargs):
...         print('Behavior to add before')
...         result = func(*args, **kwargs)
...         print('Behavior to add after')
...         return result
...     return wrapper

Since here it conveys "we don't know what is going to be passed, so we proxy everything".

And if * and ** are not familiar to you, I will probably write an article on that in the future.

Decorators with parameters

OK, now we know how the decorated function can get parameters. But what about the decorator itself? What about if we want to configure it?

E.G, what if we want to be able to do this?

... @decorator(upper_case=True)
... def normalize_user_input(user_input, default=None):
...     return str(user_input).strip().casefold() or default
>>> result = normalize_user_input("   HeLlO i'M uSer   ")
BEHAVIOR TO ADD BEFORE
BEHAVIOR TO ADD AFTER

Well, this is tricky.

The decorator itself must accept a function, the callback, as a parameter. So we can't really pass it anything.

What we can do is create the decorator on the fly, and make the function that creates it accept parameters.

Get ready for some inception!

The decorator is creating a function, the wrapper, and returns it, right?

Now, we are going to write a function that creates a decorator, which still itself creates a function that calls a callback.

Look, you started to read this article series, you are in this with me now!

Breathe, it will be alright.

def function_that_creates_a_decorator(upper_case=False):
    def decorator(func):
        def wrapper(*args, **kwargs):
            # Ugly code to keep it simple
            # Stay focus on the decorator
            if upper_case:
                print('BEHAVIOR TO ADD BEFORE')
                result = func(*args, **kwargs)
                print('BEHAVIOR TO ADD AFTER')
            else:
                print('Behavior to add before')
                result = func(*args, **kwargs)
                print('Behavior to add after')
            return result
        return wrapper
    return decorator # we return the decorator!

So you can get a feel on how it works, let's use it manually:

>>> decorator = function_that_creates_a_decorator(upper_case=True)
...
... @decorator
... def normalize_user_input(user_input, default=None):
...     return str(user_input).strip().casefold() or default
...
... result = normalize_user_input("   HeLlO i'M uSer   ")
BEHAVIOR TO ADD BEFORE
BEHAVIOR TO ADD AFTER

But that's not how you would use it in reality. This is how you would do it, IRL:

... @function_that_creates_a_decorator(upper_case=True)
... def normalize_user_input(user_input, default=None):
...     return str(user_input).strip().casefold() or default
...
... result = normalize_user_input("   HeLlO i'M uSer   ")
BEHAVIOR TO ADD BEFORE
BEHAVIOR TO ADD AFTER

Preserving introspection

The problem with decorators is that you effectively replace the original function with a newly generated one. The new one doesn't have a docstring, it doesn't have the argument signatures, it doesn't have a name.

If I define a nicely documented function like this:

def normalize_user_input(user_input: str, default: str | None=None) -> str | None:
    """Reduce PBCK as much as possible"""
    return str(user_input).strip().casefold() or default

I can then introspect it in the terminal:

>>> normalize_user_input
<function normalize_user_input at 0x7fd35ed3e560>
>>> help(normalize_user_input)
Help on function normalize_user_input in module __main__:

normalize_user_input(user_input: str, default: str | None = None) -> str | None
    Reduce PBCK as much as possible

But look at what happens when I decorate it:

@function_that_creates_a_decorator(upper_case=True)
def normalize_user_input(user_input: str, default: str | None=None) -> str | None:
    """Reduce PBCK as much as possible"""
    return str(user_input).strip().casefold() or default

All this information is lost:

>>> normalize_user_input
<function function_that_creates_a_decorator.<locals>.decorator.<locals>.wrapper at 0x7fd35ed3e3b0>
>>> help(normalize_user_input)
wrapper(*args, **kwargs)

Fortunately, python functools module contains wraps, that allows you to copy all metadata from one function to another. And what's funny is that wraps is itself a decorator :)

If we use it:

from functools import wraps
def function_that_creates_a_decorator(upper_case=False):
    def decorator(func):
        @wraps(func) # wrapper() gets func() metadata
        def wrapper(*args, **kwargs):
            if upper_case:
                print('BEHAVIOR TO ADD BEFORE')
                result = func(*args, **kwargs)
                print('BEHAVIOR TO ADD AFTER')
            else:
                print('Behavior to add before')
                result = func(*args, **kwargs)
                print('Behavior to add after')
            return result
        return wrapper
    return decorator

Then, we have all the information back!

>>> @function_that_creates_a_decorator(upper_case=True)
... def normalize_user_input(user_input: str, default: str | None=None) -> str | None:
...     """Reduce PBCK as much as possible"""
...     return str(user_input).strip().casefold() or default
...
>>> normalize_user_input
<function normalize_user_input at 0x7fd35db38af0>
>>> help(normalize_user_input)

normalize_user_input(user_input: str, default: str | None = None) -> str | None
    Reduce PBCK as much as possible

It's all good and well, but what am I supposed to do with this?

You now have enough knowledge to be dangerous with Python. You can understand and code most decorators.

But I see the doubt in your eyes.

What the hell are you going to use that for?

Next article, we are going to analyze the code of a few existing decorators from open source libraries so you can see what makes them tick, and what we use decorator for in real life.