Testing with Python (part 7): ...until you make it

Simon & Garfunkel wrote the playbook. Well, at least the Bookends.

Summary

I'm lazy, and if I can make things happen for me with less effort, I will. Generating test data is one of those things. We can:

Use seeding to ensure data consistency.

>>> random.seed(769876987698698)
>>> [random.randint(0, 10) for _ in range(10)]
[10, 9, 1, 9, 10, 6, 5, 10, 1, 9]
>>> random.seed(769876987698698)
>>> [random.randint(0, 10) for _ in range(10)]
[10, 9, 1, 9, 10, 6, 5, 10, 1, 9]

Generate fake data with mimesis.

>>> from mimesis import Generic
>>> g = Generic()
>>> for _ in range(5):
...     print(
...         f"[{g.datetime.date()}] {g.person.full_name()}, "
...         f"PIN attempt {g.code.pin()} at "
...         f"{g.finance.bank()} ({g.address.city()}) "
...         f"from {g.internet.ip_v4()}"
...     )
[2005-08-20] Rosario Howe, PIN attempt 1155 at Bank of Marin (Solon) from 171.243.230.180
[2001-11-14] Julieann Spencer, PIN attempt 4044 at State Street Corporation (Round Rock) from 155.236.115.163
[2003-07-24] Mikel Buchanan, PIN attempt 1057 at Union Bank (Scarsdale) from 211.192.167.83
[2010-10-30] Antione Duffy, PIN attempt 6219 at Sandy Spring Bancorp (Phoenix) from 207.104.0.29
[2004-06-05] Damon Stephens, PIN attempt 6115 at First National Community Bank (Newport) from 226.230.244.209

Freeze time with freeze gun:

>>> import datetime
... from freezegun import freeze_time
...
... def calendar_with_cheesy_quote():
...     print(f"{datetime.date.today():%B %d, %Y}")
...     print("Each day provide it's own gift")
...
... with freeze_time("2012-01-14"):
...     calendar_with_cheesy_quote()

Simulate a file system in memory with pyfakefs.

from pyfakefs.fake_filesystem_unittest import Patcher
from pathlib import Path
import os

with Patcher() as patcher:
    fs = patcher.fs
    temp_path = Path(os.path.join(os.sep, 'tmp', 'tempdir'))
    fs.create_dir(temp_path)

    (temp_path / 'subdir1').mkdir()
    (temp_path / 'subdir2').mkdir()

    (temp_path / 'file1.txt').write_text('This is file 1')
    (temp_path / 'file2.txt').write_text('This is file 2')

    print(f'Now you see me: {temp_path}')
    print(f'Contents: {list(temp_path.iterdir())}')

    fs.remove_object(temp_path)

print(f"Now you don't: {temp_path.exists()}"")

Record HTTP calls using VCR.py:

import pytest
import requests

@pytest.mark.vcr
def test_single():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'

And snapshot test results to posterity with inline-snapshot.

from inline_snapshot import snapshot

def test_something():
    assert 1548 * 18489 == snapshot() # snapshot() is the placeholder

Now go test things, you are free!

Less is more. And by that I mean less work and more stuff.

In the previous article, we've seen how to fake behavior with mocks, but what about faking data?

I mean, yes, you can always make it up yourself, and hard code it, but this brings with it the responsibility to:

• Clean the data up if creating it has a side effect. Like with files.

• Find good examples, diverse ones, in sufficient quantity, and produce them. Are you good at inventing user names? Cause we need a million of them.

• Create the API to inject said examples if they come from the outside world. Mocking your system clock is not fun.

• Update them when needed. Again.

It sure is possible. I've done it many times.

And it sucks.

Let's automate that stuff.

A word on seeding

Many of the tools dedicated to generate data use the concept of seeding to produce the illusion of randomness.

Indeed, getting a random value out of a hat is a complicated problem for a computer:

• If you need true unpredictability, you can't do it in software alone, because it's supposed to be predictable by design. You need an external source of randomness.

• If you need the shape of unpredictability, but in a way you can control (e.g.: so you can choose the distribution of the result), then it's not really random anymore, isn't it?

For the first problem, humans have solved that in ingenious ways, like putting lots of lava lamps on a wall and taking pictures of it:

The lava lamp wall at Cloudflare, pic from Wikipedia

But mostly, today, we have dedicated chips that use things like lasers, atomic decay, magnetism, and so on.

For the second problem, we use a seed, which is an initial number, on which you apply some mathematical transformation to get a result that looks random. This also gets you a different starting point. The next calculation, you use this new starting point to produce a new value, and generate a different starting point, and so on.

In Python, that's the difference between the module random (seeded) and the module secrets (use the OS source of randomness).

random is great for games and tests, because you get something randomish that is fast to produce, and at any time, you can set in stone to get reproducible results.

secret is great for generating passwords, keys, hashes, UUIDs, or certificates because it provides the strongest guarantees that another party will have a hard time guessing what's next.

Seeding makes for some fun Isaac replays and totally wacko speed run techniques:

It's something you can do yourself. If I generate 10 numbers from random twice, they do look random:

>>> import random
>>> [random.randint(0, 10) for _ in range(10)]
[4, 1, 2, 4, 2, 1, 4, 1, 10, 10]
>>> [random.randint(0, 10) for _ in range(10)]
[0, 2, 3, 0, 9, 6, 2, 2, 3, 7]

But if I reuse the same seed for those two sequences:

>>> random.seed(769876987698698)
>>> [random.randint(0, 10) for _ in range(10)]
[10, 9, 1, 9, 10, 6, 5, 10, 1, 9]
>>> random.seed(769876987698698)
>>> [random.randint(0, 10) for _ in range(10)]
[10, 9, 1, 9, 10, 6, 5, 10, 1, 9]

I get the same numbers.

So if you need to generate fake data, keep in mind some systems (like mimesis below does) provide a way to set the seed so that you can get reproducible tests.

General fake data

From generating an HTML filler to populating a database, faking data is an asset that serves you in proof of concepts, documentation, and of course, tests.

memesis is a superb Python library you can pip install, that will let you generate fake emails, addresses, social security numbers, colors, food items, and lorem ipsums, by the thousands.

>>> from mimesis import Generic
>>> g = Generic()
>>> for _ in range(5):
...     print(
...         f"[{g.datetime.date()}] {g.person.full_name()}, "
...         f"PIN attempt {g.code.pin()} at "
...         f"{g.finance.bank()} ({g.address.city()}) "
...         f"from {g.internet.ip_v4()}"
...     )
[2005-08-20] Rosario Howe, PIN attempt 1155 at Bank of Marin (Solon) from 171.243.230.180
[2001-11-14] Julieann Spencer, PIN attempt 4044 at State Street Corporation (Round Rock) from 155.236.115.163
[2003-07-24] Mikel Buchanan, PIN attempt 1057 at Union Bank (Scarsdale) from 211.192.167.83
[2010-10-30] Antione Duffy, PIN attempt 6219 at Sandy Spring Bancorp (Phoenix) from 207.104.0.29
[2004-06-05] Damon Stephens, PIN attempt 6115 at First National Community Bank (Newport) from 226.230.244.209

You can even set the locale. E.G, if I want some French data:

>>> from mimesis.locales import Locale
... g = Generic(locale=Locale.FR)
... for _ in range(5):
...     print(
...         f"[{g.datetime.date()}] {g.person.full_name()}, "
...         f"essai de PIN {g.code.pin()} à "
...         f"{g.finance.bank()} ({g.address.city()}) "
...         f"depuis {g.internet.ip_v4()}"
...     )
...
[2004-02-14] Joana Menard, tentative de PIN 4616 à Neuflize OBC (Colmar) depuis 193.125.166.248
[2012-07-08] Juliet Dastous, tentative de PIN 1168 à Banque de Savoie (Agen) depuis 6.232.176.31
[2018-01-23] Chloé Maurin, tentative de PIN 4101 à LCL (Menton) depuis 142.144.98.158
[2021-03-18] Nabil Jutras, tentative de PIN 6478 à Crédit Mutuel Alliance Fédérale (Bobigny) depuis 58.188.207.52
[2013-03-29] Noam Grand, tentative de PIN 4820 à AXA Banque (Poissy) depuis 186.54.107.131

It even works with Japanese, the data sets are impressive, and it has replaced my beloved faker in my toolkit.

I now have this in my PYTHONSARTUP:

try:
    import mimesis
except ImportError:
    pass
else:
    from mimesis.locales import Locale
    from mimesis import Generic

    class FakeProvider:
        def __init__(self, provider):
            self.provider = provider
            self.count = 1

        def __dir__(self):
            return dir(self.provider)

        def __repr__(self):
            return f"FakeProvider({repr(self.provider)})"

        def __getattr__(self, name):
            subprovider = getattr(self.provider, name)
            wrapper = lambda *args, count=1, **kwargs: self.call_faker(subprovider, *args, **kwargs)
            wraps(subprovider)(wrapper)
            return wrapper

        def __call__(self, count=1):
            self.count = count
            return self

        def call_faker(self, subprovider, *args, **kwargs):
            if self.count == 1:
                return subprovider(*args, **kwargs)
            else:
                return [subprovider(*args, **kwargs) for _ in range(self.count)]
            self.count = 1

    class Fake(object):

        def __init__(self, locale=Locale.EN):
            self.configure(locale)

        def get_factory(self, locale=Locale.EN, _cache={}):
            if isinstance(locale, str):
                locale = getattr(Locale, locale.upper())
            if locale in _cache:
                return _cache[locale]
            return Generic(locale=locale)

        def configure(self, locale=Locale.EN):
            self.factory = self.get_factory(locale)

        @property
        def fr(self):
            return self.get_factory(locale="fr")

        @property
        def en(self):
            return self.get_factory(locale="en")

        def __getattr__(self, name):
            return FakeProvider(getattr(self.factory, name))

        def __dir__(self):
            return ["fr", "en", *dir(self.factory)]


    fake = Fake()

So that at any time in a Python session, I can do:

>>> fake.food(10).fruit()
['Kahikatea', 'Canistel', 'Hackberry', 'Bilberry', 'Genip', 'Cocona', 'Limeberry', 'Blueberry', 'Bilimbi', 'Redcurrant']

to always have some data at hand to try things out.

About time

If you have to provide dates in code, there are two ways to make it test-friendly. Number one, make it so that you can pass the date provider as a parameter. E.G:

def create_article(text, time_provider=datetime.datetime.now):
    Article(text, created_at=date_provider())

This works, but requires more effort, plus sometimes you need to pass around those providers many layers deep, or pass multiple ones. It can become a chore.

Number two, use a terrible hack that patches all the sources of time for the duration of the test. It's not super clean, but it's easy, works surprisingly well, and keeps the whole code simpler.

Guess which one I prefer?

Meet freezegun, a third party lib that will let you suspend time for a block of code:

>>> import datetime
... from freezegun import freeze_time
...
... def calendar_with_cheesy_quote():
...     print(f"{datetime.date.today():%B %d, %Y}")
...     print("Each day provide it's own gift")
...
... with freeze_time("2012-01-14"):
...     calendar_with_cheesy_quote()
...
January 14, 2012
Each day provide it's own gift

It integrates with pytest, so you can decorate a test and pretend you are a Timelord:

@freeze_time("Jan 14th, 2020") # you can use a nice format here too
def test_sonic_screwdriver_in_overrated_show_there_I_said_it():
    ...

And it comes with some nice tricks, like starting time from a certain point, but then keep flowing:

@freeze_time("2012-01-14", tick=True)
def test_turn_it_on_turn_it_on_turn_it_on_again():
    # the clock starts again here, and the first value is 2012-01-14 
    # at 00:00:00 then whatever number of seconds goes by after that

Add 15 seconds every time you request time:

@freeze_time("2012-01-14", auto_tick_seconds=15)
def test_wheeping_angels_when_you_blink():
    # the clock starts again here, and the first value is 2012-01-14 
    # at 00:00:00 then every call to datetimes add 15 seconds

Or manually:

with freeze_time("2012-01-14") as right_here_right_now:
    right_here_right_now.tick(3600) # add 1H next time we request time
    right_here_right_now.move_to(other_datetime) # jump in time

Fun fact: hiding a freezegun call in a .pth file is a good way to make your colleagues have a breakdown on April’s Fool.

I/O

You will have to read and write to stuff, like files and sockets. Those are side effects that are very unreliable, so mocking is probably your first bet. But there are specialized tools that can make your life easier.

Known to many, the stdlib comes with the tempfile module, that can produce temporary files and directories that automatically clean after themselves:

>>> import tempfile
... from pathlib import Path
...
... with tempfile.TemporaryDirectory() as temp_dir:
...     temp_path = Path(temp_dir)
...
...     (temp_path / 'subdir1').mkdir()
...     (temp_path / 'subdir2').mkdir()
...
...     (temp_path / 'file1.txt').write_text('This is file 1')
...     (temp_path / 'file2.txt').write_text('This is file 2')
...
...     print(f'Now you see me: {temp_dir}')
...     print(f'Contents: {list(temp_path.iterdir())}')
...
... print(f"Now you don't: {Path(temp_dir).exists()}")
Now you see me: /tmp/tmp67e3hjr2
Contents: [PosixPath('/tmp/tmp67e3hjr2/file1.txt'), PosixPath('/tmp/tmp67e3hjr2/subdir2'), PosixPath('/tmp/tmp67e3hjr2/subdir1'), PosixPath('/tmp/tmp67e3hjr2/file2.txt')]
Now you don't: False

It's useful even outside of testing, but if for some reason you don't want to touch the hard drive, there is a package for that™:

from pyfakefs.fake_filesystem_unittest import Patcher
from pathlib import Path
import os

with Patcher() as patcher:
    fs = patcher.fs
    temp_path = Path(os.path.join(os.sep, 'tmp', 'tempdir'))
    fs.create_dir(temp_path)

    (temp_path / 'subdir1').mkdir()
    (temp_path / 'subdir2').mkdir()

    (temp_path / 'file1.txt').write_text('This is file 1')
    (temp_path / 'file2.txt').write_text('This is file 2')

    print(f'Now you see me: {temp_path}')
    print(f'Contents: {list(temp_path.iterdir())}')

    fs.remove_object(temp_path)

print(f"Now you don't: {temp_path.exists()}"")

Like freezegun, it’s a big hack, but a useful one. It will patch all filesystem calls, and create a make-believe FS in memory, so your file operations never leave the RAM.

Note that testing with real files will allow you to take care of edge cases like having symlinks and trees spanning on several partitions, which can be surprising at times. The right tool for the right job and all that.

Since we are in the realm of in-memory, you can do that with sqlite:

import sqlite3
connection = sqlite3.connect(':memory:')
cursor = connection.cursor()

cursor.execute('CREATE TABLE test (id INTEGER PRIMARY KEY, name TEXT)')
cursor.execute('INSERT INTO test (name) VALUES ("Alice")')
cursor.execute('INSERT INTO test (name) VALUES ("Bob")')
connection.commit()

cursor.execute('SELECT * FROM test')
rows = cursor.fetchall()
print('SQLite in-memory database contents:')
for row in rows:
    print(row)

This will create a db without touching the disk, and it will be temporary, and super fast.

Finally, what about the network?

Well, there are as many tools as there are applications for the network, which means, a lot. You'll find fake redis, placeholder HTTP API, man-in-the-middle interception, and even full-blown network control.

One that may pick your interest though, is VCR.py, a Python lib that records your HTTP requests, and can replay their response back to you.

You write a test performing a request, marking it with @pytest.mark.vcr:

import pytest
import requests

@pytest.mark.vcr
def test_single():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'

Then you run it once, against the real network:

pytest --record-mode=once test_network.py

Next time you run your test normally (without the --record-mode), VCR will replay the network response for you, and you won't touch the network.

That's a lot of mocks you don't have to write, it's easy to keep up to date, and you get the real deal, date-format-wise.

This technique is called snapshotting, and guess what, it's something we can generalize.

Snapshot

I'm already delegating half of the tests writing to ChatGPT, then triggering a breakpoint inside them, so I can steal the results of the calls and turn them into assertions. It’s a nice trick, it works in many situations.

Surely there is a way to automate that even more!

This is where you pip install inline-snapshot, a lib that lets you write placeholders where your test results should appear:

from inline_snapshot import snapshot

def test_something():
    assert 1548 * 18489 == snapshot() # snapshot() is the placeholder

Then run your tests to record the results:

pytest --inline-snapshot=create

And you get the tests filled with the values:

from inline_snapshot import snapshot

def test_something():
    assert 1548 * 18489 == snapshot(28620972)

You can now check if the values make sense (don't trust them, your code might be buggy), and if they are valid, commit them to git. Next time you run them normally (without --inline-snapshot), the data from the snapshots will be used transparently and tested against.

You can use --inline-snapshot=update to update the snapshots that changed automatically, --inline-snapshot=fix to only change the tests that are not passing anymore, or --inline-snapshot=review to manually review the changes and approve them, and even --inline-snapshot=disable to go back back to raw values, and speed up things or check if the snapshotting was the cause of an issue.

The review is interactive and quite handy:

What are those external(), you might ask.

It’s what you use if you don’t want the snapshot in your test code. The data may be very big, or be some non-text format that would mess up your file.

You can mark a test result as outsourced:

from inline_snapshot import outsource, snapshot

def test_a_big_fat_data():
    assert [
        outsource("long text\n" * times) for times in [50, 100, 1000]
    ] == snapshot()

Yes, you put it on the thing that produces the result, not on the snapshot part of the test.

Then, when you --inline-snapshot=create, it will end up like this:

from inline_snapshot import outsource, snapshot, external

def test_a_big_fat_data():
    assert [
        outsource("long text\n" * times) for times in [50, 100, 1000]
    ] == snapshot(
        [
            external("362ad8374ed6*.txt"),
            external("5755afea3f8d*.txt"),
            external("f5a956460453*.txt"),
        ]
    )

Files are stored in the default pytest config dir, and stale ones can be removed using --inline-snapshot=trim.

I like this lib because you can put the values of the snapshot directly into the code, though. I think for many tests, it makes more sense than to load it from a file.

But it has a big limitation: you can only test things that can be recreated using  repr()‘s output, so mostly Python built-ins like strings, ints, lists dicts, and some objects like datetime. If you need more complicated objects, you are out of luck.

In this case, you can use the more powerful pytest-insta, which allows pickle as a serialization format, so you can snapshot almost everything. And if you can't, you can create your own formatter, and use dark magic like cloudpickle to really turn any monster into flat bytes.

Next article, we’ll get philosophical.