Python 3.12: what didn't make the headlines
It's less interesting, but it's a niche I can fill
Summary
Performances are underwhelming
Pathlib has been improved
We get a better debugging experience
Slice objects are now hashable
An unexpected math.sumprod() appears
New cool command-line interfaces
Heavy deprecation round
It's release time!
Python 3.12 has been released, and there are thousands of articles out there to repeat what the released notes already say. Some are pretty good, though, so if you haven't read RealPython's intro on the new typing features yet, add it to your list.
And yes, I'm a fan of the better error messages (they are for me the best feature of the release), I applaud the formalization of f-string parsing, and hurrah for itertools.batched, but we have been discussing that for ages on Twitter.
What is everyone not talking about?
The performance let down
When you get an amazing product like Python for free, it's not a good look to send bad vibes to the people that have been sweating hard to bring this gift to the world.
So it's understandable people don't really want to address the elephant in the room: the performances of this new version don't live up to the hype.
By this, I mean the fact that MS is paying a team with Guido to speed up the language, and that we were all quite giddy when they announced that the target was a 50% gain at each release for the next 5 ones. We saw that coming, but still.
The release notes talk about a 5% gain in general, an underwhelming number, but I wanted to check by myself. So I grabbed 3.11 and 3.12, installed the headers, grabbed pyperformance and ran it on my laptop with:
pyperformance run --python=/usr/bin/python3.11 -o py311.json
pyperformance run --python=/usr/bin/python3.12 -o py312.json
pyperformance compare py311.json py312.jsonYou know what they say about benchmarks... Plus, despite the fact I killed everything I could and let the machine alone for the solid hour it ran, getting some serious numbers is very difficult.
That being said, here is the report:
| py311.json | |
| ========== | |
| Performance version: 1.0.9 | |
| Report on Linux-5.15.0-84-generic-x86_64-with-glibc2.31 | |
| Number of logical CPUs: 8 | |
| Start date: 2023-10-03 15:46:40.037527 | |
| End date: 2023-10-03 16:30:51.189955 | |
| py312.json | |
| ========== | |
| Performance version: 1.0.9 | |
| Report on Linux-5.15.0-84-generic-x86_64-with-glibc2.31 | |
| Number of logical CPUs: 8 | |
| Start date: 2023-10-03 16:51:38.432027 | |
| End date: 2023-10-03 17:38:09.827299 | |
| ### 2to3 ### | |
| Mean +- std dev: 310 ms +- 13 ms -> 285 ms +- 13 ms: 1.09x faster | |
| Significant (t=10.34) | |
| ### async_generators ### | |
| Mean +- std dev: 304 ms +- 53 ms -> 371 ms +- 6 ms: 1.22x slower | |
| Significant (t=-9.81) | |
| ### async_tree_cpu_io_mixed ### | |
| Mean +- std dev: 624 ms +- 47 ms -> 676 ms +- 20 ms: 1.08x slower | |
| Significant (t=-7.82) | |
| ### async_tree_cpu_io_mixed_tg ### | |
| Mean +- std dev: 589 ms +- 26 ms -> 679 ms +- 36 ms: 1.15x slower | |
| Significant (t=-15.78) | |
| ### async_tree_io ### | |
| Mean +- std dev: 934 ms +- 9 ms -> 992 ms +- 29 ms: 1.06x slower | |
| Significant (t=-15.00) | |
| ### async_tree_io_tg ### | |
| Mean +- std dev: 905 ms +- 5 ms -> 1014 ms +- 19 ms: 1.12x slower | |
| Significant (t=-41.81) | |
| ### async_tree_memoization ### | |
| Mean +- std dev: 493 ms +- 15 ms -> 521 ms +- 33 ms: 1.06x slower | |
| Significant (t=-6.03) | |
| ### async_tree_memoization_tg ### | |
| Mean +- std dev: 465 ms +- 12 ms -> 525 ms +- 21 ms: 1.13x slower | |
| Significant (t=-19.37) | |
| ### async_tree_none ### | |
| Mean +- std dev: 433 ms +- 75 ms -> 412 ms +- 8 ms: 1.05x faster | |
| Significant (t=2.18) | |
| ### async_tree_none_tg ### | |
| Mean +- std dev: 364 ms +- 4 ms -> 390 ms +- 7 ms: 1.07x slower | |
| Significant (t=-25.80) | |
| ### asyncio_tcp ### | |
| Mean +- std dev: 611 ms +- 5 ms -> 348 ms +- 3 ms: 1.76x faster | |
| Significant (t=353.16) | |
| ### asyncio_tcp_ssl ### | |
| Mean +- std dev: 2.11 sec +- 0.01 sec -> 1.17 sec +- 0.02 sec: 1.80x faster | |
| Significant (t=303.48) | |
| ### bench_mp_pool ### | |
| Mean +- std dev: 5.74 ms +- 1.20 ms -> 14.17 ms +- 6.45 ms: 2.47x slower | |
| Significant (t=-9.94) | |
| ### bench_thread_pool ### | |
| Mean +- std dev: 704 us +- 76 us -> 772 us +- 77 us: 1.10x slower | |
| Significant (t=-4.83) | |
| ### chameleon ### | |
| Mean +- std dev: 5.14 ms +- 0.08 ms -> 6.23 ms +- 0.13 ms: 1.21x slower | |
| Significant (t=-55.85) | |
| ### chaos ### | |
| Mean +- std dev: 50.4 ms +- 0.7 ms -> 55.9 ms +- 0.8 ms: 1.11x slower | |
| Significant (t=-40.36) | |
| ### comprehensions ### | |
| Mean +- std dev: 17.6 us +- 0.3 us -> 19.0 us +- 0.2 us: 1.08x slower | |
| Significant (t=-29.67) | |
| ### coroutines ### | |
| Mean +- std dev: 18.2 ms +- 0.3 ms -> 19.7 ms +- 0.3 ms: 1.08x slower | |
| Significant (t=-27.58) | |
| ### coverage ### | |
| Mean +- std dev: 317 ms +- 5 ms -> 634 ms +- 14 ms: 2.00x slower | |
| Significant (t=-168.43) | |
| ### create_gc_cycles ### | |
| Mean +- std dev: 905 us +- 6 us -> 972 us +- 7 us: 1.07x slower | |
| Significant (t=-52.69) | |
| ### crypto_pyaes ### | |
| Mean +- std dev: 58.6 ms +- 1.0 ms -> 65.6 ms +- 1.5 ms: 1.12x slower | |
| Significant (t=-29.99) | |
| ### dask ### | |
| Mean +- std dev: 375 ms +- 14 ms -> 461 ms +- 27 ms: 1.23x slower | |
| Significant (t=-22.00) | |
| ### deepcopy ### | |
| Mean +- std dev: 271 us +- 3 us -> 326 us +- 9 us: 1.20x slower | |
| Significant (t=-44.68) | |
| ### deepcopy_memo ### | |
| Mean +- std dev: 27.6 us +- 0.5 us -> 31.7 us +- 0.4 us: 1.15x slower | |
| Significant (t=-48.08) | |
| ### deepcopy_reduce ### | |
| Mean +- std dev: 2.46 us +- 0.03 us -> 3.10 us +- 0.05 us: 1.26x slower | |
| Significant (t=-88.00) | |
| ### deltablue ### | |
| Mean +- std dev: 2.82 ms +- 0.03 ms -> 2.71 ms +- 0.05 ms: 1.04x faster | |
| Significant (t=14.81) | |
| ### django_template ### | |
| Mean +- std dev: 26.1 ms +- 0.4 ms -> 33.7 ms +- 0.8 ms: 1.29x slower | |
| Significant (t=-64.15) | |
| ### docutils ### | |
| Mean +- std dev: 1.93 sec +- 0.02 sec -> 2.24 sec +- 0.05 sec: 1.16x slower | |
| Significant (t=-42.17) | |
| ### dulwich_log ### | |
| Mean +- std dev: 48.7 ms +- 0.7 ms -> 54.4 ms +- 1.1 ms: 1.12x slower | |
| Significant (t=-33.24) | |
| ### fannkuch ### | |
| Mean +- std dev: 287 ms +- 5 ms -> 315 ms +- 4 ms: 1.10x slower | |
| Significant (t=-36.51) | |
| ### float ### | |
| Mean +- std dev: 55.9 ms +- 1.0 ms -> 68.9 ms +- 1.3 ms: 1.23x slower | |
| Significant (t=-63.74) | |
| ### gc_traversal ### | |
| Mean +- std dev: 2.60 ms +- 0.02 ms -> 2.83 ms +- 0.03 ms: 1.09x slower | |
| Significant (t=-52.25) | |
| ### generators ### | |
| Mean +- std dev: 53.2 ms +- 0.7 ms -> 26.3 ms +- 0.4 ms: 2.02x faster | |
| Significant (t=250.74) | |
| ### genshi_text ### | |
| Mean +- std dev: 17.1 ms +- 0.3 ms -> 20.8 ms +- 0.5 ms: 1.22x slower | |
| Significant (t=-50.48) | |
| ### genshi_xml ### | |
| Mean +- std dev: 40.7 ms +- 0.7 ms -> 45.6 ms +- 0.6 ms: 1.12x slower | |
| Significant (t=-40.56) | |
| ### go ### | |
| Mean +- std dev: 107 ms +- 2 ms -> 112 ms +- 2 ms: 1.04x slower | |
| Significant (t=-16.10) | |
| ### hexiom ### | |
| Mean +- std dev: 4.79 ms +- 0.06 ms -> 4.80 ms +- 0.08 ms: 1.00x slower | |
| Not significant | |
| ### html5lib ### | |
| Mean +- std dev: 50.7 ms +- 2.0 ms -> 53.2 ms +- 2.5 ms: 1.05x slower | |
| Significant (t=-6.08) | |
| ### json_dumps ### | |
| Mean +- std dev: 9.77 ms +- 0.12 ms -> 9.29 ms +- 0.23 ms: 1.05x faster | |
| Significant (t=14.27) | |
| ### json_loads ### | |
| Mean +- std dev: 16.6 us +- 0.3 us -> 22.1 us +- 3.0 us: 1.34x slower | |
| Significant (t=-14.48) | |
| ### logging_format ### | |
| Mean +- std dev: 5.17 us +- 0.10 us -> 6.25 us +- 0.14 us: 1.21x slower | |
| Significant (t=-48.23) | |
| ### logging_silent ### | |
| Mean +- std dev: 76.1 ns +- 1.5 ns -> 89.3 ns +- 2.3 ns: 1.17x slower | |
| Significant (t=-36.73) | |
| ### logging_simple ### | |
| Mean +- std dev: 4.76 us +- 0.06 us -> 5.65 us +- 0.13 us: 1.19x slower | |
| Significant (t=-47.77) | |
| ### mako ### | |
| Mean +- std dev: 7.24 ms +- 0.15 ms -> 9.37 ms +- 0.24 ms: 1.29x slower | |
| Significant (t=-58.60) | |
| ### mdp ### | |
| Mean +- std dev: 1.97 sec +- 0.02 sec -> 2.44 sec +- 0.06 sec: 1.24x slower | |
| Significant (t=-56.93) | |
| ### meteor_contest ### | |
| Mean +- std dev: 81.6 ms +- 0.8 ms -> 89.9 ms +- 2.1 ms: 1.10x slower | |
| Significant (t=-28.63) | |
| ### nbody ### | |
| Mean +- std dev: 68.0 ms +- 1.0 ms -> 71.6 ms +- 1.7 ms: 1.05x slower | |
| Significant (t=-14.31) | |
| ### nqueens ### | |
| Mean +- std dev: 70.7 ms +- 0.9 ms -> 82.1 ms +- 3.6 ms: 1.16x slower | |
| Significant (t=-23.67) | |
| ### pathlib ### | |
| Mean +- std dev: 13.7 ms +- 0.2 ms -> 15.7 ms +- 0.3 ms: 1.14x slower | |
| Significant (t=-50.10) | |
| ### pickle ### | |
| Mean +- std dev: 6.71 us +- 0.11 us -> 8.28 us +- 0.09 us: 1.23x slower | |
| Significant (t=-85.78) | |
| ### pickle_dict ### | |
| Mean +- std dev: 21.6 us +- 0.4 us -> 21.6 us +- 0.3 us: 1.00x faster | |
| Not significant | |
| ### pickle_list ### | |
| Mean +- std dev: 2.78 us +- 0.04 us -> 3.23 us +- 0.07 us: 1.16x slower | |
| Significant (t=-41.39) | |
| ### pickle_pure_python ### | |
| Mean +- std dev: 236 us +- 12 us -> 269 us +- 3 us: 1.14x slower | |
| Significant (t=-19.91) | |
| ### pidigits ### | |
| Mean +- std dev: 175 ms +- 3 ms -> 179 ms +- 3 ms: 1.02x slower | |
| Significant (t=-7.66) | |
| ### pprint_pformat ### | |
| Mean +- std dev: 1.18 sec +- 0.01 sec -> 1.52 sec +- 0.04 sec: 1.29x slower | |
| Significant (t=-58.58) | |
| ### pprint_safe_repr ### | |
| Mean +- std dev: 573 ms +- 7 ms -> 753 ms +- 14 ms: 1.31x slower | |
| Significant (t=-91.17) | |
| ### pyflate ### | |
| Mean +- std dev: 325 ms +- 4 ms -> 358 ms +- 5 ms: 1.10x slower | |
| Significant (t=-41.65) | |
| ### python_startup ### | |
| Mean +- std dev: 6.72 ms +- 0.26 ms -> 8.15 ms +- 0.26 ms: 1.21x slower | |
| Significant (t=-55.41) | |
| ### python_startup_no_site ### | |
| Mean +- std dev: 4.86 ms +- 0.15 ms -> 6.44 ms +- 0.31 ms: 1.32x slower | |
| Significant (t=-65.44) | |
| ### raytrace ### | |
| Mean +- std dev: 225 ms +- 3 ms -> 274 ms +- 3 ms: 1.22x slower | |
| Significant (t=-84.20) | |
| ### regex_compile ### | |
| Mean +- std dev: 108 ms +- 2 ms -> 123 ms +- 2 ms: 1.14x slower | |
| Significant (t=-40.31) | |
| ### regex_dna ### | |
| Mean +- std dev: 141 ms +- 1 ms -> 144 ms +- 1 ms: 1.03x slower | |
| Significant (t=-15.26) | |
| ### regex_effbot ### | |
| Mean +- std dev: 2.17 ms +- 0.03 ms -> 2.35 ms +- 0.04 ms: 1.08x slower | |
| Significant (t=-30.02) | |
| ### regex_v8 ### | |
| Mean +- std dev: 15.8 ms +- 0.1 ms -> 17.6 ms +- 0.2 ms: 1.11x slower | |
| Significant (t=-56.43) | |
| ### richards ### | |
| Mean +- std dev: 37.2 ms +- 0.9 ms -> 37.6 ms +- 1.4 ms: 1.01x slower | |
| Not significant | |
| ### richards_super ### | |
| Mean +- std dev: 45.1 ms +- 1.1 ms -> 42.1 ms +- 1.2 ms: 1.07x faster | |
| Significant (t=14.52) | |
| ### scimark_fft ### | |
| Mean +- std dev: 209 ms +- 2 ms -> 246 ms +- 3 ms: 1.17x slower | |
| Significant (t=-69.77) | |
| ### scimark_lu ### | |
| Mean +- std dev: 72.9 ms +- 1.1 ms -> 92.9 ms +- 2.1 ms: 1.27x slower | |
| Significant (t=-67.02) | |
| ### scimark_monte_carlo ### | |
| Mean +- std dev: 50.1 ms +- 0.9 ms -> 58.3 ms +- 0.8 ms: 1.16x slower | |
| Significant (t=-51.06) | |
| ### scimark_sor ### | |
| Mean +- std dev: 86.9 ms +- 1.9 ms -> 102.6 ms +- 1.4 ms: 1.18x slower | |
| Significant (t=-51.19) | |
| ### scimark_sparse_mat_mult ### | |
| Mean +- std dev: 3.21 ms +- 0.08 ms -> 3.36 ms +- 0.11 ms: 1.05x slower | |
| Significant (t=-8.89) | |
| ### spectral_norm ### | |
| Mean +- std dev: 70.4 ms +- 1.2 ms -> 87.2 ms +- 1.4 ms: 1.24x slower | |
| Significant (t=-69.46) | |
| ### sqlglot_normalize ### | |
| Mean +- std dev: 221 ms +- 2 ms -> 103 ms +- 1 ms: 2.15x faster | |
| Significant (t=350.67) | |
| ### sqlglot_optimize ### | |
| Mean +- std dev: 41.2 ms +- 0.5 ms -> 50.2 ms +- 0.5 ms: 1.22x slower | |
| Significant (t=-100.42) | |
| ### sqlglot_parse ### | |
| Mean +- std dev: 1.04 ms +- 0.02 ms -> 1.12 ms +- 0.02 ms: 1.07x slower | |
| Significant (t=-25.70) | |
| ### sqlglot_transpile ### | |
| Mean +- std dev: 1.28 ms +- 0.02 ms -> 1.40 ms +- 0.02 ms: 1.10x slower | |
| Significant (t=-30.61) | |
| ### sqlite_synth ### | |
| Mean +- std dev: 1.96 us +- 0.04 us -> 2.36 us +- 0.04 us: 1.21x slower | |
| Significant (t=-55.03) | |
| ### sympy_expand ### | |
| Mean +- std dev: 388 ms +- 4 ms -> 443 ms +- 19 ms: 1.14x slower | |
| Significant (t=-21.91) | |
| ### sympy_integrate ### | |
| Mean +- std dev: 16.1 ms +- 0.2 ms -> 17.3 ms +- 0.3 ms: 1.07x slower | |
| Significant (t=-26.28) | |
| ### sympy_str ### | |
| Mean +- std dev: 227 ms +- 2 ms -> 252 ms +- 3 ms: 1.11x slower | |
| Significant (t=-52.55) | |
| ### sympy_sum ### | |
| Mean +- std dev: 121 ms +- 1 ms -> 128 ms +- 2 ms: 1.05x slower | |
| Significant (t=-19.10) | |
| ### telco ### | |
| Mean +- std dev: 5.22 ms +- 0.16 ms -> 6.47 ms +- 0.12 ms: 1.24x slower | |
| Significant (t=-49.13) | |
| ### tomli_loads ### | |
| Mean +- std dev: 1.64 sec +- 0.03 sec -> 1.98 sec +- 0.06 sec: 1.21x slower | |
| Significant (t=-41.96) | |
| ### tornado_http ### | |
| Mean +- std dev: 86.7 ms +- 2.1 ms -> 92.5 ms +- 2.3 ms: 1.07x slower | |
| Significant (t=-14.57) | |
| ### typing_runtime_protocols ### | |
| Mean +- std dev: 369 us +- 5 us -> 138 us +- 3 us: 2.67x faster | |
| Significant (t=288.51) | |
| ### unpack_sequence ### | |
| Mean +- std dev: 32.8 ns +- 0.7 ns -> 32.0 ns +- 0.5 ns: 1.03x faster | |
| Significant (t=7.23) | |
| ### unpickle ### | |
| Mean +- std dev: 9.40 us +- 0.15 us -> 12.33 us +- 0.42 us: 1.31x slower | |
| Significant (t=-51.26) | |
| ### unpickle_list ### | |
| Mean +- std dev: 2.88 us +- 0.07 us -> 3.71 us +- 0.11 us: 1.29x slower | |
| Significant (t=-48.07) | |
| ### unpickle_pure_python ### | |
| Mean +- std dev: 176 us +- 2 us -> 201 us +- 3 us: 1.14x slower | |
| Significant (t=-52.27) | |
| ### xml_etree_generate ### | |
| Mean +- std dev: 58.5 ms +- 1.0 ms -> 86.5 ms +- 2.2 ms: 1.48x slower | |
| Significant (t=-88.39) | |
| ### xml_etree_iterparse ### | |
| Mean +- std dev: 80.6 ms +- 0.8 ms -> 80.3 ms +- 1.2 ms: 1.00x faster | |
| Not significant | |
| ### xml_etree_parse ### | |
| Mean +- std dev: 119 ms +- 2 ms -> 117 ms +- 2 ms: 1.01x faster | |
| Not significant | |
| ### xml_etree_process ### | |
| Mean +- std dev: 40.7 ms +- 0.5 ms -> 58.7 ms +- 1.3 ms: 1.44x slower | |
| Significant (t=-98.24) | |
| Skipped 2 benchmarks only in py311.json: sqlalchemy_declarative, sqlalchemy_imperative |
You'll notice that 14 tests run faster, but 79 run slower.
You can't draw conclusions from that, of course, yet I can see this being not the best-selling point for the release.
The pathlib improvements
It's almost a footnote in the release, but it's actually really great news.
First, Path can now be subclassed. There is a lot of pain around pathlib because of some weird magic it used to do that made projects like path unable to be fully compatible.
Personally, I had several times in my career where I wanted a FS path wrapper, and just add to go through a lot of troubles just because this basic Python feature was not working as expected, which you don't know at first, of course.
Cherry on top, pathlib.Path.walk(), a Path() aware version of os.walk() is now available, which will make a few of my scripts look tidier.
Finally, pathlib.Path.glob() can now be case-insensitive. That's groovy.
A better debugging experience
Again, I love that the trend of better error messages continues. If I ever meet Pablo Galindo, I will pay him a Michelin restaurant. But that's not the only level up we get.
Pretty much nobody cared about comprehension inlining, but you cannot imagine how many times I move in pdb only to be stuck in this weird stack frame artificially created for comprehensions to work.
The removal of this is not just giving us better perfs and ergonomics, it's also removing a big "WTF" that all beginners will experience using the Python debugger with nobody in sight to explain to them what's going on. Not to mention tracers everywhere will rejoice at not seeing those marked mistakenly as function calls.
Another very, VERY nice change is the addition of magic variables in PDB: $_retval (for the value returned in the block) and $_exception (for the exception raised in the block).
Imagine you have this code:
def bar():
breakpoint()
return random.randint(0, 100)How do you read the return value in pdb? You can't, and have to get it from outside the function call, or create an intermediary variable. Not anymore:
-> return random.randint(0, 100)
(Pdb) $_retval
3If you add the improved monitoring machinery which should make observability cheaper and more accurate, plus support for linux perf profiler, it makes me very happy. Maybe one day we'll get time travel debugging.
slice objects are now hashable
You may not know it but:
dict keys can be something else that strings (tuples in there are quite handy);
you can create slice objects that represent the [x:y:z] operations on lists and pass that to
__getitem__.
So it seems logical to have a mapping (or set) of possible slicing, but it was not, up to this release.
I didn't know I wanted this before they told me I couldn't :)
math.sumprod()
In my last project, we had sum(itertools.starmap(operator.mul, zip(p, q))) in a lot of places, then we created a function for it, then we realized we had bugs because of length issues and added frenetically strict=True everywhere.
Seems like 3.12 is solving that once and for all.
New command line interface
The fact you can start a web server with python -m http.server is almost a meme at this point, and I love those little utilities.
3.12 adds an sqlite shell:
python3.12 -m sqlite3
sqlite3 shell, running on SQLite version 3.31.1
Connected to a transient in-memory database
Each command will be run using execute() on the cursor.
Type ".help" for more information; type ".quit" or CTRL-D to quit.
sqlite>And my favorite, an uid generator:
python3.12 -m uuid
a3b10cab-064d-456e-ac9a-58a125171218Deprecations, deprecations, deprecations
This version marks a ton of stuff to be removed, like telnetlib, 2to3 or datetime.utcnow(). 3.13 better be not released a Friday.
It also removes distutils, right now, and setuptools is kicked out of ensurepip.
You will have to test Python 3.12 a bit more vigorously than usual before migrating, which, of course, you should only do in one year. Right?
Very nice!
Nice!