What's up in the Python community?

June 2023

Summary

• Python 3.7 reaches EOL, the egg format is deprecated

• The GIL is the center of attention once again

• struct keyword and callable modules proposals

• New people at the Python Software Foundation

• Python 3.12.0 beta 3, several Pypy, pydantic 2 and Mypy 1.4

• PyCon US videos are out

Python reaches 3.7 EOL, the egg format is deprecated

That's the most important news of the month: Python 3.7 has reached End-of-Life status, which means the release is frozen, and will not receive changes in the future. From now on, libraries and systems will likely stop supporting it.

Unless you have very tight maintenance and security objectives, this doesn't mean you have to stop using 3.7 right now. It will not drop dead and stop working, current libraries supporting it will not suddenly be pulled out of Pypi. Many projects around the world will continue to use it for years to come.

But it's the signal that you have to start to think about migrating in the future. And of course, if it's possible, avoid starting a new project with it.

Since we are talking about deprecation, PEP 715, "Disabling bdist_egg distribution uploads on PyPI", has been accepted. Eggs was the old format to distribute binaries, like for c-extensions such as numpy, and has been replaced by the excellent wheels. While the old eggs will stay in the Pypi basket, if you try to upload a new one, you will now receive a notice about the deprecation. August, the 1st, uploads will get rejected.

As a user, you probably don't need to care about this one, but maintainers of very old projects might.

The GIL is the center of attention once again

There has been a lot of discussion around the Global Interpreter Lock again. After all, using all those sweet, sweet, CPU cores is still hard in Python.

Sam Gross’ PEP 703 was the proposal du jour to remove it; and after so many attempts that failed over 2 decades, this one looked like it could work. It had a clever idea: let’s ship the change with a bunch of unrelated performance improvements, so that the users don't feel like they have lost speed.

But the GIL is a very sensitive topic, that touches the way Python deals with concurrency, garbage collecting and affects heavily popular compiled extensions. It would also impact the current progress Microsoft is paying for to get a faster Python.

One suggested solution is to release two Python, one with the GIL, one without. Yet, this is also not so simple. As Guido van Rossum stated:

If there’s one lesson we’ve learned from the Python 2 to 3 transition, it’s that it would have been very beneficial if Python 2 and 3 code could coexist in the same Python interpreter. We blew it that time, and it set us back by about a decade.

This is all getting more complicated, thanks to the possible support of multiple interpreters in Python 3.13 stdlib, something that should help the concurrency story without the need to remove the GIL.

Despite the controversial nature of the topic, the whole thing has stayed incredibly civil. It's a tremendous improvement over the Python mailing list toxic shamble we used to have 20 years ago. Still, several users have made some strong (but polite) statements about the relevance of Python in today's world is if this is not sorted out. They can be summarized with the last Peter Gaultney's message:

To be brutally honest, a Python that leaves these types of dramatic evolutions sitting on the workbench gathering dust is a Python that, as far as many of us are concerned, is already on its way out

Fair to say this is not a solved problem, nor does the community has a definitive conclusion on this one.

Struct keyword and callable modules proposals

Brett Cannon, a core dev working at Microsoft, made a proposal for a new struct syntax in Python. The goal would be to have something like C structs, a lightweight group of several related data. Like a dataclass or a SimpleNamespace, but immutable, without methods, optional type hints and no import necessary. Unlike namedtuple, it's not iterable and doesn't require exec():

struct Point(x: int, y: int)

IMO, such a construct would be a welcome addition to Python: easier to use than TypedDict, lighter than anything class-based, strictly structured, and makes it easy to get good perfs while being readable yet short to type.

Unrelated, but there was also a sizeable discussion in the community about callable modules, proposing to allow:

import pprint
pprint(stuff)

Instead of:

from pprint import pprint
pprint(1, 2)

Note that this is already possible, with a hack:

>>> from pprint import pprint as old_pprint
... from types import ModuleType
... import sys
... class _CallableModule(ModuleType):
...
...     def __call__(self, *args, **kwargs):
...         return old_pprint(*args, **kwargs)
...
... sys.modules["pprint"].__class__ = _CallableModule

So it's about giving an official mechanism for it, and encouraging it's usage.

I like the idea, especially for interactive sessions. Some modules, like fraction, glob or datetime have an obvious default callable you expect. I get how it could get abused, and used for things that are not intuitive at all, though.

New people at the Python Software Foundation

PyPI had a lot going on last month, and even an outage this June, which highlight how much works it requires. It's no surprise, at the peak of 2021, the service served nearly 900 terabytes over more than 2 billion requests... per day!

That's a lot for a service that is free to use for the entire world. A lot, which costs around 2 million dollars a month in hosting.

A system of this size comes with a surface of attack and abuses of all kind. It even led to enforce 2FA four weeks ago. This month, the PSF made the hiring of Seth Michael Larson as a dedicated security developer official. He is the current lead maintainer of urllib3 and requests, and has started working on the Python Package Index this week.

If this kind of environment seems like a good fit for you, there is a new job opening as Deputy Developer in Residence. This is a supporting role that will help CPython maintainers and the Steering Council focus on their core mission.

There are also two less technical offers: a community events coordinator (PyCon US being the main dish here) and a community communications Mmanager (being an official voice of the PSF).

Finally, the Board Election Results are in.

Python 3.12.0 beta 3, several Pypy, pydantic 2 and Mypy 1.4

Python 3.12 goes into beta 3, we are getting very close to a release candidate there. Still, it's far from stable, as you can see with the surprising AttrDict feature being added, then removed. The ticket about it is very interesting, and an eye-opener on the challenges that comes with including anything in the stdlib (BTW, ipython and VSCode have completion on dict keys).

The Pypy team also released several new versions, the highlight being one of them now supports Python 3.10.

Pydantic 2 is finally out. We talked about it last month, the release is anticipated because of a new core, rewritten in Rust, that can bring up to a 50x in perf. Something welcome given validating big documents can be slow with this lib. It's time to test it. Not put it in prod, but at least play with it.

Mypy turns 1.4-teen, with improvements in type narrowing and some speedup.

Oh, and PyCon US videos are out.

Here.