What's up Python? Setuptools breaks, a better PDB, cPython on CalVer...

July 2024

Aug 01, 2024

Setuptools breaks packages, removing the test command
Tian Gao shared design goals on a better pdb
Proposal for Python to follow calendar versioning
Polars is stable, moving to 1.0, as numpy reaches 2.0.
Facebook' Segment Anything Model version 2 impresses
Long debate on how to handle the pain of publishing cPython on mobile app stores

Setuptools breaks packages

A few days ago setuptools, a library used in many packages as an installer, released its version V72, removing support for calling setup.py test to run said package tests. The test command was proposed for deprecation in 2017, so 7 years ago, but the Python community is massive, cares about stability, and comes with a proportional inertia.

So it took time to be removed.

And it was not enough.

A few days ago a ticket has been opened to signal that many packages were not installable anymore after this change. If you wonder how a test command could break package installation, that's one of the numerous downsides of using an executable format such as setup.py: it executes code, things get imported, and if they are not there, it breaks.

That's why the current best practice is to use pyproject.toml for metadata, and the wheel format to package things. It's all declarative, so those problems are less likely to happen.

Meanwhile, setuptools decided to reverse the removal with an almost immediate V72.1 bump.

Bottom line: if you have packages failing to mention "test", move out of 72.0.

And if you are still using setuptools or setup.py as a lib publisher, consider migrating your package to more modern practices if your use case allows it, which is almost always the case.

A better PDB

Python Language Summit 2024 was in June, meaning we are starting to get the presentation summaries coming out. One I particularly liked was "Should we make pdb better" from Python core dev Tian Gao.

I love PDB despite it being very raw (and we have a tutorial on it). But yes, it shows its age, and given we are getting a great REPL makeover for Python 3.13, I would love the same for our debugger.

The main areas of suggested improvements are:

Showing more lines of code around the current breakpoint.
Colors in the terminal, syntax highlighting.
Customization, with defaults being safe.
Handling of more scenarios (threads, asyncio, bytecode, remote debugging)
Better support for attach-to-process debuggers.
Moving from sys.trace to sys.monitoring to improve perfs.

Basically, making ipdb almost unnecessary, and debugging web servers much less of a pain.

Now since this is just a presentation, you might think this all theoretical. But Tian Gao is the defacto maintainer of pdb, he also has several projects to help with debugging (such as watchpoint and viztracer) and discussed it with Pablo Galindo Salgado (our error message/shell lord and savior).

The likelihood it all happens is very high.

Calendar versioning

Guido stated many times there will be no Python 4. We are stuck in V3 forever.

Or are we?

In PEP 2026, Hugo van Kemenade, Python 3.14/15 release manager make the proposal to move from major.minor.micro to a version based on the year of release, since Python now follows a yearly release schedule.

He explains people may think Python follows SemVer, and therefore use the version as an indicator of stability guarantee. But it's not the case:

People often assume Python follows SemVer and complain about breaking changes in feature releases. But Python predates SemVer by at least 15 years: the SemVer spec was introduced in 2009 and the bespoke Python scheme was added to source control in 1994 for the 1.0 release.

And indeed, Python doesn't follow SemVer, the number "3" never moves, and the "minor" part is actually incremented at every major release. Release that can contain breaking changes, although they usually come with years of deprecation warnings in advance.

Kemenade also added:

Since 2019, we’ve made a release each year:
3.15.0 will be released in 2026
3.16.0 will be released in 2027
3.17.0 will be released in 2028
3.18.0 will be released in 2029
3.19.0 will be released in 2030
This is sort of calendar-based, it’s just that it’s offset by 11 years.

CalVer for Python would look like this:

3.26.0 will be released in 2026
3.27.0 will be released in 2027
3.28.0 will be released in 2028
3.29.0 will be released in 2029
3.30.0 will be released in 2030

So we are still stuck in version 3 :)

But at least, deprecation warnings would give you an idea of what year they would start being enforced. As the PEP specifies:

DeprecationWarning: ‘ctypes.SetPointerType’ is deprecated and slated for removal in Python 3.15

Doesn't tell you much. While:

DeprecationWarning: ‘ctypes.SetPointerType’ is deprecated and slated for removal in Python 3.26

Tells you in 2 years your code will break.

And:

$ python3 --version
Python 3.8.3

If not super informative. Yet:

$ python3 --version
Python 3.19

Would have told me my stack is 5 years old.

I was not super sold the first time I heard of the idea, but it seems practical in this light.

Polars is stable

By now, you may have noticed Rust and Python are kinda BFF these days. Tools like uv and ruff are rusted just for the snake, pydantic gets a rusty core, and pyo2 lets you rewrite your hot paths seemingly in the compiled language.

The scientific stack is also ripping the benefits of this, and for the last years, Polars, an alternative to pandas, was oxidizing its way toward version 1.0, which it reached this july.

Polars is basically a dataframe-oriented lib that is faster than pandas, with a different but more congruent API. It's written in Rust, with Python and JS binding, using Apache arrow to transparently share data between the 3.

So far it had a lot of rough edges and I couldn't recommend it over pandas, so this stable release is very much welcome, and I have good hopes that I'll soon be able to spread the gospel.

This is funny because numpy 2 also came out with a ton of perf improvements and old API removal. Guess it's friendly competition over there.

SAM 2, he is back

You might not have heard of SAM, the Segment Anything Model, open-sourced and released by Facebook a while ago, but it was already a superb tool for automatic detouring.

You basically could give it any image, and it would isolate any object in it, creating a mask you could use to isolate edit it, extract it, etc.

It could even use a prompt to choose what to segment in an image.

All that a pip install away.

An entirely new SAM 2 just came out, which has all the image capabilities of the original sammy, but also works on videos!

Note that, sure, it's wrapped in Python, like most of AI projects, but remember you still need a pretty heavy model and a hefty GPU to run all that. You are not going to get that out of a Rasberry Pi.

Python feels the pain of app stores

For a couple of years now, efforts have been ongoing to enable Python dev on the most popular mobile platforms. It's been possible for longer than that, but there was no official Python build targetting specifically, say, the iPhone.

Recently though, a debate has sparked on the level of effort that should be made to be accepted on the App Store.

Indeed, the upgrade from 3.11 to Python 3.12 was rejected because the parse.py module contained the "itms-services" url scheme string, something Apple scans for automatically, but didn't explain in any way in their rejection. A typical move.

Russell Keith-Magee, working at beeware and a Django core developer, asked if "acceptable to app stores" should be a design goal for CPython, or if that compliance should be a problem left to the tools that generate application bundles for app stores, adding to nobody's surprise that:

Apple's review processes were the most ""paranoid and inscrutable"" of app-store-review processes

Gaynor's suggested, from his own experience coding the cryptography library, that while accepting a workaround from external contributions can help, he thinks we should strive to request so guarantee third parties will solve the issue in the long run to avoid:

letting large firms simply externalize their bizarre issues onto OSS projects

But good luck getting that from Cupertino.

The most pragmatic solution proposed is to have an --with-app-store-compliance option added to the release process that will apply a set of patches targetting the particular platform. But I doubt it will be the end of the story.