Explaining why the Python installation process is such a mess

Well, sit down. I hope you're not in a rush.

Apr 07, 2023

Summary

Python installation experience is lacking. This is not because nobody wants to fix it. A lot of work has been done to improve it already. But the age and popularity of the language make the task extremely complicated.

Nevertheless, providing a cross-platform installer shipping self-contained binaries and normalizing how you run the executable would help a lot. Several projects are experimenting with those ideas.

What’s a nice language like you doing in an installer like this?

In previous articles, we noted there are many ways to install Python and I recommended to avoid most of them.

Instead, I provided an installation procedure that goes back to basics, to increase the chances that things go well.However, if you read this last post, this is likely what you said to yourself:

Damn, that's a long article...
Wow, I didn't realize you should do that!
Wait, it's different for each OS?
*mumble in beard something nasty about Linux*

Even this utterly simplified procedure, which is the most reliable one I could get after years of helping people with their Python troubles, is not a fun experience.

In this article, we are going to explore what is the problem with the current ways of installing Python, why this hasn't been fixed already, what would a nice experience look like, and finally some attempts that have been made by the community to go into that direction.

Everything, everywhere, all that once. And none of them work.

I won't surprise you if I say the first issue of the Python installation process is how many ways there are to do it. People are faced with too many options, and have no time, desire and sometime skills to evaluate them.

But the paradox of choices is only the tip of the iceberg, the real trouble arises when the user uses several of them, because they don't play well with each other. Often, they will fight for the PATH.

And the PATH is what's winning or losing the war. Because when people call "python" or "pip", it's not uncommon they have several versions of those on their machine, and have no idea which one they are calling. We solve that int the recipes given the weeks before, but this requires users knowing about it, and having discipline. Requiring knowledge and discipline is the opposite of what a streamlined experience looks like.

Since we are mentioning several versions of "python" or "pip", another issue is that running those require a different magical incantation depending on the OS you are using and the method of installation. You may use the "py launcher" (recommended on Windows, but only available through the official installer), version suffixes (on Unix, or with the windows store), a GUI (anaconda launcher), a single entry point rewritten depending on your context (pyenv)...

Even if none of that were a problem, the installation may very well be broken, which may happen because they are shipped that way (homebrew, the windows store), because of a failed installation process (pyenv) or because of missing dependencies (Linux packages).

It's not like nobody knows about it, so why is it not fixed?

I mean it's one banana, Michael. What could it cost?

When somebody mentions the installation problems that come with Python, you soon see a deluge of people asking why nobody is fixing this.

But of course people are working on this.

They have been working on this for many, many years.

In fact, the installation process is tremendously better than it used to be, and it keeps improving:

pip and venv are now bundled on most platforms.
ensurepip is at the disposable of package maintainers to help them provide pip.
the "py launcher" exists on Windows. Before that, it was a nightmare to run several versions of Python on this OS.
python-full packages are slowly being rolled out in Debian to provide most of what people expect after installing python.
python 3 64bits is promoted everywhere. You would not believe how long it took before the 32bits version of Python was taken out of the python.org home page. And how long before Python 3 was the default in most Linux distros.

So the community is investing in this.

But every time you fix something, something is broken by somebody else:

you provide pip and venv, only to see distros splitting them in several packages.
you provide the "py launcher" and the python shipped in the Microsoft Store don't contain it.
you provide a beautiful new mac installer, and you have to redo the whole thing for Apple Silicon.
you get the Apple Silicon thing right, and now macOS policy prevents force you to prompt for installing SSL certificates manually.

All languages are not equal in this: if you install nodejs on debian, it will come with npm, it's not a separate package to install. There is no broken rust compiler in the windows store. They don't have to deal with a very popular commercial distribution like Anaconda that does everything in a different way.

On the other hand, Python has unique requirements:

It has a very rich and thriving ecosystem of compiled extensions, particularly in the scientific and ML stack. This means a python package may contain C, Fortran or assembly.
It is used as a plugin language in many systems like SIG, 3D modeling, IDE, etc.
It's a core language used in several OS. Python is shipped with Mac and Linux because it's a dependency.
There are a lot of Python flavors. What you call Python is in fact CPython, the reference implementation of Python in C. But there is Jython (Python on the JVM), IronPython (Python on .Net), pypy (Python on a JIT), Graal Python (Python on GraalVM), Brython (Python in JS), Pyodide (Python on WASM), uPython (Python for microcontroller), Rust Python (Python in Rust) and many others.

In that regard, Python has it quite bad.

And so as you can expect...

Fixing things in Python requires a lot of effort

Python is very old.

It was created in 1991, that's 4 years before Java. To give you a reference of how old this is in terms of computer systems, that's 5 years before people could play Pokemon on the original Game Boy. The one with 8KB of ram, and a black and white screen that worked with AAA batteries.

At this time there was no unicode everywhere, not multi-core CPU in your PC, no broadband in your home, and certainly no other package managers. Python had to solve each problem early, and several times.

This means Python has to pay for three decades of technical debts.

Add on top of that Python is very popular. The community is simply huge. And very diverse, with a large spectrum of skills, environments, objectives, constraints, etc. Youtube and Instagram devs are not PostGIS plugin authors. A bank analyst has a very different idea of stability than a student at the university. What the Fedora maintainers needs are nowhere near what the Django creators at the Washington post are looking for.

You can't just cowboy your way into a change and break things. Remember that the migration between Python 2 and 3 took 15 years, or half the language entire life, because of the inertia of this gigantic ball of thousands of user profiles.

Changing anything is a very long and hard process.

The process to make any change at the language level reflects this. You can't decide by yourself. You discuss it with people, then you write a PEP. Then you must defend it. Sometimes against users straight from XKCD 1172.

And once you solve all those puzzles, you haven't started actually implementing the solution yet. Which will take eons to roll out, because nothing is rolled out on a tiny scale in Python. You want to deprecate something? This is a commitment to a process that will literally take years, between the moment you mark something as deprecated, until the moment you can actually remove the old feature.

And when you will remove the old feature, someone, somewhere will very publicly curse you for it.

And somebody else will insult, literally, the core devs for not breaking things faster because they want a new feature.

After all that hardship, the victory is not guaranteed yet:

There are a lot of things that are not in our hands. OS maintainer policies may still undermine your goals.
You need to deal with politics now to convince people to adopt the new way. Yes, geeks dealing with politics. What could go wrong?
No matter what you provide, it will be lost in the noise for a long time, thanks to thousands of resources that are still talking about the old way of doing things. And the FUD is strong.
You will always have projects that will do things differently anyway, like Anaconda, Jython, or microPython, because they need to.

What a solution could look like?

While there is not such thing as a unique solution to a cluster of very different and complicated problems, there are technical issues that a community of motivated nerds could tackle.

A much better installation experience would have to tick the following boxes:

Avoid interacting with all existing python installation. We can't solve the bad interactions, but we can just avoid having any, bypassing the problem entirely.
Completely self contained binaries. No compilation, no certificates to install, headers, tkinter, pip and venv are always provided, on all systems..
Installing is the same workflow on every OS.
Running Python is the same on every OS (no -m vs suffix vs shims).
Allow to easily install a lot of versions of Python.
Deal with the problem of upgrading python.
Make sure the PATH is dealt with. Nobody wants to deal with "command not found".
Provide a CLI and a GUI. One or the other will let a big chunk of users behind.

This means some kind of installer like homebrew, but specialized in installing only Python. It should download a compiled binaries, and place it at the user directory (no admin, no pip breaking the system), and make sure the PATH points to it. It should use version suffixes down to the minor version. This will avoid conflict with any existing Python on the PATH since nobody does this, and allow for installing as many upgrades as you want without breaking previously created virtual environments.

The compiled binary part is more complicated on Linux, since there is not such thing as a generic Linux system. Fortunately, you don't need to have your binaries to work on all distros. You can target one libc + openssl version that covers the most popular and supported ones. That would already help immensely.

The installer could pop a window offering a list of Python versions to install, the ones that are installed, and offer to create a virtual environment for those python where you desire, explaining the commands while doing so.

Of course, for the terminal inclined, a CLI would allow something like this on any OS:

$ magic_installer list installed
3.7.1
3.7.2
3.8.5
3.8.9
3.8.10
3.10

$ magic_installer list installable [--minor --pre-release --eol]
3.7 
3.8 
3.9
3.10
3.11

And:

$ magic_installer install 3.9

> selecting lastest cPython version: 3.9.12
> downloading cPython 3.9.12
#################################### 100%
> cPython 3.9.12 is installed

If you plan to use "pip", you should create a virtual environment:

python3.9.12 -m venv .venv

And activate it:

source .venv/bin/activate # or .venv\Scripts\activate.bat on Windows

After that, you can use "python -m pip install" to install packages.

But this magic installer doesn't exist.

Let's see what does exist so far.

Since installing and running Python has been the source of a lot of pain for so many users, you can imagine that we are not the first ones to think of alternatives.

One possibility is to focus on the "running python" problem alone. This is exactly what the core dev Brett Cannon has been experimenting with in his "py launcher for Unix". It brings the Windows "py launcher" to Mac and Linux.

Personally, I entertained the idea of a compiled stand alone binary, and even worked with Anthony Sottile on it for a day. Well, he worked on it and I tried to keep up. But the result was nothing like Gregory Szorc's project python-build-standalone which does this, only seriously. The most recent release offers more than 150 executables, ranging from 3.8 to 3.10, for numerous OS, architectures and constraints.

David O'Connor's pyflow, goes farther, playing with the idea of a CLI tool that manages the project, the dependencies, and install Python if you don't have it.

Individually, none of those projects will cut the Python installation Gordian knot.

However, they do show that people care about the issue and are trying new things, concepts that may be reused by other projects, and maybe, one day, we will have the magic installer.

I hope this article would have given you a better understanding of the Python installing situation, and who knows, one reader may get inspired to start coding something. After all, this is FOSS. We are part of the solution.

Bite code!

Discussion about this post