Only problem: pip freeze gives me every transitive dependency; when sharing code, it's much nicer to see explicitly requested dependencies so users / contributors know what are the actual functional dependencies of the project. The freeze output should really go in a lock file of some kind for full reproducibility, but seeing a thousand dependencies with fixed versions is a lot of cognitive load when you're pruning or upgrading dependencies. Do you not find this to be a problem?
As strongly stated in both articles, there is no universal solution that solves all problems. I just want to help the most people possible suffer less often.
All persons that can should self exclude from the article.
Although probably less people than you might think: we have a tendency to over solve problems we don't really have, in our industry, and it's hard to realize that, in fact, you could do with a little imperfection here and there and have your life vastly simplified in a lot of situations.
And if you can self exclude, you'd have to remember you may work in a team, and your project must balance it all for all members. Having a beautiful list of clean dependencies that are easy to update once a year does nothing good if 60% of your colleagues fight the packaging system every week. Even if you are fine.
I can't remember when was the last time I read something that resonated with me 100% like your article. Thank you for saving me countless hours I would have waisted without your guide.
thanks for the extensive tutorial on virtual environments and pip.
I have been using Python for more than 10 years now and I found it painful until very recently.
I believe your tutorial make it easy but it doesn't answer 2 big questions:
- how can I install multiple versions of Python on the same machine? Pyenv solves this problem, but you don't suggest to use it. Do you have an alternative?
- why should I not use Poetry? Just saying not to use it, it's not enough. You should give a good motivation not to use it. Otherwise it is just a matter of taste. I like Poetry and I prefer it to keeping the venv and the requirements.txt in sync.
I'll shortly write a Substack article on how I streamlined an easy setup using Poetry, virtualenv and Devenv.sh (which internally use Nix)
Thank you so much for this! I've been confused about virtual environments for a long time. This really, really helps. However, this all raises a couple questions for me:
1. If I create a virtual environment for all the little scripts I write for myself, do I have to activate that virtual environment (or use the long awkward path somehow) whenever I want to run one of those scripts? That seems painful
2. If I want to distribute a script I wrote to someone else, do they now have to learn how to use virtual environments? That seems extra painful.
In the part about "Recreating an virtual environment" you probably meant "pythonX.x -m pip install -r requirements.txt" right? Since we shouldn't call pip without python -m ?
Only problem: pip freeze gives me every transitive dependency; when sharing code, it's much nicer to see explicitly requested dependencies so users / contributors know what are the actual functional dependencies of the project. The freeze output should really go in a lock file of some kind for full reproducibility, but seeing a thousand dependencies with fixed versions is a lot of cognitive load when you're pruning or upgrading dependencies. Do you not find this to be a problem?
It is a problem, but it's not as common as all other problems this recipe resolves.
Personally, when I need only the direct dependencies (and dev deps), I use pip-tools or uv, so I acknowledge the need.
However, if I'm being honest, I really benefit of using it in one project out of five. A lot of projects are fine with just pip freeze.
And that's the case for most people in the python community. Certainly that's the case for the majority of people that have "python packaging problems", as explained in https://bitecode.substack.com/p/why-not-tell-people-to-simply-use.
As strongly stated in both articles, there is no universal solution that solves all problems. I just want to help the most people possible suffer less often.
All persons that can should self exclude from the article.
Although probably less people than you might think: we have a tendency to over solve problems we don't really have, in our industry, and it's hard to realize that, in fact, you could do with a little imperfection here and there and have your life vastly simplified in a lot of situations.
And if you can self exclude, you'd have to remember you may work in a team, and your project must balance it all for all members. Having a beautiful list of clean dependencies that are easy to update once a year does nothing good if 60% of your colleagues fight the packaging system every week. Even if you are fine.
Nice. So I added
```
alias pym='python3 -m'
alias pyp='python3 -m pip'
```
in my `.bashrc` config file to avoid suffering ;-)
If you want to go full prevention, you can even do (with pip version 21+):
python3 -m pip config set global.require-virtualenv True
This way you can't even install anything outside of a venv.
Or you can change your alias to:
alias pyp='python3 -m pip --require-virtualenv'
I can't remember when was the last time I read something that resonated with me 100% like your article. Thank you for saving me countless hours I would have waisted without your guide.
thanks for the extensive tutorial on virtual environments and pip.
I have been using Python for more than 10 years now and I found it painful until very recently.
I believe your tutorial make it easy but it doesn't answer 2 big questions:
- how can I install multiple versions of Python on the same machine? Pyenv solves this problem, but you don't suggest to use it. Do you have an alternative?
- why should I not use Poetry? Just saying not to use it, it's not enough. You should give a good motivation not to use it. Otherwise it is just a matter of taste. I like Poetry and I prefer it to keeping the venv and the requirements.txt in sync.
I'll shortly write a Substack article on how I streamlined an easy setup using Poetry, virtualenv and Devenv.sh (which internally use Nix)
For anyone interested here, I wrote this article about setting up a Python virtual environment with Nix.
https://cloudnativeengineer.substack.com/p/effortless-python-development-with-nix
I have summarized some of the ideas here but provided an alternative as well.
Well, you are in luck.
Running several version will work out of the box with: https://www.bitecode.dev/p/installing-python-the-bare-minimum
Why not poetry: https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
Thank you so much for this! I've been confused about virtual environments for a long time. This really, really helps. However, this all raises a couple questions for me:
1. If I create a virtual environment for all the little scripts I write for myself, do I have to activate that virtual environment (or use the long awkward path somehow) whenever I want to run one of those scripts? That seems painful
2. If I want to distribute a script I wrote to someone else, do they now have to learn how to use virtual environments? That seems extra painful.
1/ You scripts can have a shebang line pointing to the venv:
#!/path/to/your/venv/bin/python
Mark them as executable on linux and mac, and you can run them as-is.
On windows, the py launcher understands shebang as well.
2/ Indeed, it's one of the most annoying points with Python.
While I hope PEP 723 (https://www.bitecode.dev/i/136686826/pep-follows-up-on-pep) will help one day, meanwhile you can only 3 alternatives:
- install it yourself
- package it as a zip app using something like shiv (but it's same plateform only)
- compile it as a stand alone exe using something like nuitka
All of them are a pain in one way or another.
one more question:
3. You don't check the virtual environment directory into version control, even though you put it in your project directory, correct?
Correct. You check in the requirement file.
Great article!
In the part about "Recreating an virtual environment" you probably meant "pythonX.x -m pip install -r requirements.txt" right? Since we shouldn't call pip without python -m ?
Indeed.