Envisioning a future python

I love python. It was the first language I learned when I got into programing. I wrote silly python programs and eventually websites using knowledge I got from a book titled "Head First Python: A Brain-Friendly Guide". I don't write a lot of python nowadays but I've slowly been getting back into it. Coming back to python I can't help but find a lot of rough edges, paper cuts that should be fixed but haven't been. I wanted to envision a future python, one which resolves these issues and makes python a little bit more modern. To start with I'm going to look at the good, bad and ugly sides of python.

The Good

Python's biggest strength is its syntax. It's simple and easy enough that it's often used as a first language. Additionally, its 3rd party library of packages means that it can be used for a wide variety of applications from web development to the embedded world. As a minor plus python can have multiple versions that run alongside each other with no issues, something some languages struggle with.

The Bad

Virtualenvs are a requirement for any serious development in python. The concept of a venv is simple, a python interpreter completely isolated from the rest of the system. It is the foundation for poetry and pipenv. While this is a integral part of the language today I have always believed this was a hack. Pip is the other half of this problem. Pip is outdated and can't take into account all installed packages, instead focusing on the one package required for the transactions and its dependencies. The conda world cuts out pip and offers a more robust solution but still falls back to venvs.

The ugly

While the previous points referred to problems fixable with code, the ugly comes from the culture of python. The python 2 to 3 migration was a shitshow. It took till 2020 to finally kill of python 2 despite python 3 being in the works since 2008. Rewriting from python 2 to 3 was also made more difficult by the auto porting tooling 2to3 being extremely naive. The result is that python 4.0 should have been released a while back but was not. The main evidence of this is since python 3.10 the asyncio stdlib has changed to be subtly incompatible with the previous versions.

Finally there is the no-gil project. It is a very useful project to remove the global interpreter lock from python. The aim is during python 3 to land this as the default mode. This would be again subtly incompatible with previous versions. The fallout of this will not be known until the switch happens. Many package maintains may not even want to support no-gil until it is impossible to ignore it.

Learning from other languages

Looking at other languages I have the most experience with the javascript and rust ecosystem. Both have their own issues but the js ecosystem has the best solution to the problem from my perspective. node reads node_modules by default and can be influenced to load packages via loaders, which is ~~abused~~ used by ts-node and others. pyproject.toml is a step in this direction allowing for a much much better definition of a library and dependencies. If one were to build upon pyproject.toml the logical step would be to create something like pyproject_modules.

Why not virtualenvs? Virtualenvs are a hack. A virtualenv is a little like this, a folder with a interpreter, pip and a few other bits name a folder tree thats used for python packages. This then requires you to run a activate file that sets a bunch of env variables like PYTHONPATH which points the compiler to where to read from. This is and always has been a hack to work around env pollution when you use pip without a virtualenv. Pip's other major shortcoming is its inability to update all packages instead requiring you to manually update everything and this can and will lead to issues. This is also why many distributions disable pip, requiring you to use a virtualenv and instead use the host package manager to handle system python packages.

Merely updating the language with features, without updating the tooling will result in the language stagnating. Haskell learned this lesson quite painfully. Another lesson from javascript the python ecosystem is already using, writing everything in python is not a requirement. Ruff is a linter and formater written in rust. Its performance is magnitudes better than anything else in the ecosystem. Going by this, taking an approach similar to bun would be the smart choice. Bun replaces the node runtime with one written in zig as well as a package manager all bundled together. For python maybe not a replacement for the runtime(I do know there are alternatives like rustpython for runtime) but instead a all in one tool to replace pip, pipenv and the like.

A Proposed Solution

When we look at a solution, the major task is replacing virtualenvs. Here we can look at the language closest to us. Javascript. Utilizing the standard pyproject.toml we can envision a npm like tool replacing virtualenvs with a pyproject_modules folder containing all the libraries utilized with the project. This tool would handle the full life-cycle of the project. This being, installation, updating and finally removing packages either directly or indirectly. The tool would possibly also handle uploading the project to pypi. Python, either through the use of environment variables or, more preferably, through a loader system (akin to node) would find the require libraries in the pyproject_modules folder. The tool could also execute commands via a script definition system.

What next?

Finally I am coming to the problem I stated was the ugly. Python treats minor versions like Major versions. Asyncio has had a big impact in the python world. However the subtle runtime to the asyncio library that have shipped as minor versions, make the world of pypi slowly fall apart, bitrot until it's impossible to use. Slowly old libraries with limited users will fall apart. There are libraries in the wild on pypi that are already broken. As a community we need to take the step towards being more open to major versions.

Adblocker not detected