Today I want to speak about – Python’s Quest for Simplicity. Specifically around Python tooling, where we will look at the past, present, and future of a typical Python project.
Past
First, let's step back in time (a few years) and look at a traditional Python project ...
This looks pretty standard, but there are a few issues. Such as:
- Multi config files - there are a ton of files, mainly consisting of the various config files used by the various Python linters and formatters. This clutters up the directory.
- Virtual environments - The use of the Python virtual environment module for creating virtual environments to isolate our project dependencies causes a few issues. Namely, we have to know the location when activating the virtual environment. You can somewhat overcome this by placing the virtual environment in the project directory as we have done in the example above with
.venv, but this then means you have to add exclusions to your various linters and formatters to ensure you are checking the forest of files and folders that live within the virtual environment. - Pip - Pip as our package manager for installing Python packages. Pip stores all dependencies into a single list; therefore, there is no way to “easily” see what dependency is required for which package, which can cause issues when dealing with dependency conflicts.
- Multiple tools - We are using multiple tools. 1 tool for package management, 1 tool for virtual environments, various tools for each of our formatters and then various tools for our linting.
Present-ish
Time passed, and a new tool called Poetry (intro guide here) which combined the abilities of package and virtual environment management was released. Not only this, but it overcame many of the nuances of previous tools, such as dealing with virtual environment locations and being unable to determine the full dependency tree. The main configuration file that Poetry used was called pyproject.toml.
After a few years of Poetry being released, the tool inspired the formal adoption of pyproject.toml via PEP 621 as a central file for storing the configuration for Python tooling.
PEP 621 - Storing project metadata in pyproject.toml
This PEP specifies how to write a project’s core metadata in a pyproject.toml file for packaging-related tools to consume.
- https://peps.python.org/pep-0621/
At this point, many of the tools (but not all, **cough, Flake8** ) came on board and provided the option to store their config data within this new unicorn of config files pyproject.toml.
Future-ish
At this point, our project had now evolved to:
Things were looking good. We have removed the issues with Pip and virtual environments by using Poetry. And with the adoption of PEP 621, we had centralized many of our configuration files. However, we still needed multiple tools for the formatting and linting of our Python code.
At this stage, we now see the next iteration of this simplification via their recently (beta-) released tool - Ruff.
charliermarshOn the surface, when you first see Ruff, the main benefit is around the speed enhancements over the traditional linters. And though true and a great benefit, the thing that Ruff also brings to the party (and in my opinion, the most important) is that it performs the roles of many of the current linting and formatting tools, bringing them into a single tool.
Now at the time of writing, this Ruff is still in beta, and though it does a lot, there are some tools that you would still run alongside it. Such as Black. So is it worth ditching all of your other tools? At the moment, I would say try out ruff, watch the project, kick the tyres and run it alongside your other tools.
But the main takeaway to this post is where things are going and the tools that can help you simplify your Python project's (namely Poetry and Ruff).
That wraps up this post. I hope you've enjoyed it, and thanks for reading.
