Skip to main content

Managing Version, Virtual Environments and Dependencies with Pyenv and Poetry

Software developers always have several projects on their computers using different languages, versions of those languages, libraries, and tools. So that one project does not interfere with another, they must be isolated somehow. In the case of Python projects, we will need tools that manage Python versions, virtual environments, and project dependencies. There are several options, but let's focus on two:

  1. pyenv manages different versions of Python on the same machine
  2. poetry manages virtual environments and project dependencies within those virtual environments.

How Python Virtual Environments Work

Unless you specify the full path, a command must be searched by the operating system to run. The search is done in a list of directories registered in the environment variable called PATH. The search is stopped when the first match is found and the program runs.

For example, if PATH contains $HOME/.local/bin:/usr/local/bin:/usr/bin and you run the command python --version, then the shell will look for an executable named python in $HOME/.local/bin first, then in /usr/local/bin and last in /usr/bin. The first executable file named python that is found stops the search and is executed with the parameter --version. If no file is found, then an error message is displayed.

The next concept is that a virtual environment in Python is just a directory that contains the desired Python version and the libraries needed for the project. Activating or deactivating a virtual environment is done by manipulating the list of paths contained in PATH. Activation puts the virtual environment directory at the beginning of the list, and deactivation removes it from the list of the current session.

Installing and Managing Python Versions with pyenv

With several old and new projects on the same computer, each will likely use a different version of Python. The tool that will allow us to install and choose where and which version of Python to use is pyenv.

pyenv Installation

On Ubuntu/Debian/Mint, the installation of pyenv installation is done by its installer:

$ curl https://pyenv.run | bash

On MacOs, use brew to install pyenv:

$ brew update
$ brew install pyenv

For pyenv work correctly, you need to add the following lines to your shell configuration file. For Bash, this file is ~/.bashrc:

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

Next, it is necessary to start a new terminal session or run exec $SHELL to restart the current session.

Installing Python Versions

To see all Python versions available for installation, use the command pyenv install --list:

$ pyenv install --list | grep ' 3.9'
  3.9.0
  3.9-dev
  3.9.1
  3.9.2
  3.9.3
  3.9.4

To install versions 3.7.10 and 3.9.4, use:

$ pyenv install 3.7.10
$ pyenv install 3.9.4

The command pyenv versions lists all installed versions available:

$ pyenv versions
* system (set by $HOME/.pyenv/version)
  3.7.10
  3.9.4

How pyenv Works

pyenv inserts $HOME/.pyenv/shims at the begining of PATH to intercept calls to python and other related commands. This intercept leads to special executables called shims, which redirect the calls to a specific python version 1, depending on the first configuration level found, in the following order:

  1. Shell. Version registered in the environment variable PYENV_VERSION. You can use the command pyenv shell <version> to set this variable in your current shell session, or use another equivalent command such as export PYENV_VERSION=<version>.
  2. Local. Version registered in a file .python-version, which is searched recursively from the current directory until reaching the root directory. You can use the command pyenv local <version> to generate this file.
  3. Global. Version registered in the file $(pyenv root)/version, can also be generated by the command pyenv global <version>.
  4. System. If no configuration is found, the Python version installed on the operating system is used.

Virtual Environments and Dependency Management with poetry

poetry Installation

There are a few options to install poetry. The most recommended one is using its installer, which prevents poetry dependencies from mixing with those of other libraries:

$ curl https://install.python-poetry.org | python -

On Linux, the installation is done in the directory $HOME/.local/bin. If this directory is not in your PATH, then you need to manually add it to your shell configuration file (~/.bashrc or ~/.profile) with the following lines:

if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

# pyenv configuration goes here

To test the installation, run:

$ poetry --version

If the command does not return any error message, then the installation was completed successfully!

Poetry Initial Configuration

Poetry's configuration is kept in the $HOME/.config/pypoetry/config.toml. But instead of accessing this file directly, you should use the command poetry config and its subcommands.

To list the settings, run:

$ poetry config --list

After the installation, it is advisable to modify where poetry keeps virtual environments. The default configuration creates virtual environments inside the directory $HOME/.cache/pypoetry, but it is better that each virtual environment is created inside the project root, named after .venv, because editors such as VSCode can automatically find it and configure intellisense.

To change it, run:

$ poetry config virtualenvs.in-project true

Poetry Commands

To display available commands, run:

$ poetry

Let's focus on some related to the most common tasks:

  1. Create a new project
  2. Manage dependencies
  3. Manage the virtual environment

Creating a New Project

The command poetry new creates a new project with a basic structure of directories and files. For example:

$ poetry new project-x --name app

results in the following structure:

project-x
├── app
│   └── __init__.py
├── pyproject.toml
├── README.rst
└── tests
    ├── __init__.py
    └── test_app.py

--name is optional and allows defining a different name for the project package directory. The poetry's default is to use the same project name.

The file pyproject.toml is the project's configuration file. Contains information such as name, version, dependencies, etc.

Managing Dependencies

The main commands related to dependency management are:

  1. poetry add. Adds a dependency to the project.
  2. poetry remove. Removes a dependency from the project.
  3. poetry show. Displays project dependencies.
  4. poetry update. Updates project dependencies.

Project dependencies can be divided into two groups: packages essential for project functioning and packages used only for project development, in activities such as testing and linting . When deploying the project in a staged or production environment, it is necessary to install packages from the first group, but those from the second group should be avoided to reduce the installation size.

Adding the main packages is done by the command poetry add. For example:

$ poetry add fastapi jinja2 aioredis[hiredis] databases loguru passlib[argon2]
Creating virtualenv project-x in /tmp/project-x/.venv
Using version ^0.70.0 for fastapi
Using version ^3.0.3 for Jinja2
Using version ^2.0.0 for aioredis
Using version ^0.5.3 for databases
Using version ^0.5.3 for loguru
Using version ^1.7.4 for passlib

Updating dependencies
Resolving dependencies... (8.8s)

Writing lock file

Package operations: 28 installs, 0 updates, 0 removals

• Installing idna (3.3)
• Installing pycparser (2.21)
• Installing sniffio (1.2.0)
• Installing anyio (3.4.0)
• Installing cffi (1.15.0)
• Installing greenlet (1.1.2)
...

Since versions have not been explicitly specified, the latest available versions of the packages and their dependencies are used.

To add development packages, use poetry add --dev:

$ poetry add --dev pytest alt-pytest-asyncio pytest-cov asgi-lifespan isort blue mypy httpx
The following packages are already present in the pyproject.toml and will be skipped:

• pytest

If you want to update it to the latest compatible version, you can use `poetry update package`.
If you prefer to upgrade it to the latest available version, you can use `poetry add package@latest`.

Using version ^0.6.0 for alt-pytest-asyncio
Using version ^3.0.0 for pytest-cov
Using version ^1.0.1 for asgi-lifespan
Using version ^5.10.1 for isort
Using version ^0.7.0 for blue
Using version ^0.910 for mypy
Using version ^0.21.1 for httpx

Updating dependencies
Resolving dependencies... (10.3s)

Writing lock file

Package operations: 25 installs, 0 updates, 0 removals

• Installing appdirs (1.4.4)
• Installing certifi (2021.10.8)
• Installing click (8.0.3)
• Installing h11 (0.12.0)
...

As packages are added, the file pyproject.toml is updated with information about dependencies. Note that development dependencies are in a separate section:

[tool.poetry.dependencies]
python = "^3.10"
fastapi = "^0.70.0"
Jinja2 = "^3.0.3"
aioredis = {extras = ["hiredis"], version = "^2.0.0"}
databases = "^0.5.3"
loguru = "^0.5.3"
passlib = {extras = ["argon2"], version = "^1.7.4"}

[tool.poetry.dev-dependencies]
pytest = "^5.2"
alt-pytest-asyncio = "^0.6.0"
pytest-cov = "^3.0.0"
asgi-lifespan = "^1.0.1"
isort = "^5.10.1"
blue = "^0.7.0"
mypy = "^0.910"
httpx = "^0.21.1"

Poetry accepte versions based on standard SemVer (Major.Minor.Patch) and uses some conventions to version update specification. The ^ notation indicates that a version upgrade is allowed if the new version number does not modify the left most non-zero digit in the grouping Major.Minor.Patch. For example, poetry update fastapi would accept any version >=0.70.0 and <0.71.0.

During the installation of the development dependencies, you might have noticed that pytest was not installed because it was already included by the command poetry new during project creation. However, the version 5.2 is not the latest version available. The first impulse is to try to update the version using the command poetry update pytest, but the specification ^5.2 does not allow a newer version such as 6 to be used. The solution is to explicitly add the latest pytest version as a development dependency:

$ poetry add --dev pytest@latest
Using version ^6.2.5 for pytest

Updating dependencies
Resolving dependencies... (0.7s)

Writing lock file

Package operations: 1 install, 1 update, 2 removals

• Removing more-itertools (8.12.0)
• Removing wcwidth (0.2.5)
• Installing iniconfig (1.1.1)
• Updating pytest (5.4.3 -> 6.2.5)

This installs the package and also updates the section [tool.poetry.dev-dependencies] in pyproject.toml with the new version.

Note that several other indirect dependencies are installed in addition to the specified packages. The exact relationship with all installed packages, their versions and other additional information is stored in the file poetry.lock. This file ensures that everyone in the project uses exactly the same package versions.

Rather than looking into poetry.lock directly, the best way to see the complete list of project dependencies is with the command poetry show --tree:

$ poetry show --tree
aioredis 2.0.0 asyncio (PEP 3156) Redis support
├── async-timeout *
│   └── typing-extensions >=3.6.5
├── hiredis >=1.0
└── typing-extensions *
alt-pytest-asyncio 0.6.0 Alternative pytest plugin to pytest-asyncio
└── pytest >=3.0.6
    ├── atomicwrites >=1.0
    ├── attrs >=19.2.0
    ├── colorama *
    ├── iniconfig *
    ├── packaging *
    │   └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
    ├── pluggy >=0.12,<2.0
    ├── py >=1.8.2
    └── toml *
asgi-lifespan 1.0.1 Programmatic startup/shutdown of ASGI apps.
└── sniffio *
blue 0.7.0 Blue -- Some folks like black but I prefer blue.
├── black 21.7b0
│   ├── appdirs *
│   ├── click >=7.1.2
│   │   └── colorama *
│   ├── mypy-extensions >=0.4.3
│   ├── pathspec >=0.8.1,<1
│   ├── regex >=2020.1.8
│   └── tomli >=0.2.6,<2.0.0
└── flake8 3.8.4
    ├── mccabe >=0.6.0,<0.7.0
    ├── pycodestyle >=2.6.0a1,<2.7.0
    └── pyflakes >=2.2.0,<2.3.0
...

Enabling The Virtual Environment

If you just cloned the project, you should use it poetry install to create the virtual environment directory and install the dependencies.

Once installed in .venv, there are two options to enable the virtual environment through the terminal. The first is to activate the virtual environment with the command poetry shell:

$ poetry shell

(project-x) $

The command line prompt changes to show activation. To disable the virtual environment, you can run exit, press CTRL+D, or just open a new terminal.

The second option is to use poetry run <command>, that activates the virtual environment, runs the command, and then exits the virtual environment. It is particularly useful for running in scripts.

For example, be a project containing one Makefile with the following task:

test:
    pytest --cov-report term-missing --cov-report html --cov-branch \
        --cov app/

You can launch the virtual environment and then run make test to run the tests, or you can run poetry run make test in one step.

Final Considerations

This article has covered the main points you need to know to use the pyenv + poetry combination for managing Python project independent ecosystems. Some final considerations:

  1. Do not share virtual environments between different projects.
  2. The virtual environment directory should not be kept under version control because it can be rebuilt as needed through the pyproject.toml and poetry.lock files.
  3. Do not manipulate the virtual environment directory manually. Always use poetry commands for this.
  4. For other details and poetry commands, visit the command section in the project documentation.

Complementary References

1 pyenv: How It Works
2 Managing Multiple Python Versions With pyenv
3 Modern Python Environments - dependency and workspace management
4 Pyenv + Poetry
5 Overview of python dependency management tools
6 Pipenv and Poetry: Benchmarks & Ergonomics
7 Pipenv and Poetry: Benchmarks & Ergonomics II

Comments

Comments powered by Disqus