Tox - testing on different versions of Python

Introduction

When you develop libraries or applications in Python, it is crucial to ensure their compatibility with various interpreter versions. This becomes especially important when publishing packages on PyPI, where users may be using different Python versions. Manually installing Python 3.7, 3.8, 3.9, 3.10 and testing each version consumes a lot of time and effort.

This is where tox comes in – a powerful test‑automation tool that allows you to run tests in isolated virtual environments with different Python versions. Tox handles automatic dependency installation, creates clean environments, and guarantees reproducible test results.

What Is tox and Why You Need It

Tox (short for “testing out of the box”) is a command‑line tool written in Python that automates the testing process for Python projects across multiple environments. It creates isolated virtual environments, installs dependencies, and runs tests, enabling you to verify code compatibility with different Python versions and dependency sets.

Key Benefits of tox:

Environment isolation – each test runs in its own virtual environment, eliminating dependency conflicts.

Automation – fully automated creation of environments, dependency installation, and test execution.

Multi‑version support – ability to test simultaneously on several Python versions.

Reproducibility – guarantees identical results across different environments.

CI/CD integration – easy configuration for automated testing in continuous‑integration pipelines.

Installation and Initial Setup

Installing tox

You can install tox in several ways:

# Via pip
pip install tox

# Via conda
conda install -c conda-forge tox

# Via pipx (recommended)
pipx install tox

Creating a Basic Configuration File

The main tox configuration file is tox.ini, which should reside in the project’s root directory:

[tox]
envlist = py38, py39, py310, py311

[testenv]
deps = 
    pytest
    pytest-cov
commands = 
    pytest tests/ --cov=src/

Project Structure

For optimal tox operation, a typical project layout looks like this:

my_project/
├── src/
│   └── my_package/
│       ├── __init__.py
│       └── module.py
├── tests/
│   ├── __init__.py
│   └── test_module.py
├── tox.ini
├── setup.py (or pyproject.toml)
└── requirements.txt

Getting Started with tox

Basic Commands

Tox provides a straightforward command‑line interface:

# Run all environments from envlist
tox

# Run a specific environment
tox -e py39

# Run multiple environments
tox -e py38,py39,lint

# Force recreation of environments
tox -r

# Verbose output
tox -v

# Very verbose output (debugging)
tox -vv

How tox Works

The tox workflow consists of the following steps:

1. Configuration parsing – reads tox.ini and determines the environment list.
2. Virtual‑environment creation – builds an environment for each specified Python interpreter.
3. Package installation – installs the package under test into each environment.
4. Dependency installation – installs the dependencies listed in the deps section.
5. Command execution – runs the commands defined in the commands section.
6. Result aggregation – collects test results.

Detailed tox.ini Configuration

Core Sections

The tox.ini file is divided into several main sections:

Section [tox]

Contains global settings:

[tox]
# Default environment list
envlist = py38, py39, py310, py311, pypy3

# Minimum tox version
minversion = 3.8.0

# Skip missing interpreters
skip_missing_interpreters = true

# Isolated build
isolated_build = true

Section [testenv]

Base configuration for all environments:

[testenv]
# Test dependencies
deps = 
    pytest>=6.0
    pytest-cov
    pytest-mock
    requests

# Commands to run
commands = 
    pytest {posargs:tests/} --cov=src/ --cov-report=term-missing

# Pass-through environment variables
passenv = 
    CI
    GITHUB_*
    PYTEST_*

# Set environment variables
setenv =
    PYTHONPATH = {toxinidir}
    COVERAGE_FILE = {envtmpdir}/.coverage

# Additional pip options
pip_pre = false
download = true

Specialized Environments

You can define dedicated environments for specific tasks:

[testenv:lint]
description = Code style checks
deps = 
    flake8
    black
    isort
    pylint
commands = 
    black --check --diff src/ tests/
    isort --check-only --diff src/ tests/
    flake8 src/ tests/
    pylint src/

[testenv:typecheck]
description = Type checking
deps = 
    mypy
    types-requests
commands = 
    mypy src/ --strict

[testenv:security]
description = Security analysis
deps = 
    bandit[toml]
    safety
commands = 
    bandit -r src/ -f json
    safety check

[testenv:docs]
description = Documentation build
deps = 
    sphinx
    sphinx-rtd-theme
commands = 
    sphinx-build -b html docs/ docs/_build/

Testing Across Different Python Versions

Specifying Python Versions

You can list various Python interpreters in the envlist section:

[tox]
envlist = 
    py37, py38, py39, py310, py311  # CPython
    pypy3                           # PyPy
    jython                          # Jython (if installed)

Installing Multiple Python Versions

For managing several Python releases, consider the following tools:

pyenv (Linux/macOS)

# Install pyenv
curl https://pyenv.run | bash

# Install required Python versions
pyenv install 3.8.10
pyenv install 3.9.16
pyenv install 3.10.11
pyenv install 3.11.3

# Make all versions globally available
pyenv global 3.11.3 3.10.11 3.9.16 3.8.10

Chocolatey (Windows)

# Install via Chocolatey
choco install python38 python39 python310 python311

Checking Available Interpreters

# List all environments (including missing interpreters)
tox --listenvs-all

# List only configured environments
tox --listenvs

Dependency Management

Ways to Declare Dependencies

Direct listing in deps

[testenv]
deps = 
    pytest>=6.0
    requests>=2.25.0
    numpy==1.21.0

Using requirements.txt

[testenv]
deps = 
    -r{toxinidir}/requirements.txt
    -r{toxinidir}/requirements-test.txt

Conditional dependencies

[testenv]
deps = 
    pytest
    py38: importlib-metadata  # only for Python 3.8
    py{37,38}: dataclasses    # for Python 3.7 and 3.8

Working with pyproject.toml

Since tox 4.0, you can read dependencies from pyproject.toml:

[testenv]
deps = 
extras = 
    test
    dev

Corresponding pyproject.toml example:

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my-package"
dependencies = ["requests", "click"]

[project.optional-dependencies]
test = ["pytest", "pytest-cov"]
dev = ["black", "flake8", "mypy"]

Advanced Features and Parallelism

Parallel Execution

Tox can run environments in parallel:

# Auto‑detect number of processes
tox -p auto

# Specify a fixed number of processes
tox -p 4

# Parallel run with live output
tox --parallel-live

Passing Arguments to Commands

# Forward extra arguments to pytest
tox -- --verbose --tb=short

# For a specific environment
tox -e py39 -- --maxfail=1

Using Factors

Factors let you generate combinatorial environments:

[tox]
envlist = py{38,39,310}-django{32,40,41}

[testenv]
deps = 
    django32: Django>=3.2,<3.3
    django40: Django>=4.0,<4.1
    django41: Django>=4.1,<4.2
    pytest
commands = pytest

Integration with Continuous Integration Systems

GitHub Actions

Create .github/workflows/test.yml:

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.8', '3.9', '3.10', '3.11']
    
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install tox tox-gh-actions
    
    - name: Run tests with tox
      run: tox

Corresponding tox configuration:

[tox]
envlist = py38, py39, py310, py311

[gh-actions]
python =
    3.8: py38
    3.9: py39
    3.10: py310
    3.11: py311

[testenv]
deps = pytest
commands = pytest

GitLab CI

.gitlab-ci.yml example:

image: python:3.11

stages:
  - test

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip/
    - .tox/

before_script:
  - python -V
  - pip install tox

test:
  stage: test
  script:
    - tox
  parallel:
    matrix:
      - PYTHON_VERSION: ['3.8', '3.9', '3.10', '3.11']
  image: python:$PYTHON_VERSION

Jenkins

pipeline {
    agent any
    
    stages {
        stage('Test') {
            matrix {
                axes {
                    axis {
                        name 'PYTHON_VERSION'
                        values '3.8', '3.9', '3.10', '3.11'
                    }
                }
                stages {
                    stage('Run Tests') {
                        steps {
                            sh """
                                python${PYTHON_VERSION} -m pip install tox
                                tox -e py${PYTHON_VERSION.replace('.', '')}
                            """
                        }
                    }
                }
            }
        }
    }
    
    post {
        always {
            publishHTML([
                allowMissing: false,
                alwaysLinkToLastBuild: true,
                keepAll: true,
                reportDir: 'htmlcov',
                reportFiles: 'index.html',
                reportName: 'Coverage Report'
            ])
        }
    }
}

Plugins and Extensions for tox

Popular Plugins

tox-gh-actions

Automatic integration with GitHub Actions:

pip install tox-gh-actions

[gh-actions]
python =
    3.8: py38, lint
    3.9: py39
    3.10: py310
    3.11: py311, typecheck

tox-docker

Run tests inside Docker containers:

pip install tox-docker

[testenv:integration]
docker = 
    postgres:13
    redis:6
deps = 
    pytest
    psycopg2
    redis
commands = pytest tests/integration/

tox-conda

Use Conda to manage environments:

pip install tox-conda

[testenv]
conda_deps = 
    numpy
    scipy
conda_channels = 
    conda-forge

tox-pyenv

Automatic Python discovery via pyenv:

pip install tox-pyenv

Creating Custom Plugins

Tox supports user‑written plugins through its hook system:

# tox_custom_plugin.py
import tox

@tox.hookimpl
def tox_addoption(parser):
    parser.add_argument(
        "--custom-flag",
        action="store_true",
        help="Enable custom functionality"
    )

@tox.hookimpl
def tox_configure(config):
    if config.option.custom_flag:
        # Custom logic goes here
        pass

Debugging and Diagnostics

Debug Modes

# Verbose output
tox -v

# Very verbose output
tox -vv

# Show configuration without running
tox --showconfig

# Install dependencies only (skip test commands)
tox --notest

Working with the Cache

# Force recreation of environments
tox -r

# Clear pip cache and force dependency versions
tox -r --force-dep-version

# Show environment directory locations
tox --showconfig | grep envdir

Common Issues and Solutions

#### Python Interpreter Not Found

Problem: ERROR: InterpreterNotFound: python3.9

Solution:

# Install the required Python version
pyenv install 3.9.16
pyenv global 3.11.3 3.10.11 3.9.16

# Or specify the interpreter path explicitly
[testenv:py39]
basepython = /usr/local/bin/python3.9

#### Dependency Conflicts

Problem: Incompatible package versions

Solution:

[testenv]
deps = 
    pip-tools
commands = 
    pip-compile requirements.in
    pip-sync requirements.txt
    pytest

#### Slow Dependency Installation

Problem: Packages are reinstalled on every run

Solution:

[testenv]
deps = 
    -c constraints.txt
    pytest
install_command = pip install --find-links https://my-cache {opts} {packages}

Performance Optimization

Caching

[testenv]
# Enable pip download caching
download = true

# Do not expose system site‑packages
sitepackages = false

# Reuse existing environment when possible
recreate = false

CI/CD Optimization

# GitHub Actions with caching
- name: Cache tox environments
  uses: actions/cache@v3
  with:
    path: .tox
    key: tox-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('tox.ini', 'requirements*.txt') }}
    restore-keys: |
      tox-${{ runner.os }}-${{ matrix.python-version }}-
      tox-${{ runner.os }}-

Minimizing Test Time

[testenv]
# Parallel test execution inside the environment
commands = 
    pytest -n auto --dist worksteal

# Fail fast on first error
commands = 
    pytest --maxfail=1 --tb=short

Advanced Use Cases

Testing with Different Dependency Versions

[tox]
envlist = py{38,39,310}-requests{20,25,28}

[testenv]
deps = 
    requests20: requests>=2.0,<2.1
    requests25: requests>=2.5,<2.6
    requests28: requests>=2.8,<2.9
    pytest
commands = pytest

Creating Development Environments

[testenv:dev]
description = Development environment
basepython = python3.11
usedevelop = true
deps = 
    -e .
    pytest
    black
    flake8
    mypy
    ipython
    jupyter
commands = python -c "print('Development environment ready!')"

Integration with Coverage

[testenv]
deps = 
    pytest
    pytest-cov
commands = 
    pytest --cov=src --cov-report=term-missing --cov-report=xml

[testenv:coverage-report]
description = Generate coverage report
deps = coverage[toml]
skip_install = true
commands = 
    coverage combine
    coverage report
    coverage html

Comparison with Alternative Tools

Key tox Commands and Functions

Core Configuration Sections

tox Environment Variables

Best Practices and Recommendations

Structuring tox.ini

# Recommended configuration layout
[tox]
minversion = 4.0
envlist = 
    clean,
    py{38,39,310,311},
    lint,
    typecheck,
    security,
    docs,
    report

skip_missing_interpreters = true
isolated_build = true

[testenv]
description = Run tests
deps = 
    pytest>=6.0
    pytest-cov
    pytest-xdist
setenv =
    COVERAGE_FILE = {env:COVERAGE_FILE:{envtmpdir}/.coverage.{envname}}
commands = 
    pytest {posargs:tests} --cov --cov-config=pyproject.toml --cov-report=term-missing:skip-covered --cov-report=xml:{envtmpdir}/coverage.xml --cov-report=html:{envtmpdir}/htmlcov -n auto

# Specialized environments
[testenv:clean]
description = Clean coverage files
deps = coverage[toml]
skip_install = true
commands = coverage erase

[testenv:report]
description = Generate coverage report
deps = coverage[toml]
skip_install = true
commands = 
    coverage combine
    coverage report
    coverage html

Optimizing for Large Projects

[testenv]
# Use develop mode for fast iteration
usedevelop = true

# Enable pip caching
download = true

# Reuse existing environments when possible
recreate = false

# Parallel dependency installation
install_command = python -m pip install --upgrade --upgrade-strategy eager {opts} {packages}

Security and Isolation

[testenv]
# Disallow access to system site‑packages
sitepackages = false

# Control which environment variables are passed through
passenv = 
    CI
    GITHUB_*
    HOME
    PATH
    TERM

# Set safe environment variables
setenv =
    PYTHONDONTWRITEBYTECODE = 1
    PYTHONHASHSEED = 0
    PYTHONUNBUFFERED = 1

Frequently Asked Questions

#### What if tox cannot locate a Python interpreter?

Make sure the required Python version is installed and available on your PATH. Verify with python3.9 --version. If the interpreter lives in a non‑standard location, specify its path explicitly:

[testenv:py39]
basepython = /usr/local/bin/python3.9

#### How can I speed up tox for frequent runs?

Apply these optimizations:

• Enable caching: download = true
• Avoid unnecessary recreation: recreate = false
• Use develop mode: usedevelop = true
• Run only needed environments: tox -e py39,lint

#### Can I use tox with Docker?

Yes – the tox-docker plugin provides Docker integration:

pip install tox-docker

[testenv:integration]
docker = 
    postgres:13
    redis:alpine
deps = pytest
commands = pytest tests/integration/

#### How to configure tox for a monorepo?

For a monorepo, either place a separate tox.ini in each subproject or use a single shared file:

[tox]
envlist = service1, service2, shared

[testenv:service1]
changedir = services/service1
deps = -r{toxinidir}/services/service1/requirements.txt
commands = pytest

[testenv:service2]
changedir = services/service2
deps = -r{toxinidir}/services/service2/requirements.txt
commands = pytest

#### How to integrate tox with pre‑commit?

Create a .pre-commit-config.yaml file:

repos:
  - repo: local
    hooks:
      - id: tox
        name: tox
        entry: tox
        args: ["-e", "lint,typecheck"]
        language: system
        pass_filenames: false

#### Why does tox create new environments on every run?

Possible reasons include:

• Changes in the deps list
• A different Python interpreter
• The recreate = true flag is set

Use tox -vv for detailed debugging output.

Tox is a powerful tool that greatly simplifies testing Python projects across multiple interpreter versions. Proper configuration and usage ensure high code quality, compatibility, and reliability. By following the guidance in this guide, you can fully leverage tox’s capabilities in your own projects.