Hypothesis - generative testing

Introduction

Typical tests verify that a function works for given values. But what if you don’t know in advance which inputs will cause a failure? Hypothesis – generative testing solves this problem automatically.

Hypothesis is a powerful property‑based testing library for Python that fundamentally changes the testing approach. Instead of manually crafting test data, it automatically generates thousands of diverse inputs, aiming to discover edge‑case, erroneous, or non‑standard scenarios. Rather than writing many tests with concrete values, you describe the properties and invariants of a function, and Hypothesis attempts to break them.

What Is Hypothesis

Hypothesis is a generative testing tool based on the concept of property‑based testing. The library was created to address the core limitation of traditional testing: the impossibility of anticipating every possible input.

Core Principles

The library operates on a “find a counter‑example” principle. When you define a property of a function (e.g., the result of adding two numbers is always greater than each addend), Hypothesis generates a wide range of inputs and checks whether the property holds for each set.

Advantages Over Traditional Testing

• Automatic edge‑case coverage: Hypothesis discovers rare and complex data combinations on its own.
• Error minimisation: When a failure is found, the test is automatically shrunk to the smallest failing example.
• High performance: Generation of thousands of test cases happens quickly.
• Reproducibility: Discovered bugs can be easily reproduced.

Installation and First Steps

Installing the Library

pip install hypothesis

For extra functionality you can install with extensions:

pip install hypothesis[numpy,pandas,django]

Basic Usage Example

from hypothesis import given
import hypothesis.strategies as st

@given(st.integers(), st.integers())
def test_addition_commutative(a, b):
    assert a + b == b + a

@given(st.integers(), st.integers())
def test_addition_associative(a, b, c):
    assert (a + b) + c == a + (b + c)

Running Tests

Tests can be executed with pytest, unittest, or directly:

pytest test_file.py
python -m unittest test_file.py
python test_file.py

Fundamentals of Data Generation

The @given Decorator

The @given() decorator is the centerpiece of Hypothesis. It specifies which generation strategies to use for creating test data.

@given(st.integers())
def test_absolute_value(x):
    assert abs(x) >= 0

@given(st.text(), st.integers(min_value=0))
def test_string_multiplication(s, n):
    result = s * n
    assert len(result) == len(s) * n

The hypothesis.strategies Module

All data‑generation strategies live in the hypothesis.strategies module, typically imported as st. This module provides a rich set of built‑in strategies for many data types.

Data Generation Strategies

Basic Strategies

Hypothesis offers an extensive collection of built‑in strategies for generating various data types:

# Numeric types
st.integers()   # Integers
st.floats()     # Floating‑point numbers
st.decimals()   # Decimal numbers

# Textual data
st.text()       # Unicode strings
st.binary()     # Byte strings
st.characters() # Single characters

# Boolean and temporal data
st.booleans()   # True/False
st.dates()      # datetime.date objects
st.datetimes()  # datetime.datetime objects
st.times()      # datetime.time objects

Collections and Structured Data

# Lists and tuples
st.lists(st.integers())               # Lists of integers
st.tuples(st.text(), st.integers())    # Tuples of (string, integer)

# Dictionaries and sets
st.dictionaries(st.text(), st.integers())  # Dictionaries
st.sets(st.integers())                     # Sets

# Fixed values
st.just(42)                     # Always returns 42
st.sampled_from(['a', 'b', 'c'])# Random element from the list

Specialised Strategies

# Internet‑related data
st.emails()          # Valid email addresses
st.urls()            # URL strings
st.ip_addresses()    # IP addresses
st.uuids()           # UUID identifiers

# Scientific data (requires NumPy)
st.arrays(dtype=np.int32, shape=(3, 3))  # NumPy arrays

Key Methods and Functions Overview

Property and Invariant Checking

What Is Property‑Based Testing

Property‑based testing focuses on verifying general properties and invariants of functions rather than specific input‑output pairs. This approach can uncover bugs that traditional example‑driven tests miss.

Property‑Checking Examples

@given(st.text())
def test_string_properties(s):
    # Property: length does not change after .upper()
    assert len(s) == len(s.upper())
    # Property: applying .upper() twice yields the same result
    assert s.upper().upper() == s.upper()

@given(st.lists(st.integers()))
def test_list_properties(lst):
    # Property: length stays the same after sorting
    sorted_lst = sorted(lst)
    assert len(lst) == len(sorted_lst)
    # Property: all elements are preserved
    assert sorted(lst) == sorted_lst

@given(st.integers(), st.integers())
def test_arithmetic_properties(a, b):
    # Commutativity of addition
    assert a + b == b + a
    # Multiplication by one is identity
    assert a * 1 == a

Failure Minimisation (Shrinking)

How Shrinking Works

When Hypothesis finds a failing test, it automatically runs a shrinking process to locate the smallest possible example that still triggers the failure.

Shrinking Example

@given(st.integers())
def test_fails_on_zero(x):
    assert x != 0

Running this test, Hypothesis reports:

Falsifying example: test_fails_on_zero(x=0)

Shrinking Process

1. Failure detection: Hypothesis finds a value that causes a crash.
2. Simplification: The algorithm attempts to replace the value with simpler ones.
3. Verification: Each simplified value is checked to ensure the failure still occurs.
4. Result: The minimal failing example is presented.

Advanced Strategies

Constraining Values

# Numbers within a range
@given(st.integers(min_value=1, max_value=100))
def test_positive_numbers(x):
    assert x > 0

# Strings with size limits and alphabet constraints
@given(st.text(min_size=1, max_size=10,
              alphabet=st.characters(whitelist_categories=('Lu', 'Ll'))))
def test_alphabetic_strings(s):
    assert s.isalpha()

# Lists with size limits
@given(st.lists(st.integers(), min_size=1, max_size=5))
def test_small_lists(lst):
    assert 1 <= len(lst) <= 5

Combining Strategies

# one_of to choose between strategies
@given(st.one_of(st.integers(), st.text()))
def test_mixed_types(value):
    assert isinstance(value, (int, str))

# Building complex structures
user_strategy = st.dictionaries(
    keys=st.sampled_from(['name', 'age', 'email']),
    values=st.one_of(st.text(),
                     st.integers(min_value=0, max_value=120),
                     st.emails())
)

@given(user_strategy)
def test_user_data(user):
    assert isinstance(user, dict)

Custom Strategies

Creating Composite Strategies

@st.composite
def user_data(draw):
    name = draw(st.text(min_size=1, max_size=50))
    age = draw(st.integers(min_value=0, max_value=120))
    email = draw(st.emails())
    return {
        "name": name,
        "age": age,
        "email": email,
        "is_adult": age >= 18
    }

@given(user_data())
def test_user_validation(user):
    assert "name" in user
    assert user["is_adult"] == (user["age"] >= 18)

Using builds to Construct Objects

from dataclasses import dataclass

@dataclass
class Person:
    name: str
    age: int
    email: str

person_strategy = st.builds(
    Person,
    name=st.text(min_size=1),
    age=st.integers(min_value=0, max_value=120),
    email=st.emails()
)

@given(person_strategy)
def test_person_creation(person):
    assert isinstance(person, Person)
    assert person.age >= 0

Filtering and Assumptions

Using filter()

@given(st.integers().filter(lambda x: x > 0))
def test_positive_only(x):
    assert x > 0

@given(st.text().filter(lambda s: len(s) > 0))
def test_non_empty_strings(s):
    assert len(s) > 0

Using assume()

from hypothesis import assume

@given(st.integers(), st.integers())
def test_division(a, b):
    assume(b != 0)
    result = a / b
    assert result * b == a  # May fail due to float precision

@given(st.lists(st.integers()))
def test_list_operations(lst):
    assume(len(lst) > 0)
    first_element = lst[0]
    assert first_element in lst

filter() vs. assume()

• filter(): Applied at the strategy level; filtered values are never generated.
• assume(): Applied during test execution; unsuitable examples are simply skipped.

Parametrisation and Multiple Scenarios

Using @example()

@given(st.integers())
@example(0)          # Always test zero
@example(-1)         # Always test -1
@example(1000000)    # Always test a large number
def test_with_specific_examples(x):
    assert abs(x) >= 0

Combining with pytest.mark.parametrize

import pytest

@pytest.mark.parametrize("base", [2, 8, 10, 16])
@given(st.integers(min_value=0, max_value=1000))
def test_number_conversion(base, number):
    # Convert number to different bases
    converted = format(number, f'0{base}')
    assert int(converted, base) == number

Test Execution Settings

The @settings() Decorator

from hypothesis import settings

@settings(max_examples=1000, deadline=None)
@given(st.integers())
def test_with_more_examples(x):
    # Runs 1000 times instead of the default 100
    assert isinstance(x, int)

@settings(verbosity=2)
@given(st.text())
def test_with_verbose_output(s):
    assert isinstance(s, str)

Settings Profiles

from hypothesis import settings, Verbosity

# Create profiles
settings.register_profile("debug", max_examples=10, verbosity=Verbosity.verbose)
settings.register_profile("ci", max_examples=1000, deadline=None)

# Load a profile
settings.load_profile("debug")

Integration with Testing Frameworks

Integration with pytest

Hypothesis integrates seamlessly with pytest without additional configuration:

import pytest
from hypothesis import given, settings
import hypothesis.strategies as st

class TestMathOperations:
    @given(st.integers(), st.integers())
    def test_addition(self, a, b):
        assert a + b == b + a
    
    @pytest.mark.slow
    @settings(max_examples=10000)
    @given(st.floats(), st.floats())
    def test_multiplication(self, a, b):
        assert a * b == b * a

Integration with unittest

import unittest
from hypothesis import given
import hypothesis.strategies as st

class TestStringOperations(unittest.TestCase):
    @given(st.text())
    def test_string_length(self, s):
        self.assertGreaterEqual(len(s), 0)
    
    @given(st.text(), st.text())
    def test_string_concatenation(self, s1, s2):
        result = s1 + s2
        self.assertEqual(len(result), len(s1) + len(s2))
        self.assertTrue(result.startswith(s1))
        self.assertTrue(result.endswith(s2))

API and Business‑Logic Testing

Testing a REST API

@st.composite
def api_request_data(draw):
    return {
        "user_id": draw(st.integers(min_value=1)),
        "action": draw(st.sampled_from(["create", "update", "delete"])),
        "data": draw(st.dictionaries(
            keys=st.text(min_size=1, max_size=20),
            values=st.one_of(st.text(), st.integers(), st.booleans())
        ))
    }

@given(api_request_data())
def test_api_request_validation(request_data):
    # Validate API input payload
    assert "user_id" in request_data
    assert request_data["user_id"] > 0
    assert request_data["action"] in ["create", "update", "delete"]

Testing Business Logic

@st.composite
def order_data(draw):
    items = draw(st.lists(
        st.dictionaries(
            keys=st.just("price"),
            values=st.floats(min_value=0.01, max_value=1000.0)
        ),
        min_size=1,
        max_size=10
    ))
    discount = draw(st.floats(min_value=0.0, max_value=0.5))
    return {"items": items, "discount": discount}

@given(order_data())
def test_order_total_calculation(order):
    # Verify total order calculation
    total_before_discount = sum(item["price"] for item in order["items"])
    expected_total = total_before_discount * (1 - order["discount"])
    calculated_total = calculate_order_total(order)
    assert abs(calculated_total - expected_total) < 0.01

Generating Custom Types and Models

Working with dataclasses

from dataclasses import dataclass
from typing import List, Optional

@dataclass
class Product:
    name: str
    price: float
    category: str
    in_stock: bool

@dataclass
class Customer:
    name: str
    email: str
    age: int
    orders: List[Product]

product_strategy = st.builds(
    Product,
    name=st.text(min_size=1, max_size=50),
    price=st.floats(min_value=0.01, max_value=10000),
    category=st.sampled_from(["electronics", "clothing", "books", "food"]),
    in_stock=st.booleans()
)

customer_strategy = st.builds(
    Customer,
    name=st.text(min_size=1, max_size=50),
    email=st.emails(),
    age=st.integers(min_value=18, max_value=100),
    orders=st.lists(product_strategy, max_size=10)
)

@given(customer_strategy)
def test_customer_validation(customer):
    assert customer.age >= 18
    assert "@" in customer.email
    assert all(isinstance(order, Product) for order in customer.orders)

Working with Pydantic models

from pydantic import BaseModel, EmailStr, validator
from typing import List

class User(BaseModel):
    name: str
    email: EmailStr
    age: int
    is_active: bool = True
    
    @validator('age')
    def validate_age(cls, v):
        if v < 0:
            raise ValueError('Age must be non‑negative')
        return v

user_strategy = st.builds(
    User,
    name=st.text(min_size=1, max_size=50),
    email=st.emails(),
    age=st.integers(min_value=0, max_value=120),
    is_active=st.booleans()
)

@given(user_strategy)
def test_user_model_validation(user):
    assert isinstance(user, User)
    assert user.age >= 0
    assert "@" in user.email

Reproducibility and Debugging

Controlling Randomness

from hypothesis import seed

@seed(12345)
@given(st.integers())
def test_with_fixed_seed(x):
    # This test will always generate the same sequence
    assert isinstance(x, int)

Debugging and Statistics

from hypothesis import note, event

@given(st.integers())
def test_with_debugging(x):
    note(f"Testing with value: {x}")
    event(f"Sign: {'positive' if x > 0 else 'negative' if x < 0 else 'zero'}")
    
    if x > 1000:
        event("Large number")
    
    assert abs(x) >= 0

Saving Failing Examples

Hypothesis automatically stores failing examples in a .hypothesis/examples file for later reproduction. This behaviour can be customised:

@settings(database=None)  # Disable example persistence
@given(st.integers())
def test_without_database(x):
    assert x >= 0

Advanced Techniques

Using flatmap for Dynamic Strategies

@given(st.integers(min_value=0, max_value=10).flatmap(
    lambda n: st.lists(st.integers(), min_size=n, max_size=n)
))
def test_list_with_dynamic_size(lst):
    # List size is determined at runtime
    assert len(lst) <= 10

Statistical Testing

from hypothesis import target

@given(st.lists(st.integers()))
def test_sorting_performance(lst):
    import time
    start = time.time()
    sorted_lst = sorted(lst)
    end = time.time()
    
    # Hypothesis will try to maximise the execution time
    target(end - start)
    
    assert sorted_lst == sorted(lst)

Comparison with Other Tools

Best Practices and Recommendations

What to Test with Hypothesis

1. Function invariants: Properties that must always hold.
2. Inverse operations: Functions paired with their inverses.
3. Idempotency: Operations that can be applied repeatedly without change.
4. Commutativity and associativity: Mathematical properties.
5. Boundary and constraint handling: Verifying correct processing of edge cases.

Writing Effective Tests

# Good: property‑based test
@given(st.text())
def test_string_upper_property(s):
    assert s.upper().isupper() or len(s) == 0

# Bad: concrete‑value test
@given(st.text())
def test_string_specific_case(s):
    if s == "hello":
        assert s.upper() == "HELLO"

Performance Optimisation

# Prefer filter over assume for speed
@given(st.integers().filter(lambda x: x > 0))  # Faster
def test_positive_numbers(x):
    assert x > 0

@given(st.integers())  # Slower
def test_positive_numbers_bad(x):
    assume(x > 0)
    assert x > 0

Exception Handling

@given(st.integers(), st.integers())
def test_division_by_zero(a, b):
    if b == 0:
        with pytest.raises(ZeroDivisionError):
            a / b
    else:
        result = a / b
        assert isinstance(result, float)

Frequently Asked Questions

How can I speed up test execution?

Reduce the number of generated examples with a custom setting:

@settings(max_examples=50)
@given(st.complex_strategy())
def test_fast_execution(data):
    # Faster test
    pass

How do I handle slow tests?

@settings(deadline=None)  # Disable timeout
@given(st.large_data_strategy())
def test_slow_operation(data):
    # Slow operation
    pass

How can I test external dependencies?

from unittest.mock import Mock

@given(st.text())
def test_external_api_call(data):
    with Mock() as mock_api:
        mock_api.return_value = {"status": "ok"}
        result = call_external_api(data, api_client=mock_api)
        assert result["status"] == "ok"

How do I avoid excessive use of assume()?

# Bad: many assumes
@given(st.integers())
def test_with_many_assumes(x):
    assume(x > 0)
    assume(x < 1000)
    assume(x % 2 == 0)
    # Test may run slowly

# Good: use filters
@given(st.integers(min_value=1, max_value=999).filter(lambda x: x % 2 == 0))
def test_with_filters(x):
    # Faster and clearer
    pass

Conclusion

Hypothesis is a powerful tool for raising the quality of Python application testing. It excels at:

• Discovering rare bugs and edge cases
• Testing mathematical and logical properties
• Validating parsers and validators
• Testing APIs and business logic
• Ensuring reliability of critical code

Key benefits of using Hypothesis include:

• Automatic detection of complex bugs
• Shrinking of failing examples for easier debugging
• High‑speed execution of thousands of test cases
• Excellent integration with existing testing frameworks
• Reproducible results

To get the most out of Hypothesis, focus on testing invariants and properties rather than fixed input‑output pairs. This approach reveals errors that traditional testing methods cannot uncover.