How to Use Hypothesis and Pytest for Robust Property-Based Testing in Python

Posted on Edited on In , Word count in article: 2.7k Reading time ≈ 10 mins.

When writing unit tests, it’s hard to consider all possible edge cases and validate that your code works correctly.

This is sometimes caught in production and a quick and speedy patch needs to be deployed. Only for a new bug to emerge later.

There will always be cases you didn’t consider, making this an ongoing maintenance job. Unit testing solves only some of these issues.

Property-based testing is a complementary approach to traditional unit testing, where test cases are generated based on properties or constriants that the code should satisfy.

Hypothesis addresses this limitation by automatically generating test data based on specified strategies.

This allows developers to test a much broader range of inputs and outputs than traditional unit tests, increasing the likelihood of catching edge cases and unexpected behaviour.

In this article, we’ll explore how to use Hypothesis with Pytest for property-based testing.

We’ll start with the difference between example-based testing and property-based testing and then dive into 2 examples of how to use Hypothesis with Pytest.

First we’ll go through a simple example (string/array transformations) and then move on to a more complex one — building a Shopping Cart app.

Finally, we’ll discuss best practices for using Hypothesis with Pytest, including tips for writing your own Hypothesis strategies and touch on Model-based testing.

By the end of this article, you’ll be equipped with the knowledge and tools to use Hypothesis and Pytest for efficient and comprehensive property-based testing of your code.

Let’s get started then?

Link To GitHub Repo

Objectives

By the end of this tutorial you should be able to:

  • Understand the key differences between example-based, property-based and model-based testing
  • Use the Hypothesis library with Pytest to test your code and ensure coverage for a wide range of test data
  • Apply property-based testing to your Python apps
  • Build a Shopping App and test it using property-based testing

Example-Based Testing vs Property-Based Testing

Example-based testing and property-based testing are two common approaches to software testing along with model-based testing.

Example-based testing involves writing test cases that provide specific inputs and expected outputs for functions or methods.

These tests are easy to write and understand, and they can catch many common errors.

However, they are limited in scope and may not cover all possible edge cases or unexpected scenarios.

Property-based testing involves testing properties or invariants that code should satisfy, and then automatically generating test data to check if those properties hold true.

This approach can catch a much broader range of edge cases and unexpected behaviour that may not be covered by example-based testing.

However, it can be more challenging to write and understand these tests, and it may require more time and computation power to generate test data.

As a developer, you should strive to use a combination of both to capture as many edge cases as possible to produce robust well-tested code.

Project Set Up

The project has the following structure

pytest-hypothesis-repo

Getting Started

To get started, clone the repo here, or you can create your own repo by creating a folder and running git init to initialise it.

Prerequisites

In this project, we’ll be using Python 3.10.10.

Create a virtual environment and install the requirements (packages) using

1
pip install -r requirements.txt

This will install the Hypothesis Library that we’ll be using throughout this tutorial.

Simple Example

To understand how to use this library, it’s always good to start with a simple example. Walk before trying to run.

Source Code

The source code for our simple application contains a bunch of Array (list) and String transformations in Python.

Array Operations

src/random_operations.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# Array Operations  
 
def find_largest_smallest_item(input_array: list) -> tuple:  
    """  
    Function to find the largest and smallest items in an array  
    :param input_array: Input array  
    :return: Tuple of largest and smallest items  
    """  
  
    if len(input_array) == 0:  
        raise ValueError  
    # Set the initial values of largest and smallest to the first item in the array  
    largest = input_array[0]  
    smallest = input_array[0]  
  
    # Iterate through the array  
    for i in range(1, len(input_array)):  
        # If the current item is larger than the current value of largest, update largest  
        if input_array[i] > largest:  
            largest = input_array[i]  
        # If the current item is smaller than the current value of smallest, update smallest  
        if input_array[i] < smallest:  
            smallest = input_array[i]  
  
    return largest, smallest  
  
  
def sort_array(input_array: list, sort_key: str) -> list:  
    """  
    Function to sort an array  
    :param sort_key: Sort key  
    :param input_array: Input array  
    :return: Sorted array  
    """  
    if len(input_array) == 0:  
        raise ValueError  
    if sort_key not in input_array[0]:  
        raise KeyError  
    if not isinstance(input_array[0][sort_key], int):  
        raise TypeError  
    sorted_data = sorted(input_array, key=lambda x: x[sort_key], reverse=True)  
    return sorted_data

String Operations

src/random_operations.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# String Operations  
  
def reverse_string(input_string) -> str:  
    """  
    Function to reverse a string  
    :param input_string: Input string  
    :return: Reversed string  
    """  
    return input_string[::-1]  
  
  
def complex_string_operation(input_string: str) -> str:  
    """  
    Function to perform a complex string operation  
    :param input_string: Input string  
    :return: Transformed string  
    """  
    # Remove Whitespace  
    input_string = input_string.strip().replace(" ", "")  
  
    # Convert to Upper Case  
    input_string = input_string.upper()  
  
    # Remove vowels  
    vowels = ("A", "E", "I", "O", "U")  
    for x in input_string.upper():  
        if x in vowels:  
            input_string = input_string.replace(x, "")  
  
    return input_string

Simple Example — Unit Tests

As you’ve seen in the above section, two popular types of testing are

  1. Example-based testing (tests input-output examples)
  2. Property-based testing (tests properties with various auto-generated input data)

Example-Based Testing

We write some simple example-based tests

tests/unit/test_random_operations.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import pytest  
import logging  
from src.random_operations import (  
    reverse_string,  
    find_largest_smallest_item,  
    complex_string_operation,  
    sort_array,  
)  
  
logging.basicConfig(level=logging.INFO)  
logger = logging.getLogger(__name__)  
 
  
# Example Based Unit Testing  
def test_find_largest_smallest_item():  
    assert find_largest_smallest_item([1, 2, 3]) == (3, 1)  
  
  
def test_reverse_string():  
    assert reverse_string("hello") == "olleh"  
  
  
def test_sort_array():  
    data = [  
        {"name": "Alice", "age": 25},  
        {"name": "Bob", "age": 30},  
        {"name": "Charlie", "age": 20},  
        {"name": "David", "age": 35},  
    ]  
    assert sort_array(data, "age") == [  
        {"name": "David", "age": 35},  
        {"name": "Bob", "age": 30},  
        {"name": "Alice", "age": 25},  
        {"name": "Charlie", "age": 20},  
    ]  
  
  
def test_complex_string_operation():  
    assert complex_string_operation("  Hello World  ") == "HLLWRLD"

Here we test our 4 methods using the traditional way. You give an input and you assert that input against an expected output.

Running The Unit Test

To run the unit tests, simply run

1
pytest ./tests/unit/test_random_operations.py -v -s

pytest-hypothesis-1

Great. But what if there are edge cases that you didn’t consider?

Let’s look at property-based testing.

Property-Based Testing

tests/unit/test_random_operations.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from hypothesis import given, strategies as st  
from hypothesis import assume as hypothesis_assume  
  
# Property Based Unit Testing  
@given(st.lists(st.integers(), min_size=1, max_size=25))  
def test_find_largest_smallest_item_hypothesis(input_list):  
    assert find_largest_smallest_item(input_list) == (max(input_list), min(input_list))  
  
  
@given( st.lists( st.fixed_dictionaries({"name": st.text(), "age": st.integers()}), ))  
def test_sort_array_hypothesis(input_list):  
    if len(input_list) == 0:  
        with pytest.raises(ValueError):  
            sort_array(input_list, "age")  
  
    hypothesis_assume(len(input_list) > 0)  
    assert sort_array(input_list, "age") == sorted(  
        input_list, key=lambda x: x["age"], reverse=True  
    )  
  
  
@given(st.text())  
def test_reverse_string_hypothesis(input_string):  
    assert reverse_string(input_string) == input_string[::-1]  
  
  
@given(st.text())  
def test_complex_string_operation_hypothesis(input_string):  
    assert complex_string_operation(input_string) == input_string.strip().replace(  
        " ", ""  
    ).upper().replace("A", "").replace("E", "").replace("I", "").replace(  
        "O", ""  
    ).replace(  
        "U", ""  
    )

To use Hypothesis in this example, we import the given , strategies and assume in-built methods.

The @given decorator is placed just before each test followed by a strategy.

A strategy is specified using the strategy.X method which can be st.list() , st.integers() , st.text() and so on.

Here’s a comprehensive list of strategies.

Strategies are used to generate test data and can be heavily customised to your liking for example — generate only positive/negative numbers, X/Y key, value pairs, integer with a maximum value of 100 and so on.

In our test examples above we generate simple test data using relevant strategies and assert the actual property logic holds true rather than the data itself.

Common issues discovered (that aren’t accounted for) include

- Empty lists, strings, and dicts as input
- Repeated values in lists
- Negative numbers, 0 as input
- Keys not present in dict
- ASCII characters

These can be discovered easily using the Hypothesis library.

Running The Unit Test

To run the unit tests, simply run

1
pytest ./tests/unit/test_random_operations.py -v -s

pytest-hypothesis-2

Complex Example

Now that you’ve seen a basic example, it’s time to step up your testing game.

Let’s build a simple Shopping App that lets us

  • Add items to the cart
  • Remove items from the cart
  • Calculate the total based on the quantity
  • View cart items
  • Clear cart

Source code

src/shopping_cart.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
import random  
from enum import Enum, auto  
  
  
class Item(Enum):  
    """Item type"""  
  
    APPLE = auto()  
    ORANGE = auto()  
    BANANA = auto()  
    CHOCOLATE = auto()  
    CANDY = auto()  
    GUM = auto()  
    COFFEE = auto()  
    TEA = auto()  
    SODA = auto()  
    WATER = auto()  
  
    def __str__(self):  
        return self.name.upper()  
  
  
class ShoppingCart:  
    def __init__(self):  
        """  
        Creates a shopping cart object with an empty dictionary of items  
        """  
        self.items = {}  
  
    def add_item(self, item: Item, price: int | float, quantity: int = 1) -> None:  
        """  
        Adds an item to the shopping cart  
        :param quantity: Quantity of the item  
        :param item: Item to add  
        :param price: Price of the item  
        :return: None  
        """  
        if item.name in self.items:  
            self.items[item.name]["quantity"] += quantity  
        else:  
            self.items[item.name] = {"price": price, "quantity": quantity}  
  
    def remove_item(self, item: Item, quantity: int = 1) -> None:  
        """  
        Removes an item from the shopping cart  
        :param quantity: Quantity of the item  
        :param item: Item to remove  
        :return: None  
        """  
        if item.name in self.items:  
            if self.items[item.name]["quantity"] <= quantity:  
                del self.items[item.name]  
            else:  
                self.items[item.name]["quantity"] -= quantity  
  
    def get_total_price(self):  
        total = 0  
        for item in self.items.values():  
            total += item["price"] * item["quantity"]  
        return total  
  
    def view_cart(self) -> None:  
        """  
        Prints the contents of the shopping cart  
        :return: None  
        """  
        print("Shopping Cart:")  
        for item, price in self.items.items():  
            print("- {}: ${}".format(item, price))  
  
    def clear_cart(self) -> None:  
        """  
        Clears the shopping cart  
        :return: None  
        """  
        self.items = {}

We define an Item enum that only accepts a specific custom Item object.

Complex Example — Unit Tests

Before diving head first into Property-based testing with Hypothesis, let’s get a feel for the code with some simple Example-based testing.

Example-Based Testing

tests/unit/test_shopping_cart.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import pytest  
from src.shopping_cart import ShoppingCart, Item  
  
  
@pytest.fixture()  
def cart():  
    return ShoppingCart()  
  
  
# Define Items  
apple = Item.APPLE  
orange = Item.ORANGE  
gum = Item.GUM  
soda = Item.SODA  
water = Item.WATER  
coffee = Item.COFFEE  
tea = Item.TEA  
  
  
# Example Based Testing  
def test_add_item(cart):  
    cart.add_item(apple, 1.00)  
    cart.add_item(orange, 1.00)  
    cart.add_item(gum, 2.00)  
    cart.add_item(soda, 2.50)  
    assert cart.items == {  
        "APPLE": {"price": 1.0, "quantity": 1},  
        "ORANGE": {"price": 1.0, "quantity": 1},  
        "GUM": {"price": 2.0, "quantity": 1},  
        "SODA": {"price": 2.5, "quantity": 1},  
    }  
  
  
def test_remove_item(cart):  
    cart.add_item(orange, 1.00)  
    cart.add_item(tea, 3.00)  
    cart.add_item(coffee, 3.00)  
    cart.remove_item(orange)  
    assert cart.items == {  
        "TEA": {"price": 3.0, "quantity": 1},  
        "COFFEE": {"price": 3.0, "quantity": 1},  
    }  
  
  
def test_total(cart):  
    cart.add_item(orange, 1.00)  
    cart.add_item(apple, 2.00)  
    cart.add_item(soda, 2.00)  
    cart.add_item(soda, 2.00)  
    cart.add_item(water, 1.00)  
    cart.remove_item(apple)  
    cart.add_item(gum, 2.50)  
    assert cart.get_total_price() == 8.50  
  
  
def test_clear_cart(cart):  
    cart.add_item(apple, 1.00)  
    cart.add_item(soda, 2.00)  
    cart.add_item(water, 1.00)  
    cart.clear_cart()  
    assert cart.items == {}

The above example illustrates simple exam-based testing where we test

  • Add items
  • Add/Remove items
  • Check the total
  • Check that the clear cart functionality works

If you notice, we’ve used a Fixture to initialise the Cart object with default function scope.

If you’re unfamiliar with fixtures, this article on Pytest fixtures offers a solid base.

This ensures that the Cart object is reset after each test, ensuring statelessness and test isolation.

Running The Unit Test

To run the unit tests, simply run

1
pytest ./tests/unit/test_shopping_cart.py -v -s

pytest-hypothesis-3

Property-Based Testing

Now let’s look at the big boy. How to test our complex Shopping App with property-based testing.

We want to test properties like

  • Assert the number of items in the cart is equal to the number of items added
  • Assert that if we remove an item, the quantity of items in the cart is equal to the number of items added — 1
  • Ensure the total calculation is correct.

And many more.

To keep this simple we’ll just test the above basic use cases, you can of course go as granular as you like to cover all possible edge cases.

tests/unit/test_shopping_cart_hypothesis.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
from typing import Callable  
from hypothesis import given, strategies as st  
from hypothesis.strategies import SearchStrategy  
from src.shopping_cart import ShoppingCart, Item  
  
  
# Create a strategy for items  
@st.composite  
def items_strategy(draw: Callable[[SearchStrategy[Item]], Item]):  
    return draw(st.sampled_from(list(Item)))  
  
  
# Create a strategy for price  
@st.composite  
def price_strategy(draw: Callable[[SearchStrategy[float]], float]):  
    return round(draw(st.floats(min_value=0.01, max_value=100, allow_nan=False)), 2)  
  
  
# Create a strategy for quantity  
@st.composite  
def qty_strategy(draw: Callable[[SearchStrategy[int]], int]):  
    return draw(st.integers(min_value=1, max_value=10))  
  
  
@given(items_strategy(), price_strategy(), qty_strategy())  
def test_add_item_hypothesis(item, price, quantity):  
    cart = ShoppingCart()  
  
    # Add items to cart  
    cart.add_item(item=item, price=price, quantity=quantity)  
  
    # Assert that the quantity of items in the cart is equal to the number of items added  
    assert item.name in cart.items  
    assert cart.items[item.name]["quantity"] == quantity  
  
  
@given(items_strategy(), price_strategy(), qty_strategy())  
def test_remove_item_hypothesis(item, price, quantity):  
    cart = ShoppingCart()  
  
    print("Adding Items")  
    # Add items to cart  
    cart.add_item(item=item, price=price, quantity=quantity)  
    cart.add_item(item=item, price=price, quantity=quantity)  
    print(cart.items)  
  
    # Remove item from cart  
    print(f"Removing Item {item}")  
    quantity_before = cart.items[item.name]["quantity"]  
    cart.remove_item(item=item)  
    quantity_after = cart.items[item.name]["quantity"]  
  
    # Assert that if we remove an item, the quantity of items in the cart is equal to the number of items added - 1   
    assert quantity_before == quantity_after + 1  
  
  
@given(items_strategy(), price_strategy(), qty_strategy())  
def test_calculate_total_hypothesis(item, price, quantity):  
    cart = ShoppingCart()  
  
    # Add items to cart  
    cart.add_item(item=item, price=price, quantity=quantity)  
    cart.add_item(item=item, price=price, quantity=quantity)  
  
    # Remove item from cart  
    cart.remove_item(item=item)  
  
    # Calculate total  
    total = cart.get_total_price()  
    assert total == cart.items[item.name]["price"] * cart.items[item.name]["quantity"]

Let’s break down the above tests.

Define strategies

To keep things simple to understand, we first define 3 strategies — 

  • Item strategy (randomly choose a value from our Item enum)
  • Price strategy (randomly choose a float number from 0.01 to 100)
  • Quantity strategy (randomly choose an integer number from 1 to 10)

We can then use these strategies in our tests with the @given decorator.

Running The Unit Test

To run the unit tests, simply run

1
pytest ./tests/unit/test_shopping_cart_hypothesis.py -v -s

pytest-hypothesis-4

Discover Bugs With Hypothesis

Often you will find your tests failing for input values you didn’t think about.

For example — when writing up these tests I caught bugs like

  • The Same Item Added With Different Pricing which we need to account for.

In real life, this would not happen as each product will have its own unique SKU (stock-keeping unit). Nevertheless, it’s something to think about and fix.

Define Your Own Hypothesis Strategies

As we mentioned above you can create your own custom Hypothesis strategies and go as granular as you like.

In our examples, we defined custom strategies below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Create a strategy for items  
@st.composite  
def items_strategy(draw: Callable[[SearchStrategy[Item]], Item]):  
    return draw(st.sampled_from(list(Item)))  
  
# Create a strategy for price  
@st.composite  
def price_strategy(draw: Callable[[SearchStrategy[float]], float]):  
    return round(draw(st.floats(min_value=0.01, max_value=100, allow_nan=False)), 2)  
  
# Create a strategy for quantity  
@st.composite  
def qty_strategy(draw: Callable[[SearchStrategy[int]], int]):  
    return draw(st.integers(min_value=1, max_value=10))

We use the st.composite wrapper and draw a value from a pre-determined strategy or our custom Enums (as in the case of Items).

Model-Based Testing in Hypothesis

Model-based testing is an approach where a formal model (often smaller scale) of the system is created, and test cases are generated from that model.

The Hypothesis library can be customized to generate test data that satisfies the properties and constraints of the model, allowing you to efficiently and effectively test your systems.

We’ll cover Model-Based testing with examples in a future article.

Conclusion

I hope you’ve enjoyed this article in learning how to use the awesome Hypothesis testing library.

In this article, we looked at example-based testing and how some of its shortcomings can be overcome with property-based testing.

We looked at two examples — a simple array/string transformation example and a Shopping Cart app.

For each example, we drew up example-based tests and followed with property-based testing to increase the amount and variation of test data.

As a good developer, you must always strive to test your application using as many realistic use cases as possible.

But we’re human after all and there’s always something you haven’t considered. For all of that, there’s Hypothesis.

If you have ideas for improvement or like for me to cover anything specific, please send me a message via Twitter, GitHub or Email.

Till the next time… Cheers!

Additional Reading

Welcome to Hypothesis! - Hypothesis 6.75.2 documentation