Practical Overview Of The Top 5 Python Testing Frameworks

Posted on Edited on In , Word count in article: 2.6k Reading time ≈ 9 mins.

Releasing software without testing is like flying a plane without a maintenance check.

Testing is a safety net that helps make sure your code works well, uncovers bugs and keeps the code in good shape over its lifetime.

When developing with Python, you can use the built-in testing library called PyUnit (Unittest).

However, there are other popular and highly equipped Python testing frameworks such as Pytest, Nosetest, Doctest, and Robot - which arguably may even be better than the built-in one.

Now given so many options, how do you choose which Python testing framework to use in your application? How do you balance the trade-offs between ease of use, features and performance?

Well, that’s exactly what we’ll cover in this article.

We’ve done the hard work to explore the popular Python testing frameworks, with examples, and highlight their pros and cons to help you compare and decide which framework you can use in testing your applications.

Let’s get started.

Example Code

What You’ll Learn

By the end of this tutorial, you will:

  • Understand why it’s important to use Python Testing Frameworks.
  • Have a deeper understanding of the popular Python Testing Frameworks and their core components.
  • Compare the popular testing frameworks and choose the best one for your application.
  • Set up and run tests using Unittest, Pytest, Nosetest, Doctest and Robot.

But first, let’s understand some basics.

What Is A Python Testing Framework?

In Python, a testing framework is a bundle of tools and strategies designed to simplify the process of writing, organizing, and executing tests.

These frameworks define a structured way to run tests making it easier to ensure the correctness and reliability of your code.

The core components of any Python testing framework include (but are not limited to),

  • Test Discovery: The core ability for any testing framework that locates test files, classes, and functions automatically within a codebase without explicit configuration.

  • Test Fixtures: Fixtures prepare the test environment by initializing resources using the setup method and cleaning resources using the teardown method. Check out our detailed guide on setup and teardown here.

  • Assertion Methods: Testing frameworks contain a set of assertion methods that allows you to check whether certain conditions are true during the execution of a test.

  • Test Runners: The test runner executes the tests and reports the result. It manages the lifecycle of tests, including setup, execution, and teardown. Some advanced testing frameworks (e.g., Pytest) provide options for test filtering and parallel execution.

  • Parametrized Testing: Parametrized testing, allows you to run the same test logic with different input values without writing multiple tests.

  • Mocking and Stubbing: Advanced testing frameworks (e.g., Pytest) include features to isolate the code under test from external dependencies. This feature is known as Mocking and Stubbing.

  • Test Reporting: Test reporting tools generate human-readable output summarizing the results of test runs, containing information about the number of tests executed, passed, and failed, as well as details about individual test cases.

Why Use A Python Testing Framework?

In Python development, the use of testing frameworks is indispensable for fostering a robust, reliable, and maintainable software development process.

So why should you use a testing framework? Why not write each element of the test yourself?

The reasons are as below

  • Automated Testing: Testing frameworks allow you to run tests automatically. This makes your tests reusable, ensuring consistent and repeatable results. It also allows you to execute a comprehensive suite of tests quickly and efficiently.

  • Code Quality: Testing frameworks help identify errors, bugs, and unexpected behaviours early in the development process, leading to higher quality and reduced likelihood of defects.

  • Continuous Integration (CI): Testing frameworks integrate seamlessly with CI/CD systems. This allows you to automate testing in a continuous integration workflow.

  • Faster Debugging: Testing frameworks provide valuable information about test failures, accelerate the debugging process and help you fix issues quickly.

  • Regression Testing: Testing frameworks automate tests that serve as a safety net against regressions. These frameworks check whether existing functionalities and tests still work as expected after code changes, preventing the introduction of new issues on existing code.

For an in-depth explanation of the various types of testing, this guide is a solid read.

Now that we have some background, let’s look at the top used Python testing frameworks and how to use them with an example.

Project Set up

Let’s organize our project to demonstrate executing tests using different Python testing frameworks.

Prerequisites

To follow this guide, you should have:

  • Python 3.11+ installed.

Getting Started

Our example code repo looks like this. We’ll explain what each file does and why it’s used.

1
2
3
4
5
6
7
8
9
10
├── .gitignore
├── README.md
├── requirements.txt
├───src
│   └── check_number.py
└───tests
    ├── number_tests.robot
    ├── test_check_number_with_nose2.py
    ├── test_check_number_with_pytest.py
    └── test_check_number_with_unittest.py

To get started. clone the Github Repo here, or you can create your own repo by creating a folder and running git init to initialize it.

Example Code

The example code is a straightforward Python program designed to determine whether a given number is positive, negative, or zero.

src/check_number.py

1
2
3
4
5
6
7
8
9
10
11
12
13
def check_number(num: str):
    """
    Function to check if the number is negative or positive.
    """
    num = float(num)
    if num >= 0:
        if num == 0:
            return "Zero"
        else:
            return "Positive"
    else:
        return "Negative"

The example contains a single function called check_number() that assesses whether a given number is positive, negative, or zero.

Now, let’s subject this example code to testing using various Python Testing Frameworks.

In this article, we’ll explore the following Python Testing Frameworks:

  • Pytest
  • Unittest (also called PyUnit)
  • Nose2
  • DocTest
  • Robot

Pytest

Pytest is one of the most popular testing frameworks that is well known for its simplicity, flexibility, and powerful features.

It promotes clean and readable code with a concise syntax, reduced boilerplate and more expressive tests. Not to mention the incredible community support, plugins and thorough documentation.

Key Features
Pytest possesses several advanced features that position it at the pinnacle of this list. And I don’t say that because this website is based on Pytest 😁.

  • Fixtures: Pytest offers a fixture system that allows you to organize your test environment automatically. It automates setup tasks, such as sharing data, managing database connections, initializing variables, and ensuring the proper clean-up after test completion. Learn more about Pytest Fixtures and how to use them here.
  • Markers: In Pytest, tests can be marked with custom markers, allowing selective test execution based on markers and running specific subsets of tests. This comprehensive guide talks about Pytest Markers such as skip, xfail, timeout and so on.
  • Parametrization: In Pytest, parametrization is available at both the test level and the fixture level. This allows you to run your test for a series of input values without repeating the test code. Read more about how to parametrize your tests here.
  • Simple Assert Statements: Pytest contains easy-to-use Assert statement syntax, allowing you to verify the expected behavior of your code with ease.
  • Hooks and Plugins: Pytest offers powerful customization and extensibility through hooks and plugins, allowing you to tailor the testing framework to your specific needs.
    Hooks are the set of functions with well-defined names that Pytest automatically calls at specific points during the testing process. Plugins are external Python modules and packages that extend Pytest’s functionality.
  • Parallel Test Execution: Pytest also supports parallel execution, reducing overall test suite execution time.

The above list of not exhaustive and Pytest also contains a whole host of other features.

Limitations

  • Installation is required.

Set up

Pytest is not included in the Python Standard Library. So you need to install it using pip, conda or another package manager of your choice.

1
pip install pytest

After successful installation. You can check the Pytest version:

1
pytest --version

Test Example

Now let’s run our example code with Pytest.

tests/test_check_number_with_pytest.py

1
2
3
4
5
6
7
from src.check_number import check_number
import pytest

def test_check_number():
    assert check_number("-22") == "Negative"
    assert check_number("0") == "Zero"
    assert check_number("10") == "Positive"

Here we’ve one test function test_check_number() that checks the check_number() function.

It’s time to run the test,

1
pytest -v

Output:

1
2
3
4
5
6
7
8
9
$ pytest -v
================================================= test session starts =================================================
platform win32 -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- C:\Users\Md. Aminul Islam\AppData\Local\Programs\Python\Python311\python.exe
cachedir: .pytest_cache
rootdir: G:\Pytest-with-Eric\python-testing-frameworks-example
configfile: pytest.ini
tests/test_check_number_with_pytest.py::test_check_number PASSED                                                 [100%]

================================================== 1 passed in 0.23s ==================================================

You can generate more detailed results using Pytest verbosity level (e.g., -v, -vv, and -vvv).

PyUnit or Unittest

PyUnit, commonly recognized as Unittest, is the default Python testing framework. It comes built-in with the Python Standard Library, offering clean and swift test execution and rapid generation of test reports, including XML and Unittest SML reports.

Let’s swiftly review the advantages and limitations that PyUnit presents:

Key Features

  • Built-in with Python Standard Library.
  • Automatic test discovery.
  • Provides a wide range of built-in assertion methods for validating test conditions.
  • Supports the use of setup and teardown methods (Fixtures).
  • Supports reusing test suits and test organization.
  • Flexible and easy test case execution.

Limitations

  • Unnecessary use of Boilerplate code.
  • Limited Fixture Scope.
  • Less Flexibility in Test Discovery.
  • Runs tests sequentially by default
  • Needs to write tests in classes

Set Up

Unittest is built with the Python Standard library. So no installation is required.

Test Example

Here is a test code with Unittest,

tests/test_check_number_with_unittest.py

1
2
3
4
5
6
7
8
9
10
11
12
13
import unittest
from src.check_number import check_number

class TestSum(unittest.TestCase):

    def test_check_number(self):
        self.assertEqual(check_number("-22"), "Negative")
        self.assertEqual(check_number("0"), "Zero")
        self.assertEqual(check_number("10"), "Positive")


if __name__ == '__main__':
    unittest.main()

Here we’ve one test function test_check_number().

The following command will run the test code,

1
python -m unittest

Output:

1
2
3
4
5
6
$ python -m unittest
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Nose2

Nose2 can be likened to an enhanced version of PyUnit (Unittest). While it shares similarities with Unittest, Nose2 distinguishes itself by offering additional supportive plugins for test execution, test discovery, decorators, fixtures, parameterization, and more.

Key Features

  • Plugin-based architecture, allows users to extend and customize the framework according to their specific testing needs.
  • Able to run both DocTests and UnitTests.
  • Supports advanced testing packages like fixtures, modules, classes, and more.
  • Supports parallel testing.
  • Automatic test collection.
  • Built-in support for measuring code coverage.

Limitations

  • Not actively maintained compared to other frameworks.
  • Lack of documentation.

Set Up

Execute the following command to install Nose2,

1
pip install nose2

Now check the Nose2 version

1
pip show nose2

Test Example

You can write test code with Nose2 as follows,

tests/test_check_number_with_nose2.py

1
2
3
4
5
6
7
8
9
10
11
12
13
import unittest
from src.check_number import check_number

class TestSum(unittest.TestCase):

    def test_check_number(self):
        self.assertEqual(check_number("-22"), "Negative")
        self.assertEqual(check_number("0"), "Zero")
        self.assertEqual(check_number("10"), "Positive")
        
if __name__ == '__main__':
    import nose2
    nose2.main()

You may notice that the test code is very similar to the test code we’ve written with Unittest. Here we’ve one test function test_check_number().

Let’s execute the test:

1
nose2 -v

Output:

1
2
3
4
5
6
7
$ nose2 -v
test_check_number (tests.test_check_number_with_nose2.TestSum.test_check_number) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

DocTest

DocTest is a module in the Python Standard Library that allows you to write tests directly within the DocString of your source code. This approach makes it effortless to generate tests based on the output of the standard Python interpreter shell.

DocTest is particularly well-suited for beginners looking to initiate their journey into Python testing.

Key Features

  • Easy to start.
  • No installation required.
  • Extensive code documentation.
  • Tests can be written directly in the code.

Limitations

  • Compares only Printed output.
  • Doesn’t support parameterized testing.
  • Advanced testing features like Test Discovery, Fixtures, etc not supported.

Set Up

DocTest is built with the Python Standard library. So no installation is required.

Test Example

With DocTest, we can include the tests inside the comment:

src/check_number.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def check_number(num: str):
    """
    Function to check if the number is negative or positive.
    >>> check_number(-2)
    'Negative'
    >>> check_number(0)
    'Zero'
    """
    num = float(num)
    if num >= 0:
        if num == 0:
            return "Zero"
        else:
            return "Positive"
    else:
        return "Negative"

if __name__ == "__main__":
    import doctest
    doctest.testmod()

As you can see we directly integrated the tests inside the DocString. Here the testmod() will discover the Docstring examples in the check_number() function, execute them, and print a summary of the results.

Running DocTest is similar to running a Python file,

1
python check_number.py -v

Output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ python check_number.py -v
Trying:
    check_number(-2)
Expecting:
    'Negative'
ok
Trying:
    check_number(0)
Expecting:
    'Zero'
ok
1 items had no tests:
    __main__
1 items passed all tests:
   2 tests in __main__.check_number
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

Robot

Robot framework is primarily employed for acceptance testing. Its keyword-driven approach simplifies automation, making it easier to create readable test cases.

The framework generates test reports in both XML and HTML formats, enhancing readability. It comes equipped with a variety of generic tools and test libraries.

Key Features

  • Keyword-driven testing that makes automation simpler, helping testers to create readable test cases easily.
  • Supports all types of applications (e.g., Web, Mobile).
  • Easy-to-understand report data.
  • Contains rich libraries and a toolset.
  • Easy to integrate with third-party tools.

Limitations:

  • Doesn’t support parallel testing.
  • Complex maintenance.
  • Lacks support for if-else, nested loops.
  • Requires a bit longer learning curve for beginners.

Set Up

The following command will install Robot in your system,

1
pip install robotframework

Run this command to check the version,

1
robot --version

Test Example

In Robot, we can write tests like the below,

tests/number_tests.robot

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
*** Settings ***
Library     ../src/check_number.py

*** Test Cases ***
Test Negative Number
    ${result}=    check_number    -2
    Should Be Equal As Strings    ${result}    Negative

Test Zero
    ${result}=    check_number    0
    Should Be Equal As Strings    ${result}    Zero

Test Positive Number
    ${result}=    check_number    3
    Should Be Equal As Strings    ${result}    Positive

Test scripts in Robot are written few separate parts such as Settings and Test Case.

In the example above we have three test cases Test Negative Number, Test Zero, and Test Positive Number. The Settings section defines the source file for tests.

Type the following command to run tests,

1
robot number_tests.robot

Output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ robot number_tests.robot
==============================================================================
Number Tests
==============================================================================
Test Negative Number                                                  | PASS |
------------------------------------------------------------------------------
Test Zero                                                             | PASS |
------------------------------------------------------------------------------
Test Positive Number                                                  | PASS |
------------------------------------------------------------------------------
Number Tests                                                          | PASS |
3 tests, 3 passed, 0 failed
==============================================================================
Output:  G:\Pytest-with-Eric\python-testing-frameworks-example\tests\output.xml
Log:     G:\Pytest-with-Eric\python-testing-frameworks-example\tests\log.html
Report:  G:\Pytest-with-Eric\python-testing-frameworks-example\tests\report.html

Final Thoughts and Recommendations

That concludes our article.

In this one, we exlored a brief overview of the popular Python testing frameworks, highlighting their benefits and limitations with examples.

Additionally, we covered the installation process of these frameworks and executed simple tests, offering insights into their operational procedures for a better understanding.

Choosing the most suitable Python testing framework for your project is contingent upon your specific requirements and expertise.

With an active community and fantastic documentation, Pytest stands out as the leader among all Python testing frameworks. Its advanced tools, functionalities and simplicity make it a perfect choice for any kind of project.

If you’re strongly considering Unittest, check out this article which does a detailed comparison between Pytest and Unittest.

I’ve used Unittest, Nose, Doctest or Robot for smaller projects but Pytest stands as my de-facto choice for real applications.

If you have ideas for improvement or would like me to cover anything specific, please send me a message via Twitter, GitHub or Email.

Till the next time… Cheers and Happy Testing! 🐍🚀

Additional Reading

Example Code
How Pytest Fixtures Can Help You Write More Readable And Efficient Tests
Ultimate Guide To Pytest Markers And Good Test Management
How to Effortlessly Generate Unit Test Cases with Pytest Parameterized Tests
How To Debug Failing Tests Like A Pro (Use Pytest Verbosity Options)
Pytest
Unittest
Nose2
DocTest
Robot framework
Top 6 BEST Python Testing Frameworks
Top 8 Python Testing Frameworks