Parallel Testing Made Easy With pytest-xdist

Posted on Edited on In , Word count in article: 1.6k Reading time ≈ 6 mins.

Imagine your project contains 100, if not thousands of tests. When you run pytest, the test runner executes them sequentially, taking ages to complete the tests.

What if there was a better way? A way to use resources effectively, run tests in parallel and reduce the overall test execution time and CI/CD pipeline.

This would make testing more efficient, provide faster feedback on code changes, and improve the overall development process, all while reducing your CI/CD running costs.

Pytest introduces a powerful plugin, pytest-xdist which allows you to distribute and execute tests seamlessly.

In this article, we’ll explore pytest-xdist in great detail, why to use pytest-xdist for test parallelization, and the advantages of the pytest-xdist plugin.

We’ll also demonstrate a practical example, understanding how pytest-xdist works and observe the benefits of using it to parallelize your tests.

Lastly, we’ll provide you with essential best practices when parallelizing tests with pytest-xdist. You’ll come out the other side with a solid understanding of how to leverage pytest-xdist for faster test execution.

Let’s get started.

Example Code

What You’ll Learn

By the end of this article, you’ll have

  • A good understanding of test parallelization and why it’s important.
  • How to leverage the pytest-xdist plugin to parallelize tests.
  • Essential best practices when parallelizing tests.

What is Test Parallelization?

Test parallelization runs multiple tests concurrently, reducing overall test execution time.

It distributes tests across multiple resources, such as multiple CPU cores, threads, or machines, and executes them simultaneously.

We covered Test Parallelization in great detail here so highly recommend you to check it out.

Why Parallize Tests?

Now that it’s clear what it means to run tests in parallel, why would you want to do that?

In sequential test execution, the execution time scales linearly with the number of tests.

It also requires higher computation time, making your CI/CD pipelines slower and more costly.

Most CI/CD services charge based on execution minutes. The longer the tests take, the more you pay $$$.

Besides CI/CD, slow development feedback can be frustrating. You want to know if your code changes are breaking the tests as soon as possible.

Imagine changing one variable and running a test suite that takes 30 minutes to complete.

Running selective tests is not always an option, especially when you’re working on a large project and you need to test end-to-end functionality.

That’s where test parallelization comes to the rescue, allowing you to run tests at lightning speed.

There are several other ways to improve test runtime, which we covered in detail here.

Overview of Pytest-xdist

Pytest-xdist is a popular pytest plugin that enables you to execute tests in parallel across multiple machines or CPUs.

This allows you to speed up your test suite by distributing tests across available resources, thereby reducing the overall test execution time.

With pytest-xdist, you’re able to specify the number of test processes (workers) to be used for parallel execution. This allows you to customize the level of parallelism based on your system capabilities.

Note - Due to how pytest-xdist is implemented, the -s/--capture=no option does not work. We covered stdout/stderr capturing in detail here.

Let’s look at how to use pytest-xdist for parallelizing tests.

Practical Example

Consider a simple test suite with 10 tests, each with a 1-second intentional delay.

Prerequisites

Some basics of Python and Pytest would be helpful:

  • Python (3.12+)
  • Pytest

Getting Started

Our example repo looks like this:

1
2
3
4
5
6
7
.
├── .gitignore
├── pytest.ini
├── README.md
├── requirements.txt
└── tests
    └── test_pytest_xdist.py

To get started, clone the Github Repo here, or you can create your own repo by creating a folder and running git init to initialize it.

Create a virtual environment using any package manager and install the required packages:

1
$ pip install -r requirements.txt

Test Code

Have a look at the following test code:

tests/test_pytest_xdist.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import pytest
import time

## Initializing 10 tests with 1s intentional delay

def test_with_xdist_01():
    time.sleep(1)
    assert True

def test_with_xdist_02():
    time.sleep(1)
    assert True

def test_with_xdist_03():
    time.sleep(1)
    assert True

def test_with_xdist_04():
    time.sleep(1)
    assert True

def test_with_xdist_05():
    time.sleep(1)
    assert True

def test_with_xdist_06():
    time.sleep(1)
    assert True

def test_with_xdist_07():
    time.sleep(1)
    assert True

def test_with_xdist_08():
    time.sleep(1)
    assert True

def test_with_xdist_09():
    time.sleep(1)
    assert True

def test_with_xdist_10():
    time.sleep(1)
    assert True

Our test code contains 10 simple tests with with 1s intentional delay.

Running the Test

Let’s run the test with Pytest-xdist:

1
$ pytest -v

You’ll have the following output:

pytest-xdist-example

Note that it took 10.11s to complete the test execution.

Now try with Pytest-xdist (with 5 workers):

1
$ pytest -v -n 5

This time you’ll have the following output:

pytest-xdist-example

Have a look at the execution time. 3.91s! less then half the time when running tests without pytest-xdist.

If you don’t want to specify the number of workers, you can let pytest-xdist manage that using the keyword auto.

1
$ pytest -v -n auto

You’ll have the following output:

pytest-xdist-example

Choosing the number of workers may be tricky. You can start with a small number of workers and gradually increase it based on your hardware resources and test suite characteristics.

Or if you want to let pytest-xdist automatically determine the number of workers based on the available resources, you can use the -n auto flag.

Customizing Pytest-xdist

pytest-xdist provides several options to customize the parallel test execution process.

Here are some of the most commonly used options:

Automatic Process Allocation:

  • Use pytest -n auto to automatically utilize as many processes as there are physical CPU cores on your machine.
  • Alternatively, use pytest -n logical to use logical CPU cores instead. This requires the psutil package and defaults to physical cores if the logical count is unavailable.

Explicit Process Count:

  • You can directly specify the number of processes with a number, e.g., pytest -n 8.

Custom Configuration:

  • Set the PYTEST_XDIST_AUTO_NUM_WORKERS environment variable or implement the pytest_xdist_auto_num_workers hook in conftest.py to define custom behaviors for -n auto and -n logical.

Further Parallelization Control:

  • Limit maximum workers with --maxprocesses and set restart limits with --max-worker-restart.

Test Distribution Strategies:

  • --dist load: Default mode, distributes tests to any available worker.
  • --dist loadscope: Groups tests by module or class to ensure they run in the same process.
  • --dist loadfile: Groups tests by file.
  • --dist loadgroup: Groups tests by custom group marks to ensure they run in the same worker.
  • --dist worksteal: Reassigns tests from less busy workers to idle ones to optimize fixture reuse and handle varying test durations.
  • --dist no: Normal mode, executes tests sequentially without distribution.

You can find more options and configurations in the pytest-xdist documentation.

Best Practices for Parallelizing Tests with Pytest-xdist

Let’s take a quick look at some essential best practices when parallelizing tests with pytest-xdist:

Minimize Plugin Usage: Plugins can enrich Pytest’s functionality but may slow down test execution. Only include essential plugins to maintain optimal performance. Unnecessary plugins can bloat your test setup, leading to slower execution times.

Test on Different Environments: Test your parallel execution setup on different environments (e.g., local development machines, CI/CD servers, production-like environments) to ensure compatibility and stability.

Gradual Scaling of Workers: Start with a small number of parallel workers and gradually increase it based on system’s capacity and the nature of your test suite. Monitor resource usage and test results to ensure stability.

Ensure Reproducibility: It’s good practice that your tests can be run in any order and are independent. Plugins like pytest-randomly are very helpful and can be used in conjunction with parallel testing tools like pytest-xdist.

Streamline Your Test Suite: Before implementing parallel execution, refine your test suite by reducing unnecessary setup and teardown activities, minimizing inter-test dependencies, and removing duplicates.

Isolate Test Data: When tests run in parallel, shared data can lead to unpredictable outcomes. Use isolated data sets for each test or use fixtures that ensure a fresh setup for each test to prevent conflicts and ensure that tests can run concurrently without interference.

Monitor and Optimize Resource Allocation: Keep an eye on CPU and memory usage during test runs to adjust the number of parallel workers dynamically. Overloading the system can lead to decreased performance and increased flakiness.

Utilize Efficient Ordering Strategies: Implement intelligent test ordering strategies such as grouping by test duration or categorizing by dependency. This approach can reduce the total runtime by minimizing wait times for available workers and optimizing the usage of setup and teardown operations.

Conclusion

That’s all for this one.

I hope the article was helpful and you learned something new.

pytest-xdist is a useful plugin for parallelizing tests, allowing you to distribute tests across multiple processes for quicker test execution.

We covered pytest-xdist in detail, why you should use it and how to use it with practical examples.

We also demonstrated how to combine pytest-randomly with pytest-xdist for faster and random test execution (which is a good practice against flakiness).

Lastly, you learned some essential best practices when parallelizing tests with pytest-xdist.

I highly encourage trying out pytest-xdist in your test suite and explore the benefits of parallel test execution especially as your suite grows.

If you have ideas for improvement or like for me to cover anything specific, please send me a message via Twitter, GitHub or Email.

Till the next time… Cheers!

Additional Readings

Example Code
Save Money On You CI/CD Pipelines Using Pytest Parallel (with Example)
pytest Basics: Test Parallelization with pytest-xdist
pytest-xdist
pytest-randomly
How To Run A Single Test In Pytest (Using CLI And Markers)
13 Proven Ways To Improve Test Runtime With Pytest