How To Avoid Hanging Tests Using Pytest Timeout (And Save Compute Resource)
Imagine waking up to find your daily Unit and Integration Tests have taken ages to run or are stuck in a loop?
This happens more often than you think and for any reason — slow networks, external dependencies, resource allocation issues.
Whether you’re developing code locally or running Unit Tests as part of a CI/CD pipeline, it’s important to keep tests lightweight and fast.
An interesting concept to take note of is timeout . This means exactly what the name says — the code times out if it takes longer than x units of time.
Remember your test suite needs to be fast, timeouts are a last resort, not an expected failure mode.
Good coding practices rarely result in a timeout. Keeping that in mind, if you still wish to use timeouts, pytest timeout is an interesting feature.
There are several reasons why you would want to timeout and using it ensures you don’t use more computing resources than necessary.
In this article, we’ll look at some of the reasons and several ways to add pytest timeout to your unit tests with an example.
- When To Use Pytest Timeout
- Calls to External Resources
- Sequential Test Runs
- Networking or Connectivity Issues
- Code Looping Error
- Database Connections
- How To Use Pytest Timeout — Example
- Set Pytest Timeout via CLI
- Set Pytest Timeout via pytest.ini
- Set Pytest Timeout via Decorator
- Set Pytest Timeout via Global Timeout
- Short Note — Timeout Methods
- Signal Method
- Thread Method
- Conclusion
- Additional Reading
Let’s get started then?
When To Use Pytest Timeout
Perhaps you’re wondering when it is a good idea to use Pytest timeout or timeouts?
Good question. Here are a few scenarios where this is useful.
Calls to External Resources
Modern applications pretty much make use of external resources all the time.
Very rarely if never does your application work in complete isolation. Perhaps in a highly secure on-premise military or defence application.
External resources may include APIs (vendor, internal), databases or the consumption of events from an event broker (Kafka topic) etc.
This means that your application is reliant on a resource that’s outside your control.
If the dependent server is slow or unavailable for whatever reason, it’s important that your test handles timeout otherwise your code could attempt to connect infinitely, thus costing you $$.
A good practice (alongside timeout) is NOT to use real external resources in your Unit Tests, but rather mock them.
We cover more on this in the articles on Python Rest API Unit Testing and Pytest Monkeypatching and Mocking.
Sequential Test Runs
While it’s good practice to run unit tests in parallel and keep them stateless there may be an occasional need to run them sequentially.
Some examples may include setting the authentication/authorization session cookie before you test a specific feature.
This may be followed by another test and overall the process could get long. Having a timeout is particularly useful.
Although we must always strive for stateless, event-driven architectures as much as possible with minimum coupling.
Networking or Connectivity Issues
Perhaps an external database’s DNS domain name or endpoint has changed.
Or likely somebody in your platform team redeployed a cluster to a new VPC and set the networking or security groups incorrectly and your application cannot connect to a resource.
In theory, you would have been notified of the change, but it’s not an ideal world and these things happen.
Pytest timeouts are useful in this case where your Unit test may be stuck in a loop.
Code Looping Error
As good programmers, we strive to write resilient and high-quality code.
But more often than not, bugs surface and must be addressed.
When testing edge cases, your code may go into an unpredictable loop.
In order to validate these edge cases it’s good to set a timeout at the test level.
This ensures that resources never run for longer than necessary and any execution over this timeframe would gracefully exit.
Database Connections
While most database API libraries and packages handle the opening/closing of connections, some legacy databases may not directly support this.
Ensuring database connections are terminated and threads closed once the transaction is complete is an important job.
This ensures the transaction has completed and good connectivity speed for future executions.
Pytest timeout offers you another level of protection.
How To Use Pytest Timeout — Example
I’m sure the above has given you some insight into why to use pytest timeouts.
Now let’s look at how to do this using a simple example. Here’s a link to the repo to follow along.
The example is a simple one, that calculates the factorial of a number.
If you’re unfamiliar with factorial, here’s an easy introduction.
The calculate_factorial() function in core.py executes the simple task of calculating the factorial.
core.py1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20def calculate_factorial(num: int, delay: int) -> int:
"""
Function to calculate the Factorial of a number
:param num: Number to calculate the factorial, int
:param delay: Delay to introduce in seconds, int
:return: Factorial Value, int
"""
logging.info(f"Input Value: {num}")
factorial = 1
time.sleep(delay) # Change to test delays
if num < 0:
raise ValueError("Sorry, cannot calculate "
"factorial for negative numbers")
elif num == 0:
return 1
else:
for i in range(1, num + 1):
factorial = factorial * i
logging.info(f"The factorial of {num} is {factorial}")
return factorial
The Unit Tests are defined under /tests/unit/test_factorial.py .
There are 2 tests — a success test and a ValueError test if the input is negative (-1 in this case).
test_factorial.py1
2
3
4
5
6
7
8def test_calculate_factorial_valid():
factorial = calculate_factorial(num=5, delay=10)
assert factorial == 120
def test_calculate_factorial_invalid():
with pytest.raises(ValueError):
factorial = calculate_factorial(num=-1, delay=10)
To simulate long-running tests, we’ve deliberately included a delay parameter that allows us to control the delay via a sleep function call.
Now let’s run the Unit Test without setting any timeout.
Note - In order to install the pytest timeout plugin please ensure to run1
2pip install pytest-timeout
pip install -r requirements.txt
Set Pytest Timeout via CLI
Assuming you’ve installed the plugin, you can easily set the timeout using the CLI command --timeout=X where X is the desired timeout in seconds.
E.g.1
pytest tests/unit/test_factorial.py --timeout=5
This will timeout if the tests take longer than 5 seconds. Let’s simulate this by setting a longer delay in the Unit Tests.
Set Pytest Timeout via pytest.ini
Another simple and seamless option way is to set the pytest timeout within the pytest.ini file.
This way you can avoid having to include it in all your CLI commands. Downside it that it gets globally applied to all your tests.
E.g.1
2
3
4[pytest]
timeout = 5
log_cli=true
log_level=INFO
Set Pytest Timeout via Decorator
A simple and more controlled way is to set the pytest timeout is via the @pytest.mark.timeout()decorator.
You can apply this at the top of each unit test which allows a greater level of control.
In the below screenshot you can see how we’ve configured one test to timeout in 5 seconds and the other in 1 second.
Set Pytest Timeout via Global Timeout
Another simple and less controlled way is to set a global timeout at the top of your test file.
This can be done by1
pytestmark = pytest.mark.timeout(3)
Short Note — Timeout Methods
OK, we’ve seen what is pytest timeout and how to use it.
But wouldn’t it be great to have a high-level understanding of how it works?
Pytest timeout uses 2 methods to decide how to timeout — Signal and Thread.
Signal Method
According to the official docs
This method schedules an alarm when the test item starts and cancels the alarm when the test finishes. If the alarm expires during the test the signal handler will dump the stack of any other threads running to stderr and use pytest.fail() to interrupt the test.
An important bit to remember is that the Signal method allows the tests to complete and a timeout of 1 test doesn’t affect the running of other tests.
Thread Method
According to the official docs
For each test item the pytest-timeout plugin starts a timer thread which will terminate the whole process after the specified timeout. When a test item finishes this timer thread is cancelled and the test run continues.
The downsides of this method are that there is a relatively large overhead for running each test and that test runs are not completed. The benefit of this method is that the pytest process is not terminated and the test run can complete normally.
The thread method terminates the whole process (all subsequent tests too) in the case of a timeout.
Pytest always strives to use the best timeout method based on current system requirements, so it’s not something recommend changing unless you really need to.
Conclusion
Although short, I hope this article has been useful in your understanding of pytest timeout in unit tests.
We looked at several reasons why you’d want to use pytest timeout to your unit tests including ways to solve concerns without a timeout.
We explored a real example on how to add pytest timeout — decorators, via pytest.ini , CLI and global timeouts.
You’re now better equipped to decide whether to use pytest timeout in your unit tests.
If you have ideas for improvement or like me to cover any topics please comment below or send me a message via Twitter, GitHub or Email.
Till the next time… Cheers!