3 Simple Ways To Ignore Test Directories in Pytest

Posted on Edited on In , Word count in article: 2.6k Reading time ≈ 10 mins.

Have you ever had tests fail due to irrelevant or outdated code?

Perhaps adding to the confusion of debugging and troubleshooting along with unnecessary test execution time.

Or maybe you have code in development that’s not ready to be tested yet.

In such cases, you might want to exclude certain tests from your test suite.

But how do you do it in Pytest? You can of course run pytest in the relevant directories but that’s not very efficient.

The good news? Ignoring test directories is entirely possible and very easy!

With Pytest, skipping directories and tests becomes effortless through CLI options, markers, and config files.

This not only ensures smooth test execution but keeps your test suite lean and accelerates your CI/CD pipeline.

In this article, you’ll learn how to use the Pytest ignore directory feature to efficiently exclude unnecessary tests from your test suite via different methods.

Let’s get started!

As usual, we have some example code for you.

Example Code

What You’ll Learn

This article will teach you,

  • How to ignore test directories in Pytest using various options like command line flags and config files
  • Ignore individual tests with Markers.

Understanding Test Collection in Pytest

Before exploring the heart of the matter, let’s grasp how Pytest goes about collecting tests.

Pytest initiates a recursive search for files and directories that match the configured pattern, commencing from the project’s root directory and then all subsequent directories.

Specifically:

  1. It identifies directories starting with the prefix “test” or “tests”. You can define the test directory via the testpaths variable in configuration files such as pytest.ini. If you need a refresher on pytest.ini, check out this guide.
  2. The search extends to files with names matching test patterns (e.g., test_*.py or *_test.py).
  3. Pytest recognizes functions or classes with designated names, such as test_* for functions or Test* for classes containing test methods.
  4. Fixtures, essential for preparing the test environment by initializing variables, resources and performing setup teardown, can be defined in conftest.py using the @pytest.fixture decorator.

These conventions serve as the default patterns for locating test directories.

Now, if your project’s directories and names deviate from these conventions, you might encounter the “Collected 0 items” error.

For more insights, refer to the comprehensive guide on resolving the “Collected 0 items” error.

You can also change the default Pytest test discovery using custom patterns like

1
2
3
4
[pytest]
python_files = check_*.py
python_classes = Check
python_functions = *_check

Why Ignore Directories in Pytest?

So why should you ignore directories in Pytest?

Why not let the test framework just do it’s thing.

Well, think about a large projects that typically contain 1000+ tests (e.g., Unit, Integration, Functionality, Regression, Smoke, E2E), and you asked to analyze the results of some specific test.

Or perhaps you’re working on a small feature and don’t need to run the entire test suite.

When you run pytest, Pytest will perform collection of all tests. This includes tests that are still in development, tests for upcoming modules, or tests that are irrelevant to your current task.

This makes your test suite bloated and runs slow.

You can run test based on Pytest markers but doing this repeatedly for each marker is kind of a pain.

Besides it, selective test execution via Command-line is also challenging.

This is where the Pytest ignore directories feature comes in, allowing you to cleanly exclude unnecessary tests from execution.

Faster Test Execution: By eliminating unnecessary directories, you reduce the number of tests that Pytest needs to discover and run. This can significantly speed up the overall test set up and execution time. Also, make the CI/CD process faster and saves you $$$.

Focused Testing: Ignoring directories allows you to concentrate on testing the core and relevant parts of your project, making more focused test execution.

Cleaner Test Output: Ignoring directories helps maintain a clean and organized test result output. You can avoid cluttering your test reports with irrelevant information.

Simplified Maintenance: Ignoring specific directories makes it easier to maintain and update tests. You can isolate testing efforts to particular areas of your project without interference from unrelated files.

Incorporating this feature not only optimizes your testing strategy but also contributes to a more efficient and accelerated development process.

Having understood the benefits of ignoring directories, let’s explore how to implement this feature in Pytest using a simple example.

Practical Example

Let’s start by preparing the environment.

Prerequisites

To follow this guide, you should have basic working knowledge of:

  • Python (I’ve used 3.12 but any recent version should work)
  • Pytest

Getting Started

Our example repo looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
.  
├── .gitignore
├── README.md
├── pytest.ini
├── requirements.txt
├── tests
│   ├── __init__.py
│   ├── in_progress
│   │   └── test_in_progress_modules.py
│   ├── upcoming_modules
│   │   └── test_upcoming_modules.py
│   └── test_converter.py
└── src
    └── converter.py

To get started. clone the Github Repo here, or you can create your own repo by creating a folder and running git init to initialize it.

Create a virtual environment and install the required packages using the following command:

1
pip install -r requirements.txt

Example Code

Our sample code is a simple metric convertor.

src/converter.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
def centimeter_to_meter(c: float) -> float:
    """
    Function to convert Centimeter to Meter
    """
    return c / 100


def meter_to_centimeter(m: float) -> float:
    """
    Function to convert Meter to Centimeter
    """
    return m * 100

def foot_to_inch(f: float) -> float:
    """
    Function to convert Foot to Inch
    """
    return f * 12

def inch_to_foot(i: float) -> float:
    """
    Function to convert Inch to Foot
    """
    return i / 12

def meter_to_millimeter(i: float) -> float:
    """
    Function to convert Meter to Millimeter
    UNDER DEVELOPMENT
    """
    #return i / 12

Here are four methods for converting meters, centimeters, feet, and inches. And one function under development.

Test Code

Let’s write some tests!

tests/test_converter.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from src.converter import (
    centimeter_to_meter,
    meter_to_centimeter,
    foot_to_inch,
    inch_to_foot,
)
import pytest

def test_centimeter_to_meter():
    assert centimeter_to_meter(100) == 1


def test_meter_to_centimeter():
    assert meter_to_centimeter(1) == 100


def test_foot_to_inch():
    assert foot_to_inch(1) == 12


def test_inch_to_foot():
    assert inch_to_foot(12) == 1


@pytest.mark.skip(reason="Function not implemented yet")
def test_millimeter_to_centimeter():
    assert millimeter_to_centimeter(10) == 1


@pytest.mark.xfail(reason="Function expected to fail")
def test_centimeter_to_millimeter():
    assert centimeter_to_millimeter(10) == 1

Notice that we used the @pytest.mark.skip marker on the test_millimeter_to_centimeter() function since the targeted function is still not completed.

Additionally, we marked the test_centimeter_to_millimeter() with @pytest.mark.xfail as we have not yet developed the centimeter_to_millimeter() function and expecting it to fail, or perhaps there’s a bug that needs resolving.

You can read more about Skip and Xfail in our Ultimate Guide to Using Pytest Skip Test and XFail.

To simulate test exclusion let’s make a folder in_progress to contain some tests that are still in development and another folder upcoming_modules to contain tests for functions that we plan to develop soon.

tests/in_progress/test_in_progress_modules.py

1
2
3
4
# This directory contains tests for in-prograss modules.
# The development isn't done yet. So we can skip this directory.
def test_meter_to_millimeter():
    pass

And,

tests/upcoming_modules/test_upcoming_modules.py

1
2
3
4
5
6
7
# Upcoming function that has not been developed yet
def test_kilometer_to_meter():
    pass

# Upcoming function that has not been developed yet
def test_meter_to_kilometer():
    pass

3 Ways to Ignore Directory/Tests in Pytest

Using Command-Line Options

The easiest method to skip directories with tests you don’t need is by using Pytest’s --ignore command-line option. This option lets you exclude specific directories during your tests.

For our example code, you can craft the CLI command like this:

1
pytest --ignore=tests/in_progress --ignore=tests/upcoming_modules

To ignore certain files in a directory, you can follow this CLI command:

1
pytest --ignore=tests/in_progress/test_in_progress_modules.py

Here, we ignoring only the test_in_progress_modules.py file from the directory tests/in_progress.

This approach is recommended when you only want to ignore a particular directory temporarily. But evidently, it’s not the best choice for permanent exclusion and can be cumbersome to remember every time.

Using Configuration Files

When dealing with several directories to exclude, opting for a configuration file is a more convenient choice rather than crafting an extensive command-line instruction.

Utilizing Pytest’s default configuration file, pytest.ini, allow you to effortlessly specify the directories to ignore, as illustrated below:

pytest.ini

1
2
3
4
5
6
7
8
[pytest]
testpaths =
    tests
    integration
addopts = -v 

# Specifying the directory to ignore
norecursedirs =  tests/in_progress tests/upcoming_modules

Here we used the norecursedirs keyword to ignore directories. But you can do the same thing using addopts keyword.

1
addopts = --ignore=tests/in_progress --ignore=tests/upcoming_modules

To ignore certain files in a directory, you can use:

1
addopts = --ignore=tests/in_progress/test_in_progress_modules.py

Here, we ignoring just the test_in_progress_modules.py file from the directory tests/in_progress.

This proves beneficial when you’re permanently excluding directories from your tests.

Besides pytest.ini you can also use other configuration files such as setup.cfg, tox.ini, and pyproject.toml to define Pytest config.

The following guide shows how you can define Pytest config using a variety of files.

If you’re unfamiliar with the addopts keyword, check out this guide which teaches you how to pass command-line arguments to Pytest.

Using Markers (For Individual Tests)

Markers serve as a handy tool for categorizing and personalizing your tests.

Acting as annotations, markers can be attached to functions, methods, or classes, offering a way to include extra details about your tests.

To handle individual tests, you can apply markers like @pytest.mark.skip or @pytest.mark.xfail. The @pytest.mark.skip decorator indicates skipping the test, while @pytest.mark.xfail flags a test as expected to fail.

In our example code:

tests/test_converter.py

1
2
3
4
5
6
7
8
9
10
import pytest

@pytest.mark.skip(reason="Function not implemented yet")
def test_millimeter_to_centimeter():
    assert millimeter_to_centimeter(10) == 1


@pytest.mark.xfail(reason="Function expected to fail")
def test_centimeter_to_millimeter():
    assert centimeter_to_millimeter(10) == 1

This allows you to control the behavior of specific tests based on your requirements.

When constantly ignoring certain tests or directories, I prefer to use configuration files. This approach avoids the need to type lengthy command-line instructions every time.

However, using markers is the optimal choice when omitting individual tests.

You can also apply custom markers like @pytest.mark.smoke or @pytest.mark.regression or @pytest.mark.slow to categorize your tests as explained in this article on Pytest Markers.

Running the test

You can run the test using the following command:

1
pytest

Alternatively, you can ignore directories via the CLI command:

1
pytest -v --ignore=tests/in_progress --ignore=tests/upcoming_modules

Both methods yield the same output:

pytest-ignore-directory-example-result

You’ll observe 1 skipped test and 1 xfail test, corresponding to the tests mentioned earlier in our test code.

Also, the directories tests/in_progress and tests/upcoming_modules are ignored since we instructed Pytest to do so using the CLI or configuration files.

Ignoring Files for Code Coverage

A Pytest Coverage Report provides a percentage measure of validated code.

To generate coverage reports, you can use the pytest-cov or coverage plugin which generates reports in the terminal, visually appealing HTML files or even a nice badge on your GitHub repos.

It’s also common for many companies to mandate a coverage report check during a Pull Request (PR) or CI.

You can argue that coverage reports are a vanity metric for code quality but it’s a good way to ensure that you have a good test coverage on the code you have written.

If you need to exclude specific directories or files from the coverage report, open the .coveragerc file. Use the omit keyword to specify the directories or files you want to ignore. Here’s an example:

1
2
3
4
5
6
[run]
omit =
    # Ignore everything in /in_progress
    /in_progress/*
    # Omit this single file
    tests/simple_tests.py

In the provided configurations, we excluded the entire directory /in_progress/* along with the test file tests/simple_tests.py.

Note that these paths are relative to the project root directory.

This flexibility enables you to customize the coverage report to the particular needs of your project.

This comprehensive tutorial on Pytest code coverage teaches you how to implement code coverage.

Common Issues When Ignoring Directories

When it comes to ignoring directories, you might encounter the following challenges:

  • Missing Important Tests:
    Ignoring directories without inspecting the tests within them poses the risk of excluding critical tests and releasing potential “buggy code” to production.

  • Test Fragmentation:
    The act of ignoring directories may cause breakdowns or gaps in the testing process. Striking a balance between excluding unnecessary tests for efficiency and ensuring comprehensive test coverage is crucial.

  • Maintaining Configuration Consistency:
    In a collaborative CI/CD environment, it’s paramount to uphold consistency in the configuration of ignored directories across different development environments.

  • Changes in Directory Structure:
    Ignored directories may pose challenges when frequent changes occur in the project structure. In such situations, adjustments to the test configuration may become necessary to adapt to the evolving project layout.

  • Complex Configuration Management:
    As the number of ignored directories grows, managing the configuration file and command-line arguments may become more intricate. It’s essential to navigate this complexity effectively to maintain a streamlined testing process.

Best Practices for Ignoring Directories

Let’s unwrap some tips and tricks to follow while ignoring directories,

Leverage Configuration Files:
When it comes to permanently excluding directories, opt for configuration files. This ensures automatic exclusion of unnecessary directories every time the test runs, as opposed to CLI commands which are easy to forget or get wrong when doing everytime.

  • Document Ignored Directories:
    Clearly document why certain directories are ignored. Provide a detailed explanation for excluding specific directories from tests, offering valuable context for the team.

  • Maintain CI/CD Pipeline Consistency:
    Ensure consistency in your CI/CD pipelines by employing a standard configuration for directories that are marked as ignored. This approach improves reliability and reproducibility in testing.

  • Consider Test Dependencies:
    Be mindful of dependencies between tests in different directories. Ignoring a directory might have repercussions on tests in other directories. Ideally, tests should be unique and separated. If you need to share fixtures you can create a conftest.py in that local directory and share it.

  • Regular Review and Updates:
    As the project evolves, certain directories may regain relevance for testing. Regularly review and update the list of ignored directories to keep it aligned with the project’s changing needs.

By embracing these best practices, you can harness the power of Pytest’s ignore directory feature effectively, ensuring the maintenance of a robust and comprehensive test suite.

Wrapping Up

That wraps up our exploration of Pytest’s ignore directory feature.

This article provided a quick overview of ignoring directories in Pytest and highlighted its benefits in your development process.

Through a practical example, You walked through the practical aspects of working with Pytest’s ignore directory feature via different methods - CLI, config files, and markers.

You delved into the nuances of excluding tests from code coverage, touched upon common issues associated with ignoring directories, and shared some valuable best practices for a more efficient directory exclusion process.

The article offers three methods to ignore directories during test execution. Opting for the CLI command is ideal if you wish to exclude a directory for a single test run.

For those looking to permanently exclude directories from tests, leveraging config files is the way to go. Additionally, markers provide a handy tool to skip individual tests with ease.

Feel empowered by these insights, and use Pytest’s ignore directory option to boost your test execution!

If you have any ideas for improvement or like me to cover any topics please comment below or send me a message via Twitter, GitHub or Email.

Till the next time… Cheers!

Additional Reading

Example Code Used
How To Generate Beautiful & Comprehensive Pytest Code Coverage Reports (With Example)
A Simple Guide to Fixing The ‘Pytest Collected 0 Items’” Error
Pytest Config Files - A Practical Guide To Good Config Management
An Ultimate Guide to Using Pytest Skip Test and XFail
What Is pytest.ini And How To Save Time Using Pytest Config
What is Setup and Teardown in Pytest? (Importance of a Clean Test Environment)
Ultimate Guide To Pytest Markers And Good Test Management
How To Run A Single Test In Pytest (Using CLI And Markers)
How To Use Pytest With Command Line Options (Easy To Follow Guide)
Ignore paths during test collection
Pytest options – how to skip or run specific tests
Pytest-Cov Configuration
Specifying source files