How To Create Interactive Test Reports with Pytest and Allure

As an SDET or QA Engineer, how do you share test results with your wider team?

Pytest-HTML and terminal reporting are fine for highly technical people.

But what about less technical colleagues like project managers or delivery leads?

How can you communicate with them effectively and bridge the gap between test automation, developers, and project managers?

That’s exactly why the Allure Report was developed.

Allure Reports makes automated test reporting easy and pretty.

In this article, you’ll learn how to use Allure Report with Pytest to craft visually appealing and informative test reports.

We’ll begin with a quick start guide and then move on to customizing and enhancing reports to improve readability.

We’ll also break down what different components in the report mean.

Then we’ll explore simple ways to automatically generate and share Allure Reports with the team via a CI Pipeline (GitHub Actions example).

In the end, you’ll feel more confident in using Allure Reports as part of your PR approval/feature release process.

Let’s begin.

What is Test Reporting?

Test reporting involves documenting the outcomes and details of test executions.

This helps you and your team understand which aspects of the application work as expected and which don’t.

Why is this important?

Test reporting helps provide insights into the quality and reliability of your overall codebase.

It offers a transparent, detailed view of test results, making identifying and addressing issues promptly easier.

Additionally, they are crucial for stakeholders to assess the project’s progress and for developers to prioritize bug fixes and improvements, ensuring the delivery of a robust product.

With that in mind, let’s look at Allure Reports.

What are Allure Reports?

Allure Report is a popular framework for generating test execution reports, renowned for their visually appealing and highly informative output.

Developed by Qameta Software, Allure Reports began as an internal tool to improve the visibility and analysis of test results.

They soon gained popularity in the testing community for their comprehensive features.

Unlike basic text-based reports, Allure provides a rich interface that includes graphs, timelines, images, and detailed steps.

They make navigating and understanding test outcomes easier, helping to identify issues and assess the health of your tests.

Allure Reports have quickly become an indispensable tool for modern CI environments.

Quick Start Guide

Let’s look at how to get started with Allure so you can be on your way.

Project Setup

The example code used can be found in this repo.

I’ve switched to using Poetry to manage virtual environments, if you need guidance on how to set up and use Poetry, please check out this article.

First, clone the repo and install the dependencies.

1

$ poetry install

Allure works with all popular frameworks as documented here.

In this article, we’ll learn how to use it with Pytest, but I have one for pytest-bdd in the books.

Install allure-pytest

If you’re starting with a blank pyproject.toml file then the next step is to install allureand the allure-pytest plugin.

1
2

$ poetry add allure  
$ poetry add allure-pytest

If you’ve already installed it in the previous step, then ignore the above command.

Example Code

The example code used today is a simple text processor.

text_processor/text_processor.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

def count_words(text: str) -> int | bool:  
    """Return the number of words in the given text."""  
    if not text:  
        return False
    return len(text.split())  
  
  
def reverse_string(text):  
    """Return the reversed version of the given text."""  
    return text[::-1]  
  
  
def is_palindrome(text):  
    """Check if the given text is a palindrome (ignores case and spaces)."""  
    # Check for empty string  
    if not text:  
        return False  
    cleaned_text = "".join(char.lower() for char in text if char.isalnum())  
    return cleaned_text == cleaned_text[::-1]  
  
  
def capitalize_text(text):  
    """Return the capitalized version of the given text."""  
    if not text:  
        return False  
    return text.capitalize()

The corresponding tests,

tests/test_text_processor.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

import pytest  
from text_processor.text_processor import (  
    count_words,  
    reverse_string,  
    is_palindrome,  
    capitalize_text,  
)  
  
  
def test_count_words():  
    assert count_words("hello world") == 2  
    assert count_words("") == 0  
    assert count_words("one") == 1  
  
  
def test_reverse_string():  
    assert reverse_string("abc") == "cba"  
    assert reverse_string("hello") == "olleh"  
    assert reverse_string("") == ""  
  
  
def test_is_palindrome():  
    assert is_palindrome("A man a plan a canal Panama") is True  
    assert is_palindrome("Hello") is False  
    assert is_palindrome("RaC eCar") is True  
    assert is_palindrome("") is False  
  
  
@pytest.mark.parametrize( "text, expected",  
    [  
        ("hello", "Hello"),  
        ("HELLO", "Hello"),  
        ("123", "123"),  
        ("", False),  
        (None, False),  
    ],)  
def test_capitalize_text(text, expected):  
    assert capitalize_text(text) == expected

Run Tests

Let’s run our tests to make sure it works before generating any reports.

OK perfect.

Generate Allure Report

Now that your tests are working, let’s generate a simple Allure report.

You can use either of the below commands

1
2

$ python -m pytest --alluredir allure-results   
$ pytest --alluredir allure-results

If you prefer to run it via Poetry

1

$ poetry run python -m pytest --alluredir allure-results

You can change the location of the allure-results .

Once you’ve run the tests and generated the reports, you can serve them.

1

$ allure serve allure-results

If the system doesn’t detect the allure terminal command you can install it using the instructions here — e.g. brew install allure for MacOS X (Windows commands in the link).

You can also run it with Poetry

1

$ poetry run allure serve allure-results

This will spin up a local webserver in your default browser with the report.

You can scroll, click, and play around with the report interactively.

This is how you can get started easily with Pytest and Allure Reports.

There is much more you can do to enhance the readability of these reports and collect more data.

That’s exactly what you’ll learn in the rest of this article

Report Breakdown (Step by Step)

Let’s break down the report and learn what each component does. Here’s a Demo Report for you to follow along.

Overview

The dashboard offers a quick summary of the test suite’s execution.

It displays key metrics like the number of passed, failed, skipped, and broken tests.

This high-level view includes pie charts and bar graphs for an instant visual understanding of the test outcomes.

Graphs

Graphs provide several graphical representations such as the severity and duration of tests, trends over time, and categories of failures.

These graphs are crucial for spotting patterns and making data-driven decisions about where to focus testing efforts.

Timeline

The timeline displays the execution sequence of the tests, showing their start, duration, and end times.

It’s particularly useful for identifying bottlenecks and parallel execution issues.

Suites

Here, tests are grouped by their respective suites, providing detailed results for each test including status, steps, and attachments like logs or screenshots.

This detailed view helps in drilling down to the specific cause of failures.

Categories

Failed tests are categorized (e.g., product defects, test defects), allowing teams to quickly address different types of issues systematically.

Behaviors

This section shows you a higher-level view including tests grouped by issues, pull-requests, epics, stories, features, and so on.

To get the most out of your Allure report it’s important you add the correct and up-to-date metadata so your reports are not out of date.

Enhancing Test Reports

Now let’s learn how to add metadata to your reports and make them tell a story.

Using Allure with Pytest enables you to:

• Supply detailed descriptions, links, and additional metadata for better context,
• Structure your tests into organized hierarchies,
• Break down tests into smaller, more manageable steps for clarity,
• Document the parameters for parametrized tests,
• Assign clear, descriptive titles to fixtures,
• Capture and save screenshots and other files during test execution,
• Specify which tests to execute using a test plan file.

Let’s learn how to do some of these with our example.

Specify Title, Description, Links

Let’s modify one of the tests and add a few decorators. You can find a complete list here.

tests/test_text_processor.py

1
2
3
4
5
6
7
8
9
10
11
12

@allure.title("Test count_words function")  
@allure.description("Test the count_words function")  
@allure.tag("text_processor")  
@allure.severity(allure.severity_level.CRITICAL)  
@allure.label("owner", "John Doe")  
@allure.link("https://pytest-with-eric.com", name="Test Automation Blog")  
@allure.issue("TMS-456", name="Bug in count_words function")  
@allure.testcase("TMS-123", name="Test count_words function")  
def test_count_words():  
    assert count_words("hello world") == 2  
    assert count_words("") == 0  
    assert count_words("one") == 1

Running the tests, we get

This test now has a title, description, and much more data.

An important point to note is that you also have the option to use the runtime API, for example,

1
2
3
4
5
6
7
8
9
10
11
12

import allure  
  
def test_authentication():  
    allure.dynamic.title("Test Authentication")  
    allure.dynamic.description("This test attempts to log into the website using a login and a password. Fails if any error happens.\n\nNote that this test does not test 2-Factor Authentication.")  
    allure.dynamic.tag("NewUI", "Essentials", "Authentication")  
    allure.dynamic.severity(allure.severity_level.CRITICAL)  
    allure.dynamic.label("owner", "John Doe")  
    allure.dynamic.link("https://dev.example.com/", name="Website")  
    allure.dynamic.issue("AUTH-123")  
    allure.dynamic.testcase("TMS-456")  
    ...

Which allows you to inject variables at runtime.

Organize Tests by Epics, Stories, and More

Allure also has decorators to classify your tests as JIRA epics, stories, and features.

tests/test_text_processor.py

1
2
3
4
5
6
7

@allure.epic("Text Management")  
@allure.feature("Text processing")  
@allure.story("Reverse string")  
def test_reverse_string():  
    assert reverse_string("abc") == "cba"  
    assert reverse_string("hello") == "olleh"  
    assert reverse_string("") == ""

Running the tests again

1

$ poetry run pytest --alluredir allure-results && poetry run allure serve allure-results

We get a nice “FEATURE BY STORIES” on the Overview page.

And some hierarchy.

You get the point, the more metadata the better and more readable the report.

Parametrized Testing

With parametrized testing, Allure reports automatically show the parameters — input and expected.

Fixture Setup and Teardown

Allure reports cleanly show the fixture setup and teardown steps in the reports too.

1
2
3
4
5
6
7
8
9
10
11
12

import allure  
import pytest  
  
@pytest.fixture()  
@allure.title("Prepare for the test")  
def my_fixture():  
    ...  # set up  
    yield  
    ...  # tear down  
  
def test_with_my_fixture(my_fixture):  
    ...

Image from https://allurereport.org/docs/pytest/

Let’s apply the same to our example.

Using a combination of Allure Steps, in your conftest.py file, add

tests/conftest.py

1
2
3
4
5
6
7
8
9
10
11
12

import pytest  
import allure  
  
  
@pytest.fixture  
@allure.title("Example Word Fixture")  
def word_fixture():  
    with allure.step("Setup: Prepare word fixture"):  
        print("Setup word_fixture")  
    yield "hello"  
    with allure.step("Teardown: Clean up word fixture"):  
        print("Teardown word_fixture")

And we can write a simple test that uses this fixture

tests/test_text_processor.py

1
2

def test_capitalize_text_with_fixture(word_fixture):  
    assert capitalize_text(word_fixture) == "Hello"

You can visualize the setup and teardown steps or sub-fixtures.

Attach Screenshots

Now how about if you want to share screenshots with your team?

Perhaps as part of your Playwright automation sequence or even manually.

That’s also simple with Allure Report.

Let’s add a test

tests/test_text_processor.py

1
2
3
4
5
6
7

def test_attach_screenshot():  
    png_bytes = Path("images/screenshot.png").read_bytes()  
    allure.attach(  
        png_bytes,  
        name="Home Page Screenshot",  
        attachment_type=allure.attachment_type.PNG,  
    )

The above code snippet converts the image into PNG bytes and reads them into the Allure report.

Save an image locally and pass its relative path to the Path method.

See how cool this is?

Likewise, you can also add log files or anything else you want to make the test reports more readable and useful.

Add Environment Variables

Last but not least, having context on the test environment is very important.

After all, running tests locally differs from running them in a CI/CD pipeline. Also, think about Windows vs Mac vs Linux; it’s important to know that information.

Fortunately, Allure reports make this easy.

Simply stick a file environment.properties into the allure-results directory after running the tests.

allure-results/environment.properties

1
2
3
4

os_platform = macOS-12.3-x86_64-i386-64bit  
os_release = OSX Sonoma 14.5  
os_version = 14.5  
python_version = Python 3.12.3

You can automate the generation of this file and once you stick in the information and serve your report, you should see it.

Share Allure Reports with GitHub Pages (via GitHub Actions)

Let’s get into an important feature — while serving these reports locally is great, how do you share the Allure reports with your team?

Allure has integration with the most popular CI/CD tooling — Azure DevOps, GitHub Actions, Bamboo, Jenkins, and TeamCity.

Let’s learn how to serve your Allure report on GitHub Pages via GitHub Actions.

If you’re not familiar with GitHub actions and how you can use it to automate Pytest unit testing, here’s a good starting point.

We want the GitHub Actions flow to run on the master branch.

Let’s also create a branch called gh-pages that the publish GitHub Pages action will use to host the page.

.github/workflows/generate_allure_report.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

name: Run Python tests with Poetry and publish report  
  
on:  
  push:  
    branches:  
      - master  # Adjust this as necessary for your project  
  
jobs:  
  test:  
    runs-on: ubuntu-latest  
    steps:  
      - name: Checkout repository  
        uses: actions/checkout@v3  
  
      - name: Set up Python  
        uses: actions/setup-python@v4  
        with:  
          python-version: '3.12'  
  
      - name: Install Poetry  
        run: |  
          curl -sSL https://install.python-poetry.org | python3 -  
  
      - name: Configure Poetry  
        run: |  
          poetry config virtualenvs.create false  
  
      - name: Install dependencies with Poetry  
        run: |  
          poetry install  
  
      - name: Run tests and generate Allure results  
        run: |  
          poetry run pytest --alluredir=allure-results  
  
      - name: Generate Allure report  
        uses: simple-elf/allure-report-action@v1.7  
        if: always()  
        with:  
          allure_results: allure-results  
          allure_history: allure-results  
  
      - name: Publish Allure report on GitHub Pages  
        uses: peaceiris/actions-gh-pages@v3  
        if: always()  
        with:  
          github_token: ${{ secrets.GITHUB_TOKEN }}  
          publish_branch: gh-pages  
          publish_dir: allure-results

Breaking down the workflow file

• Checkout the current repository
• Setup Python
• Install Poetry and dependencies
• Generate our Allure report using this action
• Publish the Allure report on GitHub Pages

We chose 2 branches to maintain a separation — the main workflow will run on master while gh-pages will host the published allure report.

Running this workflow gives us

You can find the link to the workflow actions here.

Pretty cool, right?

This is a simple way to host your Allure report and can even be integrated and shared within your Pull Request and possibly connected into JIRA or other agile reporting tools.

Other ways to share reports include publishing them to AWS S3 to host the static files or a Heroku or Netlify server.

Allure Reports with Pytest BDD

Allure reports also integrate very nicely with pytest-bdd and I’ll cover this in more detail in another article.

Meanwhile, you can read up on how this works here.

If you’ve never heard of or used BDD (Behavior-Driven Development), it’s a methodology of writing and running tests based on expected user behavior.

I’ve written a full-scale article to help you get started with Pytest and BDD so encourage you to check that out.

Reset Allure Reports

During development, it’s common to have a lot of test failures. However, you likely want a clean report when sharing with colleagues.

Hence the need to reset your report.

One of the easiest and fastest ways is to simply delete your allure-results folder.

Unfortunately, it means you will lose all history and probably your environment file (but you can save that).

Conclusion

OK, this is the end.

I hope you found this article useful and learned enough to get started with Allure reports and Pytest.

We started by addressing the need for testing reports and why it’s so important.

Then we looked at the backstory of Allure reports and went through a Quick Start Guide to get you creating your first report.

We broke down the report step by step and learned new ways to enhance your report e.g. add metadata, group tests by epic, story, add screenshots, environment variables, and so on.

Lastly, you learned how to generate an Allure report with GitHub Actions and share it on GitHub pages.

I hope you feel inspired and intrigued to try this on your own and implement better test reporting practices in your team to increase quality and reduce bugs.

If you have any ideas for improvement or like me to cover any topics please message me on Twitter, GitHub, or Email.

Till the next time… Cheers!

Additional Reading

Example Code Used
Allure Pytest - Official Docs
Example Allure Report
Allure integration with pytest
How To Run Pytest With Poetry (A Step-by-Step Guide)
Test Automation Made Easy with Pytest and Playwright
Automated Python Unit Testing Made Easy with Pytest and GitHub Actions
How to Effortlessly Generate Unit Test Cases with Pytest Parameterized Tests
A Complete Guide To Behavior-Driven Testing With Pytest BDD
How To Create Custom HTML Test Reports With pytest-html