Pytest with Eric

Dealing with failing tests is a misery.

Imagine your new code breaks old tests and you have no idea why, preventing you from merging your latest code.

Your tests are slow and break every time you change the code even slightly, feeling more like a massive dumbbell you dread when merging PRs, rather than a net or signal to protect or guide you.

On the other side of the spectrum, maybe you don’t even know what to test.

Should you test every function in your application? What if the code interacts with a database or external service? Should you use mocks?

Have you heard about Test-Driven Development (TDD) but have no idea what it means?

Maybe you know what it is in theory, but never applied it in practice.

Does the expectation of writing tests before code cause stress or anxiety that you’re wasting time in lost productivity?

What about if you’re working at a start-up and the code will be irrelevant in a few months? — Should you still practice TDD?

Have you ever wanted to run a single unit test only to find yourself running the entire test suite every time, wasting valuable time and resources?

Running a single test is entirely possible with Pytest, but it’s not as straightforward or user-friendly.

What if you want to define configuration for your tests, such as environment variables, custom arguments, verbosity and run them with a single click repeatedly?

How about if you wanted to add a debugger or line breakpoints to your tests?

You’d kinda struggle to do all of this in the terminal without some advanced command-line flags, what if there was an easier way?

Have you ever thought about testing your software behavior from a user’s perspective?

You’re probably familiar with unit tests, maybe even integration tests and regression tests. But what about the entire feature and its behavior?

For example, the user logs in, adds products to a shopping cart, and checks out.

How do you validate this entire end-to-end scenario? There could be infinite scenarios.

How to bridge the gap between product engineers, QA testers, and developers — to make sure you’re speaking the same language?

Have you ever struggled with testing command-line arguments for your applications or scripts?

Perhaps you’ve build a robust application, with a database and REST API and interfaced via command-line (CLI).

You’ve tested the database and REST API, but what about the CLI?

How do you test that your code correctly handles missing arguments, wrong data types, or invalid strings or characters?

Command Line Arguments are a prime error candidate for errors, given their immense interaction with the end user. Hence, its crucial to ensure your application correctly processes user inputs and handles errors gracefully.

How do you do this without redefining all arguments in your tests? How do you abstract that layer?

As a website test automation engineer, have you ever struggled with repeatedly doing the same UI/UX actions?

Human manual testing is error-prone, time-consuming and in fact… just boring.

It also lacks scale - testing time grows linearly with complexity and number of tests.

Feedback is slow and fairly manual, introducing many unknown variables due to human senses and memory.

So how can you improve this? How can you automate Web Browser UI testing?

Have you ever wondered how to customize the behaviour of your test suite in Pytest?

Perhaps you want to set a configfile path or read it, before executing your tests?

Maybe set some global variables for Dev or Prod or send Slack notifications after the test suite has finished running?

How do you achieve this in Pytest?

Have you ever had tests fail due to irrelevant or outdated code?

Perhaps adding to the confusion of debugging and troubleshooting along with unnecessary test execution time.

Or maybe you have code in development that’s not ready to be tested yet.

In such cases, you might want to exclude certain tests from your test suite.

But how do you do it in Pytest? You can of course run pytest in the relevant directories but that’s not very efficient.

Almost every backend system uses a database today. After all, you need to persist system and user information.

But with that comes a challenge, your code needs to be able to efficiently read and write from a database. And it’s your job to thoroughly test it.

Databases are capable of smoothly handling read/writes, transactions and rollbacks.

So how do you test your interactions with the database?

How do you leverage the inbuilt database features to write robust I/O operations?

What about handling errors, rollbacks, transactions or network failures? Can you test for these?

Have you ever spent more time fiddling with test settings than actually writing tests?

Maybe you’ve been through the Pytest docs and seen a bunch of config (short for Configuration) files like pytest.ini, tox.ini, pyproject.toml, and setup.cfg and wondered what they are and how to use them?

Or perhaps you’re familiar with these files but did you know you can also define Pytest config in files like pyproject.toml and setup.cfg?

Say you’re working on a large test suite with a bunch of tests, each with its own set of requirements and dependencies, across different environments.

You want to run initialization commands in different environments, set up global variables, maybe slack notifications or ping an external API via a webhook to say your tests have begun.

How do you do this in Pytest? How do you tune Pytest to automatically execute tasks before running your tests?

After all you don’t want to write another script to do this. That’s just more code to maintain and more things to go wrong.

Imagine this, you’ve just joined a company and your first task is to optimise the existing test suite and improve the CI/CD run time.

Or maybe you’re a veteran Python developer and with success, your test suite has grown substantially to 1000 tests, maybe 2000 tests?

Now it’s getting out of control and each run takes 5 mins or more. CI/CD Pipelines cost a small fortune.

You’re testing backend APIs, data ETL processes, user authentication, disk writes and so on.

How do you optimize test run time? Where do you start?

With time-dependent code, a key challenge arises in testing it across various time and dates.

Consider a scenario where you’re working on a global transaction application. This application involves several time-dependent activities such as recording transaction time, date, the timezone of the transaction, and more.

How do you ensure your application functions correctly when a transaction occurs outside your country?

One approach might be to manually adjust the time, date, and timezone of your system. However, this method is non-deterministic, time-consuming, and impractical for running individual tests.

So, what’s the solution?

With Pytest, one of the tricker aspects can be understanding and controlling the standard output (stdout) and standard error (stderr).

Often, during testing, your code generates output or errors that are directed to the console.

This can lead to cluttered and obscure test outputs, making it challenging to decipher test results and debug effectively.

So how do you control what’s shown on the console and make your tests easier to debug?

You often face daunting challenges in Python - writing robust tests, managing multiple testing environments, and handling complex dependencies.

These hurdles can turn even the most straightforward project into a labyrinth of complexities.

How do you ensure your code stands the test of time and technology?

How do you maintain consistency across various environments?

And how do you streamline your project’s dependencies without getting lost in a web of version conflicts?

You may have come across the commands pytest and python -m pytest but find yourself puzzled about their differences.

Maybe you’ve seen your colleagues use either command interchangeably, and you’re wondering if they are the same.

Well, you’re not alone! It’s time to shed light on this mystery!

Releasing software without testing is like flying a plane without a maintenance check.

Testing is a safety net that helps make sure your code works well, uncovers bugs and keeps the code in good shape over its lifetime.

When developing with Python, you can use the built-in testing library called PyUnit (Unittest).

However, there are other popular and highly equipped Python testing frameworks such as Pytest, Nosetest, Doctest, and Robot - which arguably may even be better than the built-in one.

Now given so many options, how do you choose which Python testing framework to use in your application? How do you balance the trade-offs between ease of use, features and performance?

In software development, the importance of unit testing cannot be overstated.

A poorly constructed test suite can lead to a fragile codebase, hidden bugs, and hours wasted in debugging.

Not to mention hard to maintain, extend, redundant, non-deterministic and no proper coverage. So how do you deal with the complexities of crafting effective unit tests?

How do you ensure your code works as expected? What’s the difference between good and great software?

While developing software, keeping track of events is crucial.

Logging serves as a means of tracking events to catch software bugs as they happen and learn more about the use of your code.

However, as your test suites grow in complexity, so does the need for effective logging to diagnose issues and maintain a clear understanding of test execution.

So, how can you optimize Pytest Logging?

Ever been baffled by the “Pytest collected 0 items” error message?

It’s like setting off on a testing treasure hunt, only to realize you don’t have a map. This error often leaves many scratching their heads – what’s wrong? Why can’t Pytest find my tests?

At the heart of this mystery is Pytest’s test discovery mechanism.

Even in a well-organized project, Pytest comes back empty-handed, flashing the infamous “collected 0 items” message. It’s as if your tests have turned invisible!