Easy And Quick Python Local Lambda Unit Testing for AWS Lambda Functions

Posted on In Symbols count in article: 2k Reading time ≈ 7 mins.

You may have noticed that a lot of companies have recently gone serverless with their micro-services.

Using AWS Lambda, Google Cloud Functions, Azure or other alternatives.

The benefits and ease of complexity these technologies offer are immense — no servers to manage to auto-scaling up to 10k requests/second with AWS API Gateway.

I don’t have to convince you of the benefits of serverless but just in case here’s an interesting article to help you decide if your company should go serverless.

While serverless is brilliant, it’s quite challenging for developers to fully test and validate our code in the Cloud Ecosystem locally.

For example, AWS Lamda events, contexts, and handling dependencies are not straightforward as running Python modules locally via the CLI or IDE.

Nevertheless, Python Local Lambda testing is crucial for that short feedback loop you get with local development.

Wouldn’t you like to validate and thoroughly test your source code before you deploy to AWS Lambda?

Well, that’s exactly what we’re gonna learn in this article.

We’ll look at Python Local Lambda Testing — WHAT and HOW to test your Lambda functions locally using Pytest.

We’ll look through a Real Example and learn to test

  • Source Code
  • Exception Handling
  • Response Codes

So let’s get started.

  • What Are Lambda Functions
  • Why Should You Test Your Lambda Functions?
    • Slow Feedback Loop
    • Wasted Execution Time
    • Incorrect or Inefficient Handling Of Events
    • Cannot Run Unit Tests or Coverage Reports Easily
  • What To Test In Your Lambda Functions And How?
    • Source Code Explanation
    • Execute Lambda Functions Locally
    • Unit Testing (Logic, Exception Handling, Responses and Codes)
  • Conclusion
  • Additional Reading

Link To GitHub Repo

What Are Lambda Functions

While this article is not a beginner’s guide to Lambda functions I’d like to give a really brief intro.

An informal definition — A Lambda function is a piece of serverless infrastructure that allows you to execute a block of code without managing the underlying servers or storage.

This could be as simple as a single function or as complex as an entire module or CRUD API.

The biggest benefit is that you don’t need to manage servers, pay only for execution time and easily integrate out of the box with other AWS services e.g. (API Gateway, S3, DynamoDB).

Why Should You Test Your Lambda Functions?

I’m pretty sure you’re fully convinced of the power of Lambda functions.

But if you’ve worked with Lambda functions before, you’re probably wondering,

This all sounds great. But Lambda functions have boilerplate code and
are quite challenging to test locally.

That’s precisely what this article is about.

Here are some of the issues that will haunt you if you skip local development and testing.

Slow Feedback Loop

As a developer, you need to know as quickly as possible if your code runs, or not, with logs to quickly spot and fix errors.

If you need to deploy and execute your code on AWS only to find you made a typo, that’s several wasted minutes and mental energy. Not to mention immensely frustrating.

Python local lambda testing allows you to shorten that feedback loop.

So you can focus on building/improving the core logic, not worrying about the boilerplate code.

Wasted Execution Time

AWS Lambda and other Cloud Functions charge per execution time consumed.

While Lambda functions offer over a million free executions, if you multiply that by 100s of developers, that number quickly adds up.

That’s wasted precious computing power that you could have done locally.

Incorrect or Inefficient Handling Of Events

As stateless code, Lambda functions fit nicely into Event-Driven-Architectures.

They react to events, for example — the user signs up, X button is clicked, or Y action is performed, then do Z.

You may have seen the mandatory event and context arguments when building lambda functions.

Testing Lambda functions locally allows you to test and optimise for a variety of JSON event schemas.

This short feedback loop really accelerates your development.

Cannot Run Unit Tests or Coverage Reports Easily

In this world of TDD (test-driven development), thoroughly testing your code is extremely important.

The rise of testing and data libraries like Hypothesis and Faker gives you immense power to validate your code against varying test data before shipping.

Put simply, there is no excuse for not unit-testing your code with decent coverage.

Try telling your CTO that you didn’t test your code because you don’t know how to test AWS Lambda boilerplate code.

That’s not gonna go down well.

As developers, it’s a great feeling when you’ve tested your code against a whole bunch of test data.

What To Test In Your Lambda Functions And How?

So enough talk.

Let’s look at the highly awaited code examples of how to test our Lambda functions locally.

Source Code Explanation

The GitHub repo includes the source code in the /lambda_calculator folder.

This example of based on an AWS Tutorial that you can find here.

The source code comprises a handler.py file which is typical of what you’d include when building a lambda function - Source code and a handler wrapper.

It accepts event and context arguments as expected.

The goal of this code is to accept an input event containing 3 key-value pairs — action , x and y .

The field action can take values — plus , minus , times and divided-by . x and y are integers.

We are using Pydantic (my favourite Python library) to validate the input payload against a pre-defined schema/model, which nicely alerts us if our payload fails the schema check.

We’ve also defined used enums to specify the allowable action and HTTP response status codes values.

The core logic is simple — validate the event payload and execute the actionoperation on x and y.

/lambda_calculator/handler.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
class Action(Enum):
    plus = "plus"
    minus = "minus"
    times = "times"
    divided_by = "divided-by"


class HTTPStatus(Enum):
    SUCCESS = 200
    REDIRECT = 302
    BAD_REQUEST = 400
    UN_AUTHORIZED = 401
    NOT_FOUND = 404
    CONFLICT = 409
    ERROR = 500


class InputEvent(BaseModel):
    class Config:
        use_enum_values = True

    action: Action
    x: int
    y: int


ACTIONS = {
    "plus": lambda x, y: x + y,
    "minus": lambda x, y: x - y,
    "times": lambda x, y: x * y,
    "divided-by": lambda x, y: x / y
}

def lambda_handler(event, context):
    """
    Accepts an action and two numbers,
    performs the specified action on the numbers,
    and returns the result.
    :param event: The event dict that
    contains the parameters sent when the function
                  is invoked.
    :param context: The context in which the function is called.
    :return: The result of the specified action.
    """
    # Set the log level based on a variable
    # configured in the Lambda environment.
    logger.setLevel(os.environ.get("LOG_LEVEL", logging.INFO))
    logger.debug(f"Event: {event}")
    response = None

    try:
        # Validate Input using Pydantic
        input_event = InputEvent(**event)
        action = input_event.action
        func = ACTIONS.get(input_event.action)
        x = input_event.x
        y = input_event.y
        status_code = 200

        # If input is OK, execute logic
        try:
            if func is not None and \
                    x is not None and \
                    y is not None:
                result = func(x, y)
                response = f"{x} {action} {y} = {result}"
                logger.info(response)
                status_code = HTTPStatus.SUCCESS.value
            else:
                logger.error(f"I can't calculate {x} {action} {y}")
        except ZeroDivisionError:
            logger.warning(f"I can't divide {x} by 0!")
            response = f"I can't divide {x} by 0!"

    except ValidationError as e:
        status_code = HTTPStatus.BAD_REQUEST.value
        logger.error(e.errors())
        response = json.loads(e.json())
    except Exception as general_exception:
        status_code = HTTPStatus.ERROR.value
        logger.error(general_exception)
        response = "Internal Server Error Occurred"

    return {
        "statusCode": status_code,
        "headers": {
            "Content-Type": "application/json"
        },
        "body": json.dumps({
            "Response ": response
        })
    }

Execute Lambda Functions Locally

Admittedly, I was blown away when I first discovered the python-lambda-local library.

For years I’ve been struggling to test Lamba functions locally often with a lot of monkeypatching and mocking.

I was surprised at how simple this is and really works.

Once you’ve got the handler you can easily test your Lambdas locally using

1
python-lambda-local -f lambda_handler -t 5 ./lambda_calculator/handler.py tests/unit/test_events/test_event_plus.json

Feel free to look up the CLI commands in the package documentation.

python local lambda testing

Unit Testing (Logic, Exception Handling, Responses and Codes)

OK great, now that we can run Lambda functions locally, this is amazing.

But how can we call these functions using a Unit Testing framework — say Unittest or Pytest.

Well, that’s simple. You can leverage the following block of code as detailed in the python-lambda-localdocs.

1
2
3
4
5
6
7
8
9
10
11
from lambda_local.main import call  
from lambda_local.context import Context  
  
import test  
  
event = {  
    "answer": 42  
}  
context = Context(5)  
  
call(test.handler, event, context)

Navigating to the /tests folder within the repo, you can see how we’ve used this to execute several test cases, namely

  1. test_lambda_handler_event_plus
1
2
3
4
5
6
def test_lambda_handler_event_plus(event_plus):
    result = call(handler.lambda_handler, event_plus, context)
    expected_response = ({'statusCode': 200,
                          'headers': {'Content-Type': 'application/json'},
                          'body': '{"Response ": "4 plus 4 = 8"}'}, None)
    assert result == expected_response
  1. test_lambda_handler_event_minus
1
2
3
4
5
6
def test_lambda_handler_event_minus(event_minus):
    result = call(handler.lambda_handler, event_minus, context)
    expected_response = ({'statusCode': 200,
                          'headers': {'Content-Type': 'application/json'},
                          'body': '{"Response ": "10 minus 5 = 5"}'}, None)
    assert result == expected_response
  1. test_lambda_handler_event_times
1
2
3
4
5
6
def test_lambda_handler_event_times(event_times):
    result = call(handler.lambda_handler, event_times, context)
    expected_response = ({'statusCode': 200,
                          'headers': {'Content-Type': 'application/json'},
                          'body': '{"Response ": "10 times 5 = 50"}'}, None)
    assert result == expected_response
  1. test_lambda_handler_event_divided_by
1
2
3
4
5
6
def test_lambda_handler_event_divided_by(event_divided_by):
    result = call(handler.lambda_handler, event_divided_by, context)
    expected_response = ({'statusCode': 200,
                          'headers': {'Content-Type': 'application/json'},
                          'body': '{"Response ": "10 divided-by 5 = 2.0"}'}, None)
    assert result == expected_response
  1. test_lambda_handler_event_divided_by_zero
1
2
3
4
5
6
7
def test_lambda_handler_event_divided_by_zero(event_divided_by_zero):
    result = call(handler.lambda_handler, event_divided_by_zero, context)
    expected_response = ({'statusCode': 200,
                          'headers': {'Content-Type': 'application/json'},
                          'body': '{"Response ": "I can\'t divide 10 by 0!"}'},
                         None)
    assert result == expected_response
  1. test_lambda_handler_event_error_invalid_x
1
2
3
4
5
6
7
8
def test_lambda_handler_event_error_invalid_x(event_error_invalid_x):
    result = call(handler.lambda_handler, event_error_invalid_x, context)
    expected_response = ({'statusCode': 400, 
                          'headers': {'Content-Type': 'application/json'},
                          'body': '{"Response ": [{"loc": ["x"], '
                                  '"msg": "value is not a valid integer", '
                                  '"type": "type_error.integer"}]}'}, None)
    assert result == expected_response
  1. test_lambda_handler_event_error_invalid_action
1
2
3
4
5
6
7
8
9
10
11
12
def test_lambda_handler_event_error_invalid_action(event_error_invalid_action):
    result = call(handler.lambda_handler, event_error_invalid_action, context)
    expected_response = (
        {'statusCode': 400,
         'headers': {'Content-Type': 'application/json'},
         'body': '{"Response ": [{"loc": ["action"], '
                 '"msg": "value is not a valid enumeration member; '
                 'permitted: \'plus\', \'minus\', \'times\', \'divided-by\'", '
                 '"type": "type_error.enum", "ctx": {"enum_values": '
                 '["plus", "minus", "times", "divided-by"]}}]}'}, None)

    assert result == expected_response

We’ve defined our test events under tests/unit/test_events and created fixtures in conftest.pyfor each of the test cases.

If you’re unfamiliar with conftest.py I highly recommend you to quickly check out this article on pytest conftest.

conftest.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
import json
import pytest


@pytest.fixture(scope="module")
def event_plus():
    with open("./tests/unit/test_events/"
              "test_event_plus.json") as te:
        test_event_plus = json.loads(te.read())
        return test_event_plus


@pytest.fixture(scope="module")
def event_minus():
    with open("./tests/unit/test_events/"
              "test_event_minus.json") as te:
        test_event_minus = json.loads(te.read())
        return test_event_minus


@pytest.fixture(scope="module")
def event_times():
    with open("./tests/unit/test_events/"
              "test_event_times.json") as te:
        test_event_times = json.loads(te.read())
        return test_event_times


@pytest.fixture(scope="module")
def event_divided_by():
    with open("./tests/unit/test_events/"
              "test_event_divided_by.json") as te:
        test_event_divided_by = json.loads(te.read())
        return test_event_divided_by


@pytest.fixture(scope="module")
def event_divided_by_zero():
    with open("./tests/unit/test_events/"
              "test_event_divided_by_zero.json") as te:
        test_event_divided_by_zero = json.loads(te.read())
        return test_event_divided_by_zero


@pytest.fixture(scope="module")
def event_error_invalid_x():
    with open("./tests/unit/test_events/"
              "test_event_error_invalid_x.json") as te:
        test_event_error_invalid_x = json.loads(te.read())
        return test_event_error_invalid_x


@pytest.fixture(scope="module")
def event_error_invalid_action():
    with open("./tests/unit/test_events/"
              "test_event_error_invalid_action.json") as te:
        test_event_error_invalid_action = json.loads(te.read())
        return test_event_error_invalid_action

Using this combination of fixtures and local Lambda testing, we can smoothly test our code’s response to various event payloads, including HTTP Response Status Code Responses.

This is incredibly useful when building APIs.

Banking on the fact that Pydantic will catch and validate the input payload against our pre-defined schema. Pretty smooth isn’t it? ;)

Here’s a final execution of all Unit Tests from the CLI. We’ve used the pytest-xdist library to execute tests in parallel.

python local lambda unit testing

Conclusion

Hope you found this article useful.

We looked at reasons why you NEED to test your Lambda functions before building and deploying, and how local testing could speed up development.

We explored testing Lambdas locally using the library python-lambda-local and its use in Unit Testing to validate — core logic, exception handling and HTTP response status codes.

From now on, you can treat lambda functions as just another Python module and completely ignore the boilerplate.

Just focus on the core logic and handling errors in the best way possible, as you would do otherwise.

If you have any ideas for improvement or like me to cover any topics please comment below or send me a message via Twitter, GitHub or Email.

Till the next time… Cheers!

Additional Reading

AWS Code Sample