Writing automated tests with pytest#

In this section we will go over writing automated tests using pytest which is a testing framework for python.

Learning outcomes#

You will learn about

Why you should write automated test
How to write and run easy tests for pytest’
Use numpy testing for
How to write multi-parameter tests

Required#

pytest module

Why use automated testing#

As hardware engineers we are often rightfully concerned with accuracy, reproducability and calibration of our measurement equipment and when we test equipment we often compare against an expected response, performance … However, often we take much less care about our software and numerical code and if it performs correctly and instead do basic ad-hoc testing. For example do you always confirm that a change you made to your code did not result in an unexpected side-effect?

Two horror stories:

Automatic testing allows you to:

ensure that expected functionality is preserved
verify that code is doing what it is supposed to do
easier refactoring of code
easier contributions from external developers

Testing concepts#

Imperfect tests that exists and are run are better than perfect tests that do not exist.

When testing you should

test often
ideally test automatically (using continuous integration for example)
test with numerical accuracy in mind

There are basically three types of tests.

Unit tests#

Unit tests test a single unit, e.g. module or function
Provide documentation of capability of the function (some frameworks even integrate them into the documentation)

Integration tests#

Also called functionality tests

Verify that your modules are working well together
Test the functionality of your project, e.g. do your simulations get correct results for known cases

Regression tests#

Test over different versions of the code base
Can for example test if performance remains the same or does not regress

We will not discuss regression tests here and largely focus on unit tests.

Test frameworks#

There exists a number of testing frameworks for Python, many with different advantages and disadvanteges. The most common are:

We will focus on pytest.

pytest#

Pytest is easy to use and write test for.

Installation#

If you are using anaconda you can install pytest via conda or the anaconda navigator. It is also contained in many linux distributions or alternatively it can be installed with pip install pytest.

A first test#

Let us write a function an test it’s functionality. Here we are writing function and tests in the same file, in practical projects tests are typically stored in a separate directory (e.g. tests) and the functions and modules to be tested are imported.

# contents of test_example1.py
def fahrenheit2celcius(T_f):
    """
    Convert temperature in Fahrenheit to Celcius
    """
    T_c = (T_f - 32.) * (5/9.)
    return T_c


def test_fahrenheit2celcius():
    T_c = fahrenheit2celcius(32.)
    expected_result = 0.
    assert T_c == expected_result

!pytest test_example1.py

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 1 item                                                               

test_example1.py .                                                       [100%]

============================== 1 passed in 0.04s ===============================

Above we have a function which converts temperature in Fahrenheit to temperature in Celcius. We test that the function works correctly by testing against a known result at 0 degress Celcius.

pytest will run all files of the form test_*.py or *_test.py in the current directory and

all test prefixed functions or methods inside these files
all test prefixed functions or methods inside Test prefixed test classes (without an init method)

You can review the test discovery conventions here

Generally you should use assert in your tests as this will allow pytest to use advanced assertion introspection which will give you more information on the test failure.

Parametrizing#

Now in the above test we only test for a single result. We could write test for multiple known results, but that quickly becomes tedious. Fortunately pytest offers a way to avoid much of this boilerplate.

import pytest

# contents of test_example2.py
def fahrenheit2celcius(T_f):
    """
    Convert temperature in Fahrenheit to Celcius
    """
    T_c = (T_f - 32.) * (5/9.)
    return T_c


@pytest.mark.parametrize("t", [(32,0), (451, 232.778)])
def test_fahrenheit2celcius(t):
    T_c = fahrenheit2celcius(t[0])
    expected_result = t[1]
    assert T_c == expected_result

!pytest test_example2.py

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 
collected 2 items                                                              

test_example2.py .

F                                                      [100%]

=================================== FAILURES ===================================
_________________________ test_fahrenheit2celcius[t1] __________________________

t = (451, 232.778)

    @pytest.mark.parametrize("t", [(32,0), (451, 232.778)])
    def test_fahrenheit2celcius(t):
        T_c = fahrenheit2celcius(t[0])
        expected_result = t[1]
>       assert T_c == expected_result
E       assert 232.7777777777778 == 232.778

test_example2.py:16: AssertionError
=========================== short test summary info ============================
FAILED test_example2.py::test_fahrenheit2celcius[t1] - assert 232.7777777777778 == 232.778
========================= 1 failed, 1 passed in 0.03s ==========================

This test iterated over all parameters, however it fails for the second parameter set due to numerical accuracies.

Numpy testing#

We could use introduce our own way of accounting for numerical accuracy and do somthing like:

assert abs(T_c - expected_result) < 1e-5

Fortunately there is the numpy testing module that was specifically designed for testing numerical code with limited accuracty, so lets use that instead.

import pytest
import numpy.testing as npt

# contents of test_example3.py
def fahrenheit2celcius(T_f):
    """
    Convert temperature in Fahrenheit to Celcius
    """
    T_c = (T_f - 32.) * (5/9.)
    return T_c


@pytest.mark.parametrize("t", [(32,0), (451, 232.778)])
def test_fahrenheit2celcius(t):
    T_c = fahrenheit2celcius(t[0])
    expected_result = t[1]
    npt.assert_almost_equal(T_c, expected_result, 3)

!pytest test_example3.py

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 2 items                                                              

test_example3.py ..                                                      [100%]

============================== 2 passed in 0.05s ===============================

import numpy.testing as npt
help(npt.assert_almost_equal)

Help on function assert_almost_equal in module numpy.testing._private.utils:

assert_almost_equal(actual, desired, decimal=7, err_msg='', verbose=True)
    Raises an AssertionError if two items are not equal up to desired
    precision.
    
    .. note:: It is recommended to use one of `assert_allclose`,
              `assert_array_almost_equal_nulp` or `assert_array_max_ulp`
              instead of this function for more consistent floating point
              comparisons.
    
    The test verifies that the elements of `actual` and `desired` satisfy.
    
        ``abs(desired-actual) < float64(1.5 * 10**(-decimal))``
    
    That is a looser test than originally documented, but agrees with what the
    actual implementation in `assert_array_almost_equal` did up to rounding
    vagaries. An exception is raised at conflicting values. For ndarrays this
    delegates to assert_array_almost_equal
    
    Parameters
    ----------
    actual : array_like
        The object to check.
    desired : array_like
        The expected object.
    decimal : int, optional
        Desired precision, default is 7.
    err_msg : str, optional
        The error message to be printed in case of failure.
    verbose : bool, optional
        If True, the conflicting values are appended to the error message.
    
    Raises
    ------
    AssertionError
      If actual and desired are not equal up to specified precision.
    
    See Also
    --------
    assert_allclose: Compare two array_like objects for equality with desired
                     relative and/or absolute precision.
    assert_array_almost_equal_nulp, assert_array_max_ulp, assert_equal
    
    Examples
    --------
    >>> from numpy.testing import assert_almost_equal
    >>> assert_almost_equal(2.3333333333333, 2.33333334)
    >>> assert_almost_equal(2.3333333333333, 2.33333334, decimal=10)
    Traceback (most recent call last):
        ...
    AssertionError:
    Arrays are not almost equal to 10 decimals
     ACTUAL: 2.3333333333333
     DESIRED: 2.33333334
    
    >>> assert_almost_equal(np.array([1.0,2.3333333333333]),
    ...                     np.array([1.0,2.33333334]), decimal=9)
    Traceback (most recent call last):
        ...
    AssertionError:
    Arrays are not almost equal to 9 decimals
    <BLANKLINE>
    Mismatched elements: 1 / 2 (50%)
    Max absolute difference: 6.66669964e-09
    Max relative difference: 2.85715698e-09
     x: array([1.         , 2.333333333])
     y: array([1.        , 2.33333334])

numpy.testing also contains test for equality of arrays testing for absolute or relative equality etc. We highly recommend to use it in your tests.

Testing for exceptions#

Sometimes it might be desirable to test that functions raise exceptions when given the wrong input, instead of continuing and possibly producing bogus results much further down the line (which is often much harder to debug).

In our example it does not make sense that the input temperature is complex, but the conversion function would still calculate with a complex input which might have unexpected consequences further down the line. Let us raise and exception and test for it.

!cat test_example4.py

# contents of test_example4.py
import pytest
import numpy.testing as npt
import numpy as np

def fahrenheit2celcius(T_f):
    """
    Convert temperature in Fahrenheit to Celcius
    """
    if not np.isreal(T_f):
        raise TypeError("Temperature needs to be a real value")
    T_c = (T_f - 32.) * (5/9.)
    return T_c


@pytest.mark.parametrize("t", [(32,0), (451, 232.778)])
def test_fahrenheit2celcius(t):
    T_c = fahrenheit2celcius(t[0])
    expected_result = t[1]
    npt.assert_almost_equal(T_c, expected_result, 3)

def test_fahrenheit2celcius_type():
    with pytest.raises(TypeError):
        fahrenheit2celcius(1+1j)

!pytest test_example4.py

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 3 items                                                              

test_example4.py ...                                                     [100%]

============================== 3 passed in 0.05s ===============================

Grouping tests#

Sometimes it is desirable to group tests that belong together. For example group all tests for a specific function. This can be done by placing all the tests into a class prefixed with Test. Here is an example:

# contents of test_example5.py
import pytest
import numpy.testing as npt
import numpy as np

def celcius2fahrenheit(T_c):
    """
    Convert temperature in Fahrenheit to Celcius
    """
    if not np.isreal(T_c):
        raise TypeError("Temperature needs to be a real value")
    T_f = 9/5* T_c  + 32.
    return T_f

def fahrenheit2celcius(T_f):
    """
    Convert temperature in Fahrenheit to Celcius
    """
    if not np.isreal(T_f):
        raise TypeError("Temperature needs to be a real value")
    T_c = (T_f - 32.) * (5/9.)
    return T_c

class TestConversionf2c(object):
    @pytest.mark.parametrize("t", [(32,0), (451, 232.778)])
    def test_value(self, t):
        T_c = fahrenheit2celcius(t[0])
        expected_result = t[1]
        npt.assert_almost_equal(T_c, expected_result, 3)

    def test_type():
        with pytest.raises(TypeError):
            fahrenheit2celcius(1+1j)


class TestConversionc2f(object):
    @pytest.mark.parametrize("t", [(0, 32), ( 232.778, 451)])
    def test_value(self, t):
        T_c = fahrenheit2celcius(t[0])
        expected_result = t[1]
        npt.assert_almost_equal(T_c, expected_result, 3)

    def test_type():
        with pytest.raises(TypeError):
            fahrenheit2celcius(1+1j)

!pytest test_example5.py

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 6 items                                                              

test_example5.py ......                                                  [100%]

============================== 6 passed in 0.06s ===============================

Specifying tests#

This has the advantage that we can run the tests of only one group (class), for example if we working on one function but don’t want to rerun all tests in the file (e.g. because it takes to long). To run only the test in the Celcius to Fahrenheit conversion tests one would use:

!pytest test_example5.py::TestConversionc2f

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 3 items                                                              

test_example5.py ...                                                     [100%]

============================== 3 passed in 0.05s ===============================

One can similarly only run the test of one method

!pytest test_example5.py::TestConversionc2f::test_type

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 1 item                                                               

test_example5.py .                                                       [100%]

============================== 1 passed in 0.05s ===============================

More complex selections are also possible (see pytest docs specifying test for more details). For example run all type tests.

!pytest test_example5.py -k "test_type"

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/jschrod/Work/Code/OFCshortcourse/Notebooks/Hands_on_Advanced
plugins: anyio-4.3.0
collecting ... 

collected 6 items / 4 deselected / 2 selected                                  

test_example5.py ..                                                      [100%]

======================= 2 passed, 4 deselected in 0.05s ========================

Integration for automated testing#

We do not have the time to cover how to integrate testing with other systems to achieve fully automated testing, however it should be noted that testing develops its full power when it integrates with the build or version control system to create a fully automated test environment. Two possibilities are:

integrate testing with the build system (setuptools, disttools) to run tests every time a new release is made.
integrate with the hosted version control system to run tests every time changes are added to the master (or a release branch). This is typically achieved using so-called continuous integration.

Note on hardware#

While we focused here on running tests for software packages, it is quite easy to extend this work with hardware. This would enable to straight forwardly test performance of measurements or devices. Luceda Photonics have shown some interesting work on automated testing of integrated circuits from design and modelling to measurement and validation of fabricated devices.

Summary#

Hopefully we have given you a taste of how you can use automated testing to improve your wor

Writing automated tests with pytest

Contents