All Software Has Bugs

Random tests are a tool to find bugs you didn’t anticipate. There are many ways to accomplish this, and a few libraries dedicated to it. Here is one simple way that works in Pytest.

The Function We’ll Test

def my_floor(x: float) -> int:
    """Return the floor of x as an integer.

    :param x: The number to floor.
    :return: The largest integer less than or equal to x.
    """
    return int(x)

If you know anything about the floor function, you know exactly when that’s going to fail, but let’s pretend we don’t.

Generate Test Input

We’ll test my_floor by feeding it random numbers and comparing the output to math.floor. To do that, we’ll need random numbers, so write a Python function to generate them.

Note that this is not a pytest fixture.

import random
from collections.abc import Iterator

def iter_random_floats() -> Iterator[float]:
    """Generate random positive and negative numbers."""
    # some floats
    yield from (random.uniform(-1e6, 1e6) for _ in range(20))
    # some integers
    yield from (random.randint(-1000, 1000) for _ in range(20))
    # even where these are included in our random range, likehood of any specific
    # value is very low, throw in some of the usual suspects for breaking things.
    yield 0
    yield math.inf
    yield -math.inf

Feed the Test Function

Now we can write our test function using the pytest.mark.parametrize decorator. This decorator takes some flexible arguments, but here we’ll only need two simple arguments:

• a parameter name as a string
• an iterable of values to use for that parameter

Generate the iterable with a call to iter_random_floats. This is a standard function call without any obscure rules. You can pass arguments to your function to constrain your random variables or ask for a specific number of values. You could also use something like (x/50 for x in range(-100, 100)) or random.sample(range(-100, 100), 20) or [0, math.inf, -math.inf]. Any iterable will do.

Remember to include the parameter name in the function signature—like a pytest fixture.

import math
import pytest

@pytest.mark.parametrize("x", iter_random_floats())
def test_v_math(x: float) -> None:
    """For float input, return the same as math.floor."""
    assert my_floor(x) == math.floor(x)

Run the Tests

When we run this with pytest, we get nice output, telling us which inputs succeeded and which failed. We can use this to find bugs and to re-test against values that failed in the past.

tests/test_my_floor.py::test_v_math[-6399.586] FAILED
tests/test_my_floor.py::test_v_math[5941.9685] PASSED
tests/test_my_floor.py::test_v_math[1857.5379] PASSED
tests/test_my_floor.py::test_v_math[-1603.95252] FAILED
tests/test_my_floor.py::test_v_math[5624.3452] PASSED
tests/test_my_floor.py::test_v_math[-6841.8029] FAILED
tests/test_my_floor.py::test_v_math[477] PASSED
tests/test_my_floor.py::test_v_math[-582] PASSED
tests/test_my_floor.py::test_v_math[-530] PASSED
tests/test_my_floor.py::test_v_math[365] PASSED
...
tests/test_my_floor.py::test_v_math[0] PASSED
tests/test_my_floor.py::test_v_math[inf] FAILED
tests/test_my_floor.py::test_v_math[-inf] FAILED

If we scroll down, we get specific information about each failure. In this case, we got assertion errors for negative floats and overflow errors for positive and negative infinity. The input values are shown there too.

________ test_v_math[-inf] _________

x = -inf

    def my_floor(x: float) -> int:
>       return int(x)
E       OverflowError: cannot convert float infinity to integer

tests\test_my_floor.py:12: OverflowError

Use random tests to throw millions of wrenches into the gears. You may find a few surprises before your users do.