In this post, I'm going to show you how to dynamically generate tests within Pytest using, in my opinion, one of its best features - parametrization.
But first of all ...
What is Pytest?
Pytest is a flexible open-source framework for testing Python code.
Its key strength lies in its simplicity, making tests easy to write and read. With a user-friendly CLI, Pytest automatically discovers and runs tests, presenting results clearly. This makes it ideal for both traditional Python code testing and network testing.
Here's a quick example. We create a Python test using an assert statement placed within a test function and test script. The assert statement will check if the condition is True. If not, an exception is raised. It is this exception that Pytest catches upon runtime to determine which tests have failed.
test.py
def test_vlan():
device_vlan = "100"
expected_vlan = "101"
assert device_vlan == expected_vlan
$ pytest test.py
pytest test.py
...
test.py F [100%]
============================================================ FAILURES ============================================================
___________________________________________________________ test_vlan ____________________________________________________________
def test_vlan():
device_vlan = "100"
expected_vlan = "101"
> assert device_vlan == expected_vlan
E AssertionError: assert '100' == '101'
E - 101
E + 100
test.py:5: AssertionError
==================================================== short test summary info =====================================================
FAILED test.py::test_vlan - AssertionError: assert '100' == '101'
======================================================= 1 failed in 0.01s ========================================================
What is Parametrization?
Pytest parametrization is a feature that allows you to run a single test function against multiple sets of data inputs. Pytest will then dynamically generate tests for each of the input values. Benefits are:
- We don't have to handcraft each test.
- We can stack the parametrization to create a matrix of tests
- It provides better visibility into the result of testing all the inputs.
Before we look at parametrization, let's look at a test without parametrization.
Without Parametrization
Let's say we have the following test script where we will check to see if our expected VLANs are within the device VLANs (on the device). Like so:
test_without_param.py
def test_vlan():
device_vlans = ["100"]
expected_vlans = ["100", "101", "102"]
for v in expected_vlans:
assert v in device_vlans
Note: This is a simplified version in the real world, the VLAN data would be pulled from a device, and the expected VLANs would be pulled from a YAML and an SoT, such as NetBox.
If we were to run our test now, we would see a few things. Mainly that our test failed upon checking for VLAN 101, which meant we had no visibility into the success of VLAN 102.
$ pytest test_without_param.py -vv --tb=short
...
test_without_param.py::test_vlan FAILED [100%]
============================================================ FAILURES ============================================================
___________________________________________________________ test_vlan ____________________________________________________________
test_without_param.py:6: in test_vlan
assert v in device_vlans
E AssertionError: assert '101' in ['100']
...
Using Parametrization
This is where we can use parametrization. To do so, we apply the @pytest.mark.parametrize decorator to our test with the input values. Which for this example will be our expected VLANs. Like so:
test_with_param.py
import pytest
@pytest.mark.parametrize("expected_vlan", ["100", "101", "102"])
def test_vlan(expected_vlan):
device_vlans = ["100"]
assert expected_vlan in device_vlans
If we run our test, we will see that Pytest will dynamically create a test for each input value. In turn, giving us full visibility into the success of each of our input values.
$ pytest test_with_param.py -vv --tb=short
...
test_with_param.py::test_vlan[100] PASSED [ 33%]
test_with_param.py::test_vlan[101] FAILED [ 66%]
test_with_param.py::test_vlan[102] FAILED [100%]
...
The great thing about parametrization is that we can also stack them; below is an example. But in the real world, you could stack the interface error types, interfaces, devices etc., to generate 100’s of tests with a minimal code base.
test_with_param_stacked.py
import pytest
device_vlans = {
"rtr001": ["100", "101"],
"rtr002": ["101", "102"],
"rtr003": ["100", "102"],
}
@pytest.mark.parametrize("device", ["rtr001", "rtr002", "rtr003"])
@pytest.mark.parametrize("expected_vlan", ["100", "101", "102"])
def test_vlan(expected_vlan, device):
assert expected_vlan in device_vlans.get(device, [])
Once run you can see that we have a test created for each device and each expected VLAN.
$ pytest test_with_param_stacked.py -vv
test_with_param_stacked.py::test_vlan[100-rtr001] PASSED [ 11%]
test_with_param_stacked.py::test_vlan[100-rtr002] FAILED [ 22%]
test_with_param_stacked.py::test_vlan[100-rtr003] PASSED [ 33%]
test_with_param_stacked.py::test_vlan[101-rtr001] PASSED [ 44%]
test_with_param_stacked.py::test_vlan[101-rtr002] PASSED [ 55%]
test_with_param_stacked.py::test_vlan[101-rtr003] FAILED [ 66%]
test_with_param_stacked.py::test_vlan[102-rtr001] FAILED [ 77%]
test_with_param_stacked.py::test_vlan[102-rtr002] PASSED [ 88%]
test_with_param_stacked.py::test_vlan[102-rtr003] PASSED [100%]
...
Here's another example (report generated using the Pytest HTML plugin) of using stacked parametrization. But with this example, I stacked the device names, interface names and error types to generate tests for each error type for each device interface.
That wraps this post around Pytest parametrization. It's a top feature of Pytest that is well worth knowing when automating the testing of your network.
