Blogs

Developer Zone

Advanced Software Development with MATLAB

Equivalence Testing

Posted by Tim Johns,

36 views (last 30 days)  |   Likes   |   0 comment
This is part 4 in my series on MATLAB Test in which we’ll look at the new equivalence testing functionality it provides for MATLAB code. Previously, I have covered:
  1. An introduction to MATLAB Test and the Test Manager
  2. The use of advanced coverage metrics
  3. Measuring and monitoring code quality.

What is equivalence testing and why do I need it?

In MATLAB, we can transform MATLAB code to C or C++ using MATLAB Coder, or to .NET assemblies, Java classes, or Python packages using MATLAB Compiler SDK. In such cases, we will often want to verify that the transformed version of our code behaves in exactly the same way as the original MATLAB code. This is particularly important in safety critical environments such as medical devices or ADAS where it is the transformed code that is used in the final product.
Equivalence (or back-to-back) testing is a form of dynamic testing that checks that these two software systems produce the same output when given the same inputs. MATLAB Test provides a new framework that makes it easy to write such equivalence tests.

Basic syntax

C/C++ code

The syntax for writing equivalence tests using the framework provided by MATLAB Test will be very familiar if you’re used to writing class-based unit tests. Given a function myAdd defined as:
function y = myAdd(a,b)
y = a + b;
end
The equivalence test for C code is:
classdef tEquivalence < matlabtest.coder.TestCase
methods(Test)
function tMyAdd(testCase)
buildResults = build(testCase,"myAdd",Inputs={1,2});
executionResults = execute(testCase,buildResults);
verifyExecutionMatchesMATLAB(testCase,executionResults)
end
end
end
There are a few things to note:
  1. Our test class inherits from matlabtest.coder.TestCase rather than the usual matlab.unittest.TestCase.
  2. The new build method builds the C binary that we will execute as part of the test. We specify the name of the entry point function we want to build (myAdd) and some default inputs. These "compile-time" inputs are required for code generation and are similar to codegen’s ­-args flag. It’s an optional argument as it’s possible to write functions with no input arguments.
  3. The new execute method executes the C binary using the default inputs. You can also specify different "run-time" inputs here if you so wish.
  4. The verifyExecutionMatchesMATLAB method does what it says – it produces a verification failure if the output from the C code does not match that of MATLAB.
The documentation contains more detailed information such as how to configure additional code generation options, reuse existing generated C or C++ code, or test generated code with multiple entry points.

.NET, Java, Python

For MATLAB Compiler SDK workflows, the syntax is very similar:
classdef tDeployment < matlabtest.compiler.TestCase
methods (Test)
function pythonEquivalence(testCase)
buildResults = build(testCase,"myAdd.m","pythonPackage");
executionResults = execute(testCase,buildResults,{1,2});
verifyExecutionMatchesMATLAB(testCase,executionResults);
end
end
end
We now inherit from matlabtest.compiler.TestCase and the signatures of the build and execute functions are very slightly different to those of matlabtest.coder.TestCase that we looked at above.
Again, the documentation gives full details of the syntax such as how to specify additional MATLAB Compiler SDK options.

Reusing unit tests for equivalence testing

In this section I’m going to show how tests that have be written to verify a MATLAB algorithm can be repurposed for equivalence testing. The implementation shortestPath and corresponding 14 test points are based on the example project that comes with MATLAB Test.

Overview

The basic structure of the tests is:
classdef tMATLABTests < matlab.unittest.TestCase
methods (Test)
function aTestPoint(testCase)
% Generate inputs
adjacency = …
startIdx = …
endIdx = …
expectedResult = …
debugTxt = …
verifyPathLength(testCase, adjacency,startIdx,endIdx, expectedResult,debugTxt);
end
end
end
where the method verifyPathLength is of the form:
function verifyPathLength(testCase,adjacency,startIdx,endIdx, expectedResult,debugTxt)
% Execute the design
actualResult = shortestPath(adjacency, startIdx, endIdx);
 
% Confirm the expected
msgTxt = sprintf('Failure context: %s', debugTxt);
testCase.verifyEqual(actualResult, expectedResult, msgTxt);
end
I can use the Test Browser to add tMATLABTests to my list of tests and run them:
testBrowser-matlab.png
Output of tMATLABTests in the Test Browser.
verifyPathLength contains all the code that we will need to modify to switch the tests from being standard MATLAB unit tests to equivalence tests. However, it would be nice to retain the original tests whilst adding the ability to run equivalence tests without duplicating code. We can do this by subclassing tMATLABTests and overloading the verifyPathLength method.

Equivalence test for C code

To support the equivalence testing of C code, I’m going to create a subclass of tMATLABTests called tEquivalenceTestsForC. This code needs to do 4 things:
  1. Inherit from matlabtest.coder.TestCase in addition to tMATLABTests so that we have access to the equivalence testing functionality that we need.
  2. Build the code – it’s most efficient if we do this once for the test class rather than for each test point individually. We can use a TestClassSetup block to do this and store the result as a property. There’s an additional catch that the adjacency input is not of fixed size. We need to tell MATLAB Coder about this using the coder.typeof function.
  3. Execute the code for the given inputs.
  4. Compare the results to MATLAB.
Here’s the code:
classdef tEquivalenceTestsForC < tMATLABTests & matlabtest.coder.TestCase
properties (Access = private)
BuildResults
end
methods (TestClassSetup)
function generateCode(testCase)
% Build the function once and store the results for reuse -
% much more efficient that building for every test point.
% Define input arguments that will allow shortestPath to build.
% Since adjacency can be of variable size, we use coder.typeof
% to declare that it's a double matrix that can be up to 20x20
% and that both rows and columns can change size.
adjacency = coder.typeof(0,[20 20],[true true]);
inputs = { adjacency -1 2};
% Build the function
testCase.BuildResults = testCase.build("shortestPath",Inputs=inputs);
end
end
methods
function verifyPathLength(testCase,adjacency,startIdx,endIdx,~,~)
% Execute the function in both MATLAB and C using the inputs
% provided by each test point.
executionResults = testCase.execute(testCase.BuildResults,Inputs={adjacency,startIdx,endIdx});
% Verify C matches MATLAB.
testCase.verifyExecutionMatchesMATLAB(executionResults)
end
end
end
As before, we can add tEquivalenceTestsForC to the Test Browser and run the tests:
testBrowser-c.png
Output of tEquivalenceTestsForC in the Test Browser.

Equivalence test for Python

To support the equivalence testing of Python code, I’m going to create a second subclass of MATLABTests called tEquivalenceTestsForPython. Whilst this example is for Python, you can follow the same procedure for Java or .NET. Our code needs to do 4 things:
  1. Inherit from matlabtest.compiler.TestCase (note compiler, not coder!) in addition to tMATLABTests so that we have access to the equivalence testing functionality that we need.
  2. Build the code – it’s most efficient if we do this once for the test class rather than for each test point individually. We can use a TestClassSetup block to do this and store the result as a property.
  3. Execute the code for the given inputs.
  4. Compare the results to MATLAB.
Note that since Python is a dynamically typed language, we don’t need to use the coder.typeof construct that we had for C.
Here’s the code:
classdef tEquivalenceTestsForPython < tMATLABTests & matlabtest.compiler.TestCase
properties (Access = private)
BuildResults
end
methods (TestClassSetup)
function generateCode(testCase)
% Build the function once and store the results for reuse -
% much more efficient that building for every test point.
testCase.BuildResults = testCase.build("shortestPath.m","pythonPackage");
end
end
methods
function verifyPathLength(testCase,adjacency,startIdx,endIdx,~,~)
% Execute the function in both MATLAB and Python using the
% inputs provided by each test point.
executionResults = testCase.execute(testCase.BuildResults,{adjacency,startIdx,endIdx});
% Verify Python matches MATLAB.
testCase.verifyExecutionMatchesMATLAB(executionResults)
end
end
end
We can then run the tests in the Test Browser. Note that you must have a compatible version of Python installed – see pyenv for more details.
testBrowser-python.png
Output of tEquivalenceTestsForPython in the Test Browser.
In this case we have a test failure which we will need to investigate further.

Conclusion

In this blog post we’ve looked at the new functionality available in MATLAB Test to support equivalence testing when transforming MATLAB code to C, C++, Python, Java, or .NET. Equivalence testing is particularly important in safety critical environments such as medical devices or ADAS. We also looked at how inheritance can be leveraged to reuse existing MATLAB tests for equivalence testing.
In the next and final post in the series, I will explore dependency-based test selection.

Code listing

shortestPath

function pathLength = shortestPath(adjMatrix, startIdx, endIdx) %#codegen
% SHORTEST_PATH - Finds length of shortest path between nodes in a graph
%
% OUT = SHORTEST_PATH(ADJMTX, STARTIDX, ENDIDX) Takes a graph represented by
% its adjacency matrix ADJMTX along with two node STARTIDX, ENDIDX as
% inputs and returns a integer containing the length of the shortest path
% from STARTIDX to ENDIDX in the graph.
 
% Copyright 2021 The MathWorks, Inc.
%% Validy testing on the inputs
% This code should never throw an error and instead should return
% error codes for invlid inputs.
ErrorCode = 0;
pathLength = -1;
 
% Check the validity of the adjacency matrix
if (~isAdjMatrixValid(adjMatrix))
ErrorCode = -9;
end
 
% Check the validity of the startIdx
if ~isNodeValid(startIdx)
ErrorCode = -19;
end
 
% Check the validity of the endIdx
if ~isNodeValid(endIdx)
ErrorCode = -29;
end
 
[nodeCnt, n] = size(adjMatrix);
 
% Start or end node is too large
if startIdx > nodeCnt || endIdx > nodeCnt
ErrorCode = -99;
end
 
% Start or end node is too small
if startIdx < 1 || endIdx < 1
ErrorCode = -199;
end
 
if (ErrorCode<0)
pathLength = ErrorCode;
return;
end
 
%% Self-loop path is always 0
if startIdx == endIdx
pathLength = 0;
return;
end
%% Dijkstra's Algorithm
% Dijkstra's Algorithm is used to iteratively explore the graph breadth
% first and update the shortest path until we reach the end node.
 
% Initialization
max = realmax;
visited = false(1, nodeCnt);
 
% The distance vector maintains the current known shortest path from
% the start node to every node. As nodes are processed one by one
% the distance vestor is updated
distance = repmat(max, 1, nodeCnt);
distance(startIdx) = 0;
 
for iterStep = 1:nodeCnt
% At each iteration identify the current node to process that
% is not yet visited and has the smallest distance from the start.
% This breadth first search ensures that we will always reach nodes
% by the shortest possible path.
min = max;
nodeIdx = -1;
for v = 1:n
if ~visited(v) && distance(v) <= min
min = distance(v);
nodeIdx = v;
end
end
 
% Stop iterating when the current distance is maximum because
% this indicates no remaining nodes are reachable
if (min==max)
return;
end
% Mark the current node visited and check if this is end index
visited(nodeIdx) = true;
if nodeIdx == endIdx
pathLength = distance(nodeIdx);
 
if (pathLength==realmax)
% No path exists so set distance to -1;
pathLength = -1;
end
return;
end
% Update distances of unvisited nodes adjacent to the current node
for v = 1:nodeCnt
if(~visited(v) && adjMatrix(nodeIdx, v) ~= 0 && distance(nodeIdx) ~= max)
distVal = distance(nodeIdx) + adjMatrix(nodeIdx, v);
if distVal < distance(v)
distance(v) = distVal;
end
end
end
end
end
 
function out = isNodeValid(node)
% For full coverage we need to create negative tests that make each
% successively make each validity condition false
if(isscalar(node) && isnumeric(node) && ~isinf(node) && floor(node) == node)
out = true;
else
out = false;
end
end
 
function out = isAdjMatrixValid(adjMatrix)
% need to be a square matrix with only 0, 1, or realmax entries.
[m, n] = size(adjMatrix);
 
% For full coverage we need to create negative tests that make each
% successively make each validity condition false
if (m==n) && isempty(find((adjMatrix ~= 0) & (adjMatrix ~= 1), 1))
out = true;
else
out = false;
end
end

tMATLABTests

classdef tMATLABTests < tCommon
% Copyright 2021 The MathWorks, Inc.
 
methods (Test,TestTags="InputTests")
 
function check_invalid_start_1(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = -1;
endIdx = 2;
expOut = -199;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid start index, idx<1');
end
 
function check_invalid_start_2(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 12;
endIdx = 2;
expOut = -99;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid start index, idx>NodeCnt');
end
 
function check_invalid_end_1(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 1;
endIdx = -3;
expOut = -199;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid end index, idx<1');
end
 
function check_invalid_end_2(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 1;
endIdx = 12;
expOut = -99;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid end index, idx>NodeCnt');
end
end
 
methods(Test,TestTags="EdgelessTests")
 
function check_edgeless_graph(testCase)
adjMatrix = zeros(20,20);
startIdx = 1;
endIdx = 18;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Edgeless graph');
end
 
function check_edgeless_start(testCase)
adjMatrix = testCase.graph_some_nodes_edgeless();
startIdx = 1;
endIdx = 4;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Edgeless graph');
end
 
function check_edgeless_end(testCase)
adjMatrix = testCase.graph_some_nodes_edgeless();
startIdx = 3;
endIdx = 1;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Edgeless graph');
end
function check_edgeless_graph_self_loop(testCase)
adjMatrix = zeros(20,20);
startIdx = 16;
endIdx = 16;
expOut = 0;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Self loop in edgeless graph');
end
 
end
 
methods (Test)
 
function check_longest_path(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 1;
endIdx = 4;
expOut = 3;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Longest theoretic path');
end
 
function check_unity_path(testCase)
adjMatrix = testCase.graph_all_edge();
startIdx = 2;
endIdx = 3;
expOut = 1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Path length 1');
end
 
function check_non_unique(testCase)
adjMatrix = testCase.graph_square();
startIdx = 4;
endIdx = 2;
expOut = 2;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Non-unique path');
end
 
function check_no_path(testCase)
adjMatrix = testCase.graph_disconnected_components();
startIdx = 1;
endIdx = 5;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'No path');
end
 
function check_start_end_same(testCase)
adjMatrix = testCase.graph_all_edge();
startIdx = 3;
endIdx = 3;
expOut = 0;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Start and end index same');
end
 
function check_invalid_idx_empty_adj(testCase)
adjMatrix = [];
startIdx = 1;
endIdx = 1;
expOut = -99;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Degenerate empty graph');
end
end
 
methods (Static)
% Utility functions to create common adjacency graph matrices
function adj = graph_straight_seq()
% Create the graph:
% 1---2---3---4
 
adj = [0 1 0 0; ...
1 0 1 0; ...
0 1 0 1; ...
0 0 1 0];
end
 
function adj = graph_square()
% Create the graph:
% 1---2
% | |
% 4---3
 
adj = [0 1 0 1; ...
1 0 1 0; ...
0 1 0 1; ...
1 0 1 0];
end
 
function adj = graph_all_edge()
% Create the graph:
% 1---2
% |\ /|
% |/ \|
% 4---3
 
adj = [0 1 1 1; ...
1 0 1 1; ...
1 1 0 1; ...
1 1 1 0];
end
 
function adj = graph_disconnected_components()
% Create the graph:
% 2 5
% / \ / \
% 1---3 4---6
 
adj = [0 1 1 0 0 0; ...
1 0 1 0 0 0; ...
1 1 0 0 0 0; ...
0 0 0 0 1 1; ...
0 0 0 1 0 1; ...
0 0 0 1 1 0];
end
 
function adj = graph_some_nodes_edgeless()
% Create the graph:
% 2
% / \
% 4---3
%
% Nodes 1, 5, 6 are edgeless
 
adj = [0 0 0 0 0 0; ...
0 0 1 1 0 0; ...
0 1 0 1 0 0; ...
0 1 1 0 0 0; ...
0 0 0 0 0 0; ...
0 0 0 0 0 0];
end
 
end
 
methods
 
function verifyPathLength(testCase,adjacency,startIdx,endIdx,expectedResult,debugTxt)
 
% Execute the design
actualResult = shortestPath(adjacency, startIdx, endIdx);
 
% Confirm the expected
msgTxt = sprintf('Failure context: %s', debugTxt);
testCase.verifyEqual(actualResult, expectedResult, msgTxt);
end
 
end
 
end
|
Category: 
MATLAB Test,
Testing
  • print

See Also

Comments

To leave a comment, please click here to sign in to your MathWorks Account or create a new one.