Developer Zone https://blogs.mathworks.com/developer Developing, testing, and integrating production grade software using MATLAB. Thu, 26 Oct 2023 10:53:14 +0000 en-US hourly 1 https://wordpress.org/?v=6.2.2 Dependency-based Test Selection https://blogs.mathworks.com/developer/2023/10/26/dependency-based-test-selection/?s_tid=feedtopost https://blogs.mathworks.com/developer/2023/10/26/dependency-based-test-selection/#respond Thu, 26 Oct 2023 10:53:14 +0000 https://blogs.mathworks.com/developer/?p=3168

This is the final part in my series on MATLAB Test where we’ll look at how to select tests based on files they depend on. Previously, I have covered:An introduction to MATLAB Test and the Test... read more >>

]]>
This is the final part in my series on MATLAB Test where we’ll look at how to select tests based on files they depend on. Previously, I have covered:
  1. An introduction to MATLAB Test and the Test Manager
  2. The use of advanced coverage metrics
  3. Measuring and monitoring code quality
  4. Equivalence testing.
In large and complex projects, running the entire test suite every time something changes can be prohibitively expensive in time and/or compute resource. In these instances, just running the tests that are affected by the changes may be preferable. MATLAB Test provides mechanisms for easily doing this:
In the above, val is a string array of files and folders that the tests must depend on to be selected. A MATLAB Test licence is required to use this feature.
In the following example, I have a project with source code and tests. The Dependency Analyzer shows me that some tests depend on AccountManager whilst those in DocPolynomTest don’t.
dependency-analysis.png
Example project and dependencies.
If I create a test suite from the project, I get 8 test points from both test classes:
>> s = testsuite(pwd);
>> {s.Name}'
ans =
8×1 cell array
{'BankAccountTest/testConstructor' }
{'BankAccountTest/testConstructorNotEnoughInputs'}
{'BankAccountTest/testDeposit' }
{'BankAccountTest/testWithdraw' }
{'BankAccountTest/testNotifyInsufficientFunds' }
{'DocPolynomTest/testConstructor' }
{'DocPolynomTest/testAddition' }
{'DocPolynomTest/testMultiplication' }
If I instead specify that I only want tests that depend on AccountManager, I get the expected subset:
>> s = testsuite(pwd,DependsOn="source/AccountManager.m");
>> {s.Name}'
ans =
5×1 cell array
{'BankAccountTest/testConstructor' }
{'BankAccountTest/testConstructorNotEnoughInputs'}
{'BankAccountTest/testDeposit' }
{'BankAccountTest/testWithdraw' }
{'BankAccountTest/testNotifyInsufficientFunds' }
For more control, you can use the matlabtest.selectors.DependsOn selector which allows you to control whether or not subfolders of the specified folders are included in the search, and what the maximum search depth is. The respective defaults are to not include subfolders and to search the full dependency hierarchy.
The DependsOn selector can also be used in conjunction with the Test Manager’s custom test suite functionality. From the drop-down menu, select “Manage Custom Test Suites”:
manage-test-suites.png
Manage Custom Test Suites.
Create a new test suite that uses the DependsOn selector:
test-suite-manager.png
Create a new test suite using the DependsOn selector.
Save and close, then select the new test suite from the Test Manager drop-down menu to view it:
test-manager.png
New test suite in the Test Manager.
Note that for complex projects, performing the dependency analysis may take some time. There’s therefore a trade-off to be found between the time it takes to do the analysis and the time it takes to run the entire suite.
So how do we get the list of changed files to pass into the DependsOn selector? If you’re using Projects and you have uncommitted changes, the Projects API will tell you:
>> prj = currentProject();
>> prj.listModifiedFiles
ans =
ProjectFile with properties:
Path: "C:\blogs-test-selection\source\BankAccount.m"
Revision: "8974b348f16d445c01409bbcd00f7b9777aa40fd"
SourceControlStatus: Modified
Labels: [1×1 matlab.project.Label]
However, if you have committed changes and/or want to compare one branch to other (e.g. for a pull request), you’ll have to do some manual work with Git to get the list of changed files.
The DependsOn selector determines dependencies with static analysis. Static analysis has some limitations (documented here) which means that it's not infallible. Therefore, I still recommend running a full suite of tests at key stages of your development process.
In conclusion, MATLAB Test provides a mechanism for selecting tests that are impacted by source code changes, allowing you to run the subset of tests that matter and saving testing time.
This completes the series on MATLAB Test. I’ll be back with updates when new MATLAB releases come out!
]]>
https://blogs.mathworks.com/developer/2023/10/26/dependency-based-test-selection/feed/ 0
Equivalence Testing https://blogs.mathworks.com/developer/2023/10/26/equivalence-testing/?s_tid=feedtopost https://blogs.mathworks.com/developer/2023/10/26/equivalence-testing/#respond Thu, 26 Oct 2023 10:36:45 +0000 https://blogs.mathworks.com/developer/?p=3150

This is part 4 in my series on MATLAB Test in which we’ll look at the new equivalence testing functionality it provides for MATLAB code. Previously, I have covered:An introduction to MATLAB Test and... read more >>

]]>
This is part 4 in my series on MATLAB Test in which we’ll look at the new equivalence testing functionality it provides for MATLAB code. Previously, I have covered:
  1. An introduction to MATLAB Test and the Test Manager
  2. The use of advanced coverage metrics
  3. Measuring and monitoring code quality.

What is equivalence testing and why do I need it?

In MATLAB, we can transform MATLAB code to C or C++ using MATLAB Coder, or to .NET assemblies, Java classes, or Python packages using MATLAB Compiler SDK. In such cases, we will often want to verify that the transformed version of our code behaves in exactly the same way as the original MATLAB code. This is particularly important in safety critical environments such as medical devices or ADAS where it is the transformed code that is used in the final product.
Equivalence (or back-to-back) testing is a form of dynamic testing that checks that these two software systems produce the same output when given the same inputs. MATLAB Test provides a new framework that makes it easy to write such equivalence tests.

Basic syntax

C/C++ code

The syntax for writing equivalence tests using the framework provided by MATLAB Test will be very familiar if you’re used to writing class-based unit tests. Given a function myAdd defined as:
function y = myAdd(a,b)
y = a + b;
end
The equivalence test for C code is:
classdef tEquivalence < matlabtest.coder.TestCase
methods(Test)
function tMyAdd(testCase)
buildResults = build(testCase,"myAdd",Inputs={1,2});
executionResults = execute(testCase,buildResults);
verifyExecutionMatchesMATLAB(testCase,executionResults)
end
end
end
There are a few things to note:
  1. Our test class inherits from matlabtest.coder.TestCase rather than the usual matlab.unittest.TestCase.
  2. The new build method builds the C binary that we will execute as part of the test. We specify the name of the entry point function we want to build (myAdd) and some default inputs. These "compile-time" inputs are required for code generation and are similar to codegen’s ­-args flag. It’s an optional argument as it’s possible to write functions with no input arguments.
  3. The new execute method executes the C binary using the default inputs. You can also specify different "run-time" inputs here if you so wish.
  4. The verifyExecutionMatchesMATLAB method does what it says – it produces a verification failure if the output from the C code does not match that of MATLAB.
The documentation contains more detailed information such as how to configure additional code generation options, reuse existing generated C or C++ code, or test generated code with multiple entry points.

.NET, Java, Python

For MATLAB Compiler SDK workflows, the syntax is very similar:
classdef tDeployment < matlabtest.compiler.TestCase
methods (Test)
function pythonEquivalence(testCase)
buildResults = build(testCase,"myAdd.m","pythonPackage");
executionResults = execute(testCase,buildResults,{1,2});
verifyExecutionMatchesMATLAB(testCase,executionResults);
end
end
end
We now inherit from matlabtest.compiler.TestCase and the signatures of the build and execute functions are very slightly different to those of matlabtest.coder.TestCase that we looked at above.
Again, the documentation gives full details of the syntax such as how to specify additional MATLAB Compiler SDK options.

Reusing unit tests for equivalence testing

In this section I’m going to show how tests that have be written to verify a MATLAB algorithm can be repurposed for equivalence testing. The implementation shortestPath and corresponding 14 test points are based on the example project that comes with MATLAB Test.

Overview

The basic structure of the tests is:
classdef tMATLABTests < matlab.unittest.TestCase
methods (Test)
function aTestPoint(testCase)
% Generate inputs
adjacency =
startIdx =
endIdx =
expectedResult =
debugTxt =
verifyPathLength(testCase, adjacency,startIdx,endIdx, expectedResult,debugTxt);
end
end
end
where the method verifyPathLength is of the form:
function verifyPathLength(testCase,adjacency,startIdx,endIdx, expectedResult,debugTxt)
% Execute the design
actualResult = shortestPath(adjacency, startIdx, endIdx);
 
% Confirm the expected
msgTxt = sprintf('Failure context: %s', debugTxt);
testCase.verifyEqual(actualResult, expectedResult, msgTxt);
end
I can use the Test Browser to add tMATLABTests to my list of tests and run them:
testBrowser-matlab.png
Output of tMATLABTests in the Test Browser.
verifyPathLength contains all the code that we will need to modify to switch the tests from being standard MATLAB unit tests to equivalence tests. However, it would be nice to retain the original tests whilst adding the ability to run equivalence tests without duplicating code. We can do this by subclassing tMATLABTests and overloading the verifyPathLength method.

Equivalence test for C code

To support the equivalence testing of C code, I’m going to create a subclass of tMATLABTests called tEquivalenceTestsForC. This code needs to do 4 things:
  1. Inherit from matlabtest.coder.TestCase in addition to tMATLABTests so that we have access to the equivalence testing functionality that we need.
  2. Build the code – it’s most efficient if we do this once for the test class rather than for each test point individually. We can use a TestClassSetup block to do this and store the result as a property. There’s an additional catch that the adjacency input is not of fixed size. We need to tell MATLAB Coder about this using the coder.typeof function.
  3. Execute the code for the given inputs.
  4. Compare the results to MATLAB.
Here’s the code:
classdef tEquivalenceTestsForC < tMATLABTests & matlabtest.coder.TestCase
properties (Access = private)
BuildResults
end
methods (TestClassSetup)
function generateCode(testCase)
% Build the function once and store the results for reuse -
% much more efficient that building for every test point.
% Define input arguments that will allow shortestPath to build.
% Since adjacency can be of variable size, we use coder.typeof
% to declare that it's a double matrix that can be up to 20x20
% and that both rows and columns can change size.
adjacency = coder.typeof(0,[20 20],[true true]);
inputs = { adjacency -1 2};
% Build the function
testCase.BuildResults = testCase.build("shortestPath",Inputs=inputs);
end
end
methods
function verifyPathLength(testCase,adjacency,startIdx,endIdx,~,~)
% Execute the function in both MATLAB and C using the inputs
% provided by each test point.
executionResults = testCase.execute(testCase.BuildResults,Inputs={adjacency,startIdx,endIdx});
% Verify C matches MATLAB.
testCase.verifyExecutionMatchesMATLAB(executionResults)
end
end
end
As before, we can add tEquivalenceTestsForC to the Test Browser and run the tests:
testBrowser-c.png
Output of tEquivalenceTestsForC in the Test Browser.

Equivalence test for Python

To support the equivalence testing of Python code, I’m going to create a second subclass of MATLABTests called tEquivalenceTestsForPython. Whilst this example is for Python, you can follow the same procedure for Java or .NET. Our code needs to do 4 things:
  1. Inherit from matlabtest.compiler.TestCase (note compiler, not coder!) in addition to tMATLABTests so that we have access to the equivalence testing functionality that we need.
  2. Build the code – it’s most efficient if we do this once for the test class rather than for each test point individually. We can use a TestClassSetup block to do this and store the result as a property.
  3. Execute the code for the given inputs.
  4. Compare the results to MATLAB.
Note that since Python is a dynamically typed language, we don’t need to use the coder.typeof construct that we had for C.
Here’s the code:
classdef tEquivalenceTestsForPython < tMATLABTests & matlabtest.compiler.TestCase
properties (Access = private)
BuildResults
end
methods (TestClassSetup)
function generateCode(testCase)
% Build the function once and store the results for reuse -
% much more efficient that building for every test point.
testCase.BuildResults = testCase.build("shortestPath.m","pythonPackage");
end
end
methods
function verifyPathLength(testCase,adjacency,startIdx,endIdx,~,~)
% Execute the function in both MATLAB and Python using the
% inputs provided by each test point.
executionResults = testCase.execute(testCase.BuildResults,{adjacency,startIdx,endIdx});
% Verify Python matches MATLAB.
testCase.verifyExecutionMatchesMATLAB(executionResults)
end
end
end
We can then run the tests in the Test Browser. Note that you must have a compatible version of Python installed – see pyenv for more details.
testBrowser-python.png
Output of tEquivalenceTestsForPython in the Test Browser.
In this case we have a test failure which we will need to investigate further.

Conclusion

In this blog post we’ve looked at the new functionality available in MATLAB Test to support equivalence testing when transforming MATLAB code to C, C++, Python, Java, or .NET. Equivalence testing is particularly important in safety critical environments such as medical devices or ADAS. We also looked at how inheritance can be leveraged to reuse existing MATLAB tests for equivalence testing.
In the next and final post in the series, I will explore dependency-based test selection.

Code listing

shortestPath

function pathLength = shortestPath(adjMatrix, startIdx, endIdx) %#codegen
% SHORTEST_PATH - Finds length of shortest path between nodes in a graph
%
% OUT = SHORTEST_PATH(ADJMTX, STARTIDX, ENDIDX) Takes a graph represented by
% its adjacency matrix ADJMTX along with two node STARTIDX, ENDIDX as
% inputs and returns a integer containing the length of the shortest path
% from STARTIDX to ENDIDX in the graph.
 
% Copyright 2021 The MathWorks, Inc.
%% Validy testing on the inputs
% This code should never throw an error and instead should return
% error codes for invlid inputs.
ErrorCode = 0;
pathLength = -1;
 
% Check the validity of the adjacency matrix
if (~isAdjMatrixValid(adjMatrix))
ErrorCode = -9;
end
 
% Check the validity of the startIdx
if ~isNodeValid(startIdx)
ErrorCode = -19;
end
 
% Check the validity of the endIdx
if ~isNodeValid(endIdx)
ErrorCode = -29;
end
 
[nodeCnt, n] = size(adjMatrix);
 
% Start or end node is too large
if startIdx > nodeCnt || endIdx > nodeCnt
ErrorCode = -99;
end
 
% Start or end node is too small
if startIdx < 1 || endIdx < 1
ErrorCode = -199;
end
 
if (ErrorCode<0)
pathLength = ErrorCode;
return;
end
 
%% Self-loop path is always 0
if startIdx == endIdx
pathLength = 0;
return;
end
%% Dijkstra's Algorithm
% Dijkstra's Algorithm is used to iteratively explore the graph breadth
% first and update the shortest path until we reach the end node.
 
% Initialization
max = realmax;
visited = false(1, nodeCnt);
 
% The distance vector maintains the current known shortest path from
% the start node to every node. As nodes are processed one by one
% the distance vestor is updated
distance = repmat(max, 1, nodeCnt);
distance(startIdx) = 0;
 
for iterStep = 1:nodeCnt
% At each iteration identify the current node to process that
% is not yet visited and has the smallest distance from the start.
% This breadth first search ensures that we will always reach nodes
% by the shortest possible path.
min = max;
nodeIdx = -1;
for v = 1:n
if ~visited(v) && distance(v) <= min
min = distance(v);
nodeIdx = v;
end
end
 
% Stop iterating when the current distance is maximum because
% this indicates no remaining nodes are reachable
if (min==max)
return;
end
% Mark the current node visited and check if this is end index
visited(nodeIdx) = true;
if nodeIdx == endIdx
pathLength = distance(nodeIdx);
 
if (pathLength==realmax)
% No path exists so set distance to -1;
pathLength = -1;
end
return;
end
% Update distances of unvisited nodes adjacent to the current node
for v = 1:nodeCnt
if(~visited(v) && adjMatrix(nodeIdx, v) ~= 0 && distance(nodeIdx) ~= max)
distVal = distance(nodeIdx) + adjMatrix(nodeIdx, v);
if distVal < distance(v)
distance(v) = distVal;
end
end
end
end
end
 
function out = isNodeValid(node)
% For full coverage we need to create negative tests that make each
% successively make each validity condition false
if(isscalar(node) && isnumeric(node) && ~isinf(node) && floor(node) == node)
out = true;
else
out = false;
end
end
 
function out = isAdjMatrixValid(adjMatrix)
% need to be a square matrix with only 0, 1, or realmax entries.
[m, n] = size(adjMatrix);
 
% For full coverage we need to create negative tests that make each
% successively make each validity condition false
if (m==n) && isempty(find((adjMatrix ~= 0) & (adjMatrix ~= 1), 1))
out = true;
else
out = false;
end
end

tMATLABTests

classdef tMATLABTests < tCommon
% Copyright 2021 The MathWorks, Inc.
 
methods (Test,TestTags="InputTests")
 
function check_invalid_start_1(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = -1;
endIdx = 2;
expOut = -199;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid start index, idx<1');
end
 
function check_invalid_start_2(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 12;
endIdx = 2;
expOut = -99;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid start index, idx>NodeCnt');
end
 
function check_invalid_end_1(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 1;
endIdx = -3;
expOut = -199;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid end index, idx<1');
end
 
function check_invalid_end_2(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 1;
endIdx = 12;
expOut = -99;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Invalid end index, idx>NodeCnt');
end
end
 
methods(Test,TestTags="EdgelessTests")
 
function check_edgeless_graph(testCase)
adjMatrix = zeros(20,20);
startIdx = 1;
endIdx = 18;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Edgeless graph');
end
 
function check_edgeless_start(testCase)
adjMatrix = testCase.graph_some_nodes_edgeless();
startIdx = 1;
endIdx = 4;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Edgeless graph');
end
 
function check_edgeless_end(testCase)
adjMatrix = testCase.graph_some_nodes_edgeless();
startIdx = 3;
endIdx = 1;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Edgeless graph');
end
function check_edgeless_graph_self_loop(testCase)
adjMatrix = zeros(20,20);
startIdx = 16;
endIdx = 16;
expOut = 0;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Self loop in edgeless graph');
end
 
end
 
methods (Test)
 
function check_longest_path(testCase)
adjMatrix = testCase.graph_straight_seq();
startIdx = 1;
endIdx = 4;
expOut = 3;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Longest theoretic path');
end
 
function check_unity_path(testCase)
adjMatrix = testCase.graph_all_edge();
startIdx = 2;
endIdx = 3;
expOut = 1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Path length 1');
end
 
function check_non_unique(testCase)
adjMatrix = testCase.graph_square();
startIdx = 4;
endIdx = 2;
expOut = 2;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Non-unique path');
end
 
function check_no_path(testCase)
adjMatrix = testCase.graph_disconnected_components();
startIdx = 1;
endIdx = 5;
expOut = -1;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'No path');
end
 
function check_start_end_same(testCase)
adjMatrix = testCase.graph_all_edge();
startIdx = 3;
endIdx = 3;
expOut = 0;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Start and end index same');
end
 
function check_invalid_idx_empty_adj(testCase)
adjMatrix = [];
startIdx = 1;
endIdx = 1;
expOut = -99;
verifyPathLength(testCase, adjMatrix, startIdx, endIdx, expOut, 'Degenerate empty graph');
end
end
 
methods (Static)
% Utility functions to create common adjacency graph matrices
function adj = graph_straight_seq()
% Create the graph:
% 1---2---3---4
 
adj = [0 1 0 0; ...
1 0 1 0; ...
0 1 0 1; ...
0 0 1 0];
end
 
function adj = graph_square()
% Create the graph:
% 1---2
% | |
% 4---3
 
adj = [0 1 0 1; ...
1 0 1 0; ...
0 1 0 1; ...
1 0 1 0];
end
 
function adj = graph_all_edge()
% Create the graph:
% 1---2
% |\ /|
% |/ \|
% 4---3
 
adj = [0 1 1 1; ...
1 0 1 1; ...
1 1 0 1; ...
1 1 1 0];
end
 
function adj = graph_disconnected_components()
% Create the graph:
% 2 5
% / \ / \
% 1---3 4---6
 
adj = [0 1 1 0 0 0; ...
1 0 1 0 0 0; ...
1 1 0 0 0 0; ...
0 0 0 0 1 1; ...
0 0 0 1 0 1; ...
0 0 0 1 1 0];
end
 
function adj = graph_some_nodes_edgeless()
% Create the graph:
% 2
% / \
% 4---3
%
% Nodes 1, 5, 6 are edgeless
 
adj = [0 0 0 0 0 0; ...
0 0 1 1 0 0; ...
0 1 0 1 0 0; ...
0 1 1 0 0 0; ...
0 0 0 0 0 0; ...
0 0 0 0 0 0];
end
 
end
 
methods
 
function verifyPathLength(testCase,adjacency,startIdx,endIdx,expectedResult,debugTxt)
 
% Execute the design
actualResult = shortestPath(adjacency, startIdx, endIdx);
 
% Confirm the expected
msgTxt = sprintf('Failure context: %s', debugTxt);
testCase.verifyEqual(actualResult, expectedResult, msgTxt);
end
 
end
 
end
]]>
https://blogs.mathworks.com/developer/2023/10/26/equivalence-testing/feed/ 0
Measuring and Monitoring Code Quality https://blogs.mathworks.com/developer/2023/09/18/measuring-and-monitoring-code-quality/?s_tid=feedtopost https://blogs.mathworks.com/developer/2023/09/18/measuring-and-monitoring-code-quality/#comments Mon, 18 Sep 2023 07:34:59 +0000 https://blogs.mathworks.com/developer/?p=3134

This is the third instalment in my series on MATLAB Test. My previous posts introduced MATLAB Test and the Test Manager and the use of advanced coverage metrics. Today I’m going to look at the Code... read more >>

]]>
This is the third instalment in my series on MATLAB Test. My previous posts introduced MATLAB Test and the Test Manager and the use of advanced coverage metrics.
Today I’m going to look at the Code Quality Dashboard which centralises a variety of metrics for your project and allows you to answer questions such as:
  • Does my code have any issues identified by the MATLAB Code Analyzer?
  • Does my project meet required coverage thresholds?
  • Do I have failing tests and if so, how many?
  • Are all my requirements linked to tests and have those tests been run?
To explore this more, I’m going to use the example project that comes with MATLAB Test which you can access by running:
openExample('matlabtest/CompleteTheVerificationOfAMATLABAlgorithmExample')
As always, the documentation provides additional detail to what I cover here.
With your project open, open the Code Quality Dashboard by clicking the icon in the tools gallery:
codeQuality_toolstrip.png
Open the Code Quality Dashboard from the tools gallery.
The first time the dashboard opens it will look like this:
codeQuality_initialState.png
Initial state of Code Quality Dashboard.
There are 4 sections in the dashboard that bring together metrics from the MATLAB Code Analyzer, code coverage, tests, and requirements.
We’re currently missing test data as we haven’t run them yet – the tests are not run automatically as this could take a significant amount of time in a large project. You can run the tests from the Test Manager or you can use the button in the dashboard – click the 3 dots menu to the right of “Coverage” or “Tests” and select “Run Tests and Refresh Data”.
codeQuality_runTests.png
Run tests from the Code Quality Dashboard.
The dashboard now looks like this:
codeQuality_postRun.png
Code Quality Dashboard after tests have been run.
You can use this information to assess the current state of your project. For example:
  • We have 1 suggestion identified by the Code Analyzer that we should review.
  • Our statement coverage is 88%. This might mean that our code meets our company specified quality threshold.
  • However, our MC/DC is low so there are definite gaps in our testing despite the high statement coverage.
  • Some of our tests are failing so we’ll need to fix those.
  • 17.4% of our requirements are not linked to tests in our project. Either they need to be linked to existing tests or we will need to write new tests.
To view more detail on any of the sections in the Code Quality Dashboard, click on a given section and a dedicated report will be launched. For example, clicking on the Code Analyzer section will launch this report:
codeQuality_codeAnalyzer_pre.png
Code Analyzer report launched from the Code Quality Dashboard.
I’ll click the “Fix All” button to fix the issue. At this point a banner appears in the Code Quality Dashboard notifying us that files in the project have changed so the dashboard needs refreshing.
codeQuality_codeAnalyzer_post.png
Code Analyzer issue fixed and Code Quality Dashboard indicating that a refresh is required.
Once we click the “Update Artifacts” button in the banner, we then need to click a subsequent “Refresh All” button to refresh the metrics, and our dashboard is up to date again.
codeQuality_noIssues.png
No more Code Analyzer issues following the update.
In summary, the Code Quality Dashboard brings together key metrics about your project in one place, allowing you to make informed decisions. It serves as the centre point of a “find and fix” workflow, allowing you to jump to more detailed views when you need them, and then coming back to the dashboard to view how your changes have affected the overall project status.
In the next post I’ll look at equivalence testing.
]]>
https://blogs.mathworks.com/developer/2023/09/18/measuring-and-monitoring-code-quality/feed/ 3
We’ve got you covered! https://blogs.mathworks.com/developer/2023/08/30/weve-got-you-covered/?s_tid=feedtopost https://blogs.mathworks.com/developer/2023/08/30/weve-got-you-covered/#respond Wed, 30 Aug 2023 08:04:22 +0000 https://blogs.mathworks.com/developer/?p=3098

In the previous blog post of this series, I gave an overview of MATLAB Test and went into the details of the Test Manager. Today’s topic is code coverage. Code coverage is a measure of “how much”... read more >>

]]>
In the previous blog post of this series, I gave an overview of MATLAB Test and went into the details of the Test Manager. Today’s topic is code coverage.
Code coverage is a measure of “how much” of your code has been executed. By measuring and examining code coverage, you can gain insights into which parts of your code have not been fully tested, and therefore where you should direct your efforts in writing more test cases. Similarly, if the code coverage report is saying that some parts of your code are not being executed and you know you’re satisfying your requirements, chances are you can delete this dead logic.
There are a variety of ways of measuring code coverage. Base MATLAB provides function and statement coverage whilst MATLAB Test adds to that by providing decision, condition, and modified condition/decision coverage metrics. Here’s a summary of the different coverage metrics:
coverageSummary.png

What level of coverage do I need?

Given the above coverage types, which one(s) should you use and what level of coverage do you need to achieve? The answer to is that it’s largely your choice based on your project’s requirements and constraints.
Statement coverage is the starting point for general purpose applications. However, you may have constraints imposed by regulations if you’re working in a safety critical environment such as medical (IEC 62304), aerospace (DO-178), or automotive (ISO 26262).
One of my colleagues likes to use the analogy of choosing an insurance policy – if the risk is low and the impact of anything going wrong is also low, you’ll likely choose the cheapest, most basic insurance. However, when the risk and consequences of something going wrong are high, you’ll want a comprehensive policy which comes at additional cost. In the world of testing, that more comprehensive insurance corresponds to achieving increasing levels of coverage with more advanced metrics. The cost is your time to write the additional tests.

Collecting coverage

How do we collect coverage? There are three ways!

Test Browser

If you’re using the Test Browser (MATLAB R2023a onwards), click the coverage button, check “Enable Coverage reporting”, and add the files or folders of code that you want to measure coverage for. Then run your tests and an easy-to-read HTML coverage report will open.
testBrowser.png
Collecting statement coverage via the Test Browser.
Note that if you want to record decision, condition, or MC/DC, you will still need a MATLAB Test licence.

Test Manager

If you’re using the Test Manager (MATLAB Test, R2023a onwards), click the coverage button and check “Enable Coverage” in the pop-out menu.
testManager.png
Collecting statement coverage via the Test Manager.
Run the tests and then view the report by clicking the adjacent menu button:
testManager_viewReport.png
Opening the coverage report from the Test Manager.

Command Window

Finally, if you’re running your tests programmatically using runtests, you can use the ReportCoverageFor option to specify the files, folders, and packages that you want to record coverage for.
(This will give you the default statement and function coverage. If you want to use the more advanced coverage metrics, you’ll need to use the longhand method to create your own TestRunner and add the CodeCoveragePlugin here you can specify the coverage level.)
Once your tests have finished running, a hyperlink will appear in the Command Window taking you to the coverage report.

Interactive HTML coverage report

The code coverage report looks like this:
coverageReportOverview.png
Example code coverage report.
The top section provides a summary of all the code you have measured coverage for, for all the different coverage types you have selected. Use the “Currently viewing” drop-down menu to choose which type of coverage is being shown in the rest of the report.
The middle section provides a breakdown on a per file basis. Click a line in the table to choose which file is displayed in the bottom section.
The bottom section provides coverage information on a line-by-line basis for the selected coverage type and the selected file. We’ll explore the details of this next.

Types of coverage

In the following, I expand on what’s covered in the documentation for types of coverage available with MATLAB Test. This is a great reference so keep it to hand!

Function coverage

Function and statement coverage are the default coverage metrics that MATLAB records. Function coverage indicates whether each function in your code gets executed at least once. (This is not to be confused with functionalcoverage which relates to the overall functionality being correct. This and requirements-based testing are topics for another day!)
In the following example, we have three functions – the top-level function myFunction and two nested functions fcnOne and fcnTwo. I’ve recorded coverage for when I call myFunction(0,0,0).
functionCoverage.png
Example function coverage report with 67% coverage.
This executes myFunction itself and fcnTwo but not fcnOne. We therefore have 3 functions in total of which 2 have been executed, giving 2/3 = 67% function coverage. There are clearly holes on our testing so let’s move onto statement coverage.

Statement coverage

Statements are small chunks of code that MATLAB executes that are separated by a comma, semicolon, or newline character. To achieve 100% statement coverage, each statement much be executed at least once.
The following snippet from the coverage report shows statement coverage for when I call myFunction(0,0,0).
statementCoverage.png
Example statement coverage report with 60% coverage.
The branch is not executed so lines 4 and 12 are not covered and show in red. We have 5 statements in total, 3 of which have been executed, so the statement coverage is 3/5 = 60%. By inspection we can see that the ifstatement is only ever false; we’re missing the case of it being true. Let’s look at how we can view this more formally.

Decision coverage

Decisions are expressions that alter the execution flow of the code in conjunction with the MATLAB keywords if, elseif, switch, for, or while. Decision coverage measures the extent to which all these decisions have been tested to be both true and false. It therefore provides you with a view of code branches with missing tests.
(In addition, the short-circuit operators || and && are also decisions since the value on the left-hand side of the expression determines whether the right-hand side is evaluated. However, to be consistent with Simulink Coverage, these expressions (“non-branch decisions”) are excluded from the metrics when recording decision coverage but are included when recording MC/DC.)
In the previous example we achieved full statement coverage for the if decision on line 3. In the following example, I’ve once again tested my code by calling myFunction(0,0,0) and have measured decision coverage.
decisionCoverage_partial.png
Example decision coverage report with 50% coverage.
We now see that line 3 is only partially covered (orange) as it only evaluates to false. There is 1 decision with 2 possible values, so 2 combinations in total. We hit 1 of them so that’s 1/2 = 50% decision coverage.
I’ll now add in an additional test point so that I’m now calling myFunction(0,0,0) and myFunction(1,0,0). This achieves full decision coverage:
decisionCoverage_full.png
Example decision coverage report with 100% coverage.
You may also have noticed that the coverage recording level is cumulative – recording decision coverage also includes statement and function coverage.

Condition coverage

Decisions can be composed of two or more logical expressions separated by a short-circuit operator (|| or &&) – these are called conditions. Here is an example with 1 decision and 2 conditions:
if (x > -3) || (x == -5)
The following contains one decision but no conditions as there are no expressions separated by a short-circuit operator:
if x == 0
Below is the condition coverage report for our test code when I call myFunction(0,0,0) and myFunction(1,0,0). Remember how we achieved full coverage previously? Not anymore! Condition coverage reveals yet more holes in our testing.
conditionCoverage_partial.png
Example condition coverage report with 50% coverage.
The x condition is covered (green) as it takes values of both true and false. The y condition is partially covered (orange) as it only evaluates to false. The z condition is not covered (red) as it is never executed – why? y is always false and so the && short circuits and doesn’t bother evaluating z.
We have 3 conditions, each of which can take two values (true or false), so there are 6 combinations in total. We’re hitting 3/6 = 50% coverage.
Let’s modify our tests to fill in the missing coverage:
  • myFunction(0,0,0)
  • myFunction(1,0,0)
  • myFunction(0,1,0)
  • myFunction(0,1,1)
conditionCoverage_full.png
Example condition coverage report with 100% coverage.

MC/DC

MC/DC stands for Modified condition/decision coverage. It identifies how tests independently exercise conditions within decisions. To achieve full MC/DC, both of the following must be achieved:
  1. All conditions within decisions are evaluated to both true and false.
  2. Every condition within a decision independently affects the outcome of the decision.
MC/DC is both stricter than condition coverage whilst allowing full coverage to be achieved with fewer tests than would be required to test all combinations of conditions exhaustively. Consider our if statement of 1 decision with 3 conditions:
if x || (y && z)
All the possible combinations of x, y, and z and the corresponding result are listed in the table below.
mcdcCombinations.png
Looking at this table, we can identify test combinations that satisfy the requirements of MC/DC. Each condition must evaluate to true and false, and independently affect the result. For example:
  • x – 2 and 6.
  • y – 2 and 4.
  • z – 3 and 4.
We can therefore achieve full MC/DC with test combinations 2, 3, 4, and 6. Here’s the corresponding coverage report when I run these test combinations:
mcdc_full.png
Example MC/DC report with 100% coverage.
Under the “Test Pairs” heading, you can see the various combinations of input values used to achieve 100% MC/DC. “T” denotes true, “F” false, and “x” is don’t care.
mcdc_explanation.png
By contrast, here’s an example of incomplete MC/DC when I only ran test combinations 2 & 6:
mcdc_partial.png
Example MC/DC report with 33% coverage.
As per the bulleted list above, test combinations 2 & 6 provide full MC/DC for x, and for y to independently produce a result of false. Note how the framework has used test condition 2 (FFx) in the first test pair as opposed to test condition 3 previously.
We have 1 out of our 3 conditions achieving MC/DC, so the overall coverage is 1/3 = 33%. By comparison, we have tested 3/6 conditions so have achieved 50% condition coverage. This demonstrates how MC/DC is a stricter metric than condition coverage.

Summary

We’ve looked at the following types of coverage:
  • Function (MATLAB)
  • Statement (MATLAB)
  • Decision (MATLAB Test)
  • Condition (MATLAB Test)
  • Modified Condition / Decision Coverage (MCDC) (MATLAB Test).
These metrics can be used to guide your testing and therefore help you to produce high quality software whether that’s for everyday desktop use or for safety critical applications. Remember, getting 100% coverage is not the goal – the goal is better quality software consistent with the time budget of the project, the needs of the end deliverable, and where it will be deployed.
In the next post in this series, we’ll look at MATLAB Test’s Code Quality dashboard.
]]>
https://blogs.mathworks.com/developer/2023/08/30/weve-got-you-covered/feed/ 0
Introducing MATLAB Test! https://blogs.mathworks.com/developer/2023/08/16/introducing-matlab-test/?s_tid=feedtopost https://blogs.mathworks.com/developer/2023/08/16/introducing-matlab-test/#respond Wed, 16 Aug 2023 14:51:55 +0000 https://blogs.mathworks.com/developer/?p=3044

MATLAB R2023a was released back in March, and with it was a new product MATLAB Test. In a series of blog posts starting today, I’m going to show you how MATLAB Test can help you improve the quality... read more >>

]]>
MATLAB R2023a was released back in March, and with it was a new product MATLAB Test. In a series of blog posts starting today, I’m going to show you how MATLAB Test can help you improve the quality of your code whether you’re developing a MATLAB Toolbox to share with your colleagues or writing code that will be deployed into a safety critical environment such as for medical devices.

Overview

There are five key features in the first release of MATLAB Test which I’m going to cover. They will be extended and added to over the coming releases:
  1. Test Manager – manage your tests and results (today’s topic).
  2. Extended coverage – measure condition, decision, and modified condition and decision coverage (MCDC) to help ensure your code is fully tested and meets regulatory requirements.
  3. Code quality dashboard – see an overview of your project’s quality status such as coverage, Code Analyzer warnings, and requirements links.
  4. Equivalence testing with MATLAB Coder and MATLAB Compiler SDK – easily compare the behaviour of generated C or C++ code, Java or Python packages, or .NET assemblies to that of desktop MATLAB.
  5. Dependency based test selection – execute only the tests in your suite that are impacting by changes in your code base.
In addition, tool qualification artifacts are available for MATLAB Test in our IEC Certification Kit making MATLAB Test suitable for use in the development of high integrity software.

What’s not changing?

Don’t worry! The MATLAB unit testing (including measuring statement coverage), mocking, app testing, and performance testing frameworks are not changing and remain in base MATLAB.
In addition, R2023a sees the introduction to base MATLAB of the Test Browser, a simple interface to run tests and browse results.
testBrowser.png
Test Browser
The Test Browser will launch automatically whenever you run tests from a test file using the “Run Tests” (F5) or “Run Current Test” (Ctrl + Enter) buttons. If you run your tests from the command line (runtests(“tDemoSmokeTest”)), the Test Browser will not launch and you will instead see the classic Command Window output.

Test Manager

The Test Manager allows you to create and manage test suites, view and search results, view requirements links, manage coverage settings, and persist results between MATLAB sessions.
So what’s the difference between the Test Browser and the Test Manager? The Test Browser is a simple viewer for iteratively developing code and tests within a MATLAB session. The Test Manager allows you to manage tests at scale, link to requirements, and persist results between sessions.
To use Test Manager, you will need to be using a Project. (For this blog post, I’m using the shipping shortest path example.) The Test Manager needs Projects for dependency information and for persisting test results. You can launch the Test Manager from the Project Toolstrip under the Apps section:
testManagerShortcut.png
Launch the Test Manager from the Project toolstrip.
When you first launch the Test Manager, it will automatically find all tests in your project. You can add additional columns to the table view (click the green plus icon to the top right of the table) to view test tags and linked requirements (requires the Requirements Toolbox).
testManagerWithRequirements.png
Test Manager with “Tags” and “Requirements” columns added.
Once you have run the tests (hit the green play button!), you can view your tests. You can filter the results by their pass/fail/incomplete/not run status, or search based on some text. Here I’ve searched for “input” which finds all the tests that have the tag “InputTests”.
testManagerSearch.png
Test results filtered to show those that include “input” in their metadata.

Collecting coverage information

Coverage can be configured using the option in the menu bar:
coverageSettings.png
Coverage settings.
Whilst the definition of statement coverage is fairly obvious, the differences between the others can be subtle. We’ve added a page in the documentation to explain the meaning of each type of coverage as they apply to MATLAB source code. I will also explore it more in a future blog post.
The coverage recording level in MATLAB Test is cumulative, i.e. MC/DC also includes Statement, Decision, and Condition.
Coverage results can be viewed by clicking the adjacent menu item after all tests have been run:
coverageReportButton.png
Access the coverage report.
(As of R2023a, coverage information is only available when all tests in the current project are run; coverage is not recorded when running a custom test suite. This restriction will be removed in a future release.)
Coverage is recorded for all files in your project that have the classification of “Design”. By default, all files in a project will be classified as “Design” except those that use the unit testing framework which are classified as “Test”. If you want to remove a file from the coverage report, change the classification to something else.
projectLabels.png
Change a file’s classification to something other than “Design” to exclude it from coverage recording.

Creating custom test suites

The Test Manager provides a graphical interface for creating custom test suites by using this menu option:
createTestSuite.png
Create a custom test suite.
Here, I’ve created a new test suite to select all tests that have the tag “InputTests”:
testSuiteManager.png
Creating a new test suite for all tests in the project with the tag “InputTests”.
I can then select my suite from the dropdown menu and just run that subset:
testManagerTestSuite.png
Viewing tests that are part of the custom suite.

Session persistence

Finally, since the Test Manager persists the results of tests from session to session, I can close MATLAB, open it up again later, reopen the Test Manager, and the results of my last test run will still be there.

Summary

MATLAB Test’s Test Manager helps you to manage your test suite, coverage, and requirements as you scale up your testing activities.
In the next post, I’ll take a look at the new coverage options in MATLAB Test.
]]>
https://blogs.mathworks.com/developer/2023/08/16/introducing-matlab-test/feed/ 0
We’ve All Got Issues https://blogs.mathworks.com/developer/2023/03/15/static-analysis-code-checking-and-linting-with-codeissues/?s_tid=feedtopost https://blogs.mathworks.com/developer/2023/03/15/static-analysis-code-checking-and-linting-with-codeissues/#respond Wed, 15 Mar 2023 19:54:49 +0000 https://blogs.mathworks.com/developer/?p=2964

Who among us doesn't have issues, amirite? Let's just take a moment and acknowledge this fact and I think we can always be a bit more honest and understanding of all of our unique issues and the... read more >>

]]>

Who among us doesn't have issues, amirite? Let's just take a moment and acknowledge this fact and I think we can always be a bit more honest and understanding of all of our unique issues and the various idiosyncrasies we exhibit. While we can all offer understanding and grace to each other, some of the issues we face can be important to address quickly, and some we can perhaps choose to work on when the time is right.

...and so it is with our code. Issues, bugs, and other sub-optimal constructs crop up all the time in any serious codebase. Many of you are already aware of the features of the Code Analyzer in MATLAB that do a great job of alerting you to your issues directly in the MATLAB editor. However, sometimes issues still can creep into a code base through a number of subtle means, even including simply failing to act when a helpful editor is trying to tell me I have a problem in my code.

However, in R2022b there was a great improvement to the programmatic interface for identifying and addressing with these code issues. In fact, this new interface is itself a new function called codeIssues!

This new API for the code analyzer is in a word - powerful. Let's explore how you can now use this to build quality into your project. Starting with another look at our standard mass-spring-damper mini-codebase. Most recently we talked about this when describing the new MATLAB build tool. This code base is pretty simple. It includes a simulator, a function to return some design constants (spring constant, damping coefficient, etc.), and some tests.

To get started on a codebase like this, just call codeIssues on the folder you want to analyze:

You can see that with just a simple call to codeIssues you can quickly get an overview of all the details a static analyzer dreams of. You can easily dig into the files that were analyzed, the configuration, and the very handy table of the issues found, as well as any issues that have been suppressed through suppression pragmas in the editor. If you are in MATLAB you can even click on each issue to get right to it in the MATLAB editor where it can be fixed or suppressed if needed.

Now with this beautiful API at our fingertips, and with the build tool to boot, we can lock down our code in a much more robust, automated way. We can start roughly where we left off at the end of our build tool post with the following buildfile with a mex and a test task:

function plan = buildfile
plan = buildplan(localfunctions);

plan("test").Dependencies = "mex";
plan.DefaultTasks = "test";
end

function mexTask(~)
% Compile mex files
mex mex/convec.c -outdir toolbox/;
end

function testTask(~)
% Run the unit tests
results = runtests;
disp(results);
assertSuccess(results);
end


Let's go ahead and add a "codeIssues" task to this build by creating a new local function called codeIssuesTask:

function codeIssuesTask(~)
% Get all the issues under toolbox
allIssues = codeIssues("toolbox");

% Assert that no errors creep into the codebase
errorIdx = allIssues.Issues.Severity == "error";
errors = allIssues.Issues(errorIdx,:);
otherIssues = allIssues.Issues(~errorIdx,:);
if ~isempty(errors)
    disp("Found critical errors in code:");
    disp(errors);
else
    disp("No critical errors found.");
end

% Display all the other issues
if ~isempty(otherIssues)
    disp("Other Issues:")
    disp(otherIssues);
else
    disp("No other issues found either. (wow, good for you!)")
end

assert(isempty(errors));
end

This is quite simple, we just want to find all the issues under the "toolbox" folder and throw an assertion error if any of them are of Severity "error". This is just about the quickest win you can apply to a code base to build quality into it. This can find syntax and other errors statically, without even writing or running a single test. There really is no reason at all we shouldn't apply this task to every project. It costs virtually nothing and can be remarkably efficient at finding bugs. On that note, let's add it as a default task in our buildfile:

plan.DefaultTasks = ["codeIssues" "test"];

...and with that we now have this check built right into our standard development process. To show this let's first put a file with an error into the code base and then we can call the build tool:

function syntaxError

disp("Forgot the closing parentheses!"

copyfile .changes/syntaxError.m toolbox/syntaxError.m
try
    buildtool
catch ex
    disp(ex.getReport("basic"));
end
** Starting codeIssues
Failed! Found critical errors in code:
       Location        Severity                                     Description                                      CheckID    LineStart    LineEnd    ColumnStart    ColumnEnd                                             FullFilename        
    _______________    ________    ______________________________________________________________________________    _______    _________    _______    ___________    _________    _______________________________________________________________________________________________

    "syntaxError.m"     error      "A '(' might be missing a closing ')', causing invalid syntax at end of line."    EOLPAR         3           3            5             5        "/Users/acampbel/Library/CloudStorage/OneDrive-MathWorks/repos/msd_blog2/toolbox/syntaxError.m"

Other Issues:
             Location             Severity                     Description                     CheckID    LineStart    LineEnd    ColumnStart    ColumnEnd                                                   FullFilename             
    __________________________    ________    _____________________________________________    _______    _________    _______    ___________    _________    __________________________________________________________________________________________________________

    "springMassDamperDesign.m"    warning     "Value assigned to variable might be unused."     NASGU         4           4            3             3        "/Users/acampbel/Library/CloudStorage/OneDrive-MathWorks/repos/msd_blog2/toolbox/springMassDamperDesign.m"
    "springMassDamperDesign.m"    warning     "Value assigned to variable might be unused."     NASGU         6           6            3             3        "/Users/acampbel/Library/CloudStorage/OneDrive-MathWorks/repos/msd_blog2/toolbox/springMassDamperDesign.m"

## -----------------------------------------------------------------------------
## Error using assert
## Assertion failed.
## 
## Error in buildfile>codeIssuesTask (line 67)
## assert(isempty(errors));
## -----------------------------------------------------------------------------
** Failed codeIssues

Error using buildtool
Build failed.

The build tool has done its part in stopping our development process in its tracks when it sees we have some important syntax errors to address.

Let's remove that bunk error so we can see our "codeIssues" task complete successfully. When we do this we also successfully execute the other "mex" and "test" tasks to complete the whole workflow.

delete toolbox/syntaxError.m
buildtool
** Starting codeIssues
No critical errors found.
Other Issues:
             Location             Severity                     Description                     CheckID    LineStart    LineEnd    ColumnStart    ColumnEnd                                                   FullFilename             
    __________________________    ________    _____________________________________________    _______    _________    _______    ___________    _________    __________________________________________________________________________________________________________

    "springMassDamperDesign.m"    warning     "Value assigned to variable might be unused."     NASGU         4           4            3             3        "/Users/acampbel/Library/CloudStorage/OneDrive-MathWorks/repos/msd_blog2/toolbox/springMassDamperDesign.m"
    "springMassDamperDesign.m"    warning     "Value assigned to variable might be unused."     NASGU         6           6            3             3        "/Users/acampbel/Library/CloudStorage/OneDrive-MathWorks/repos/msd_blog2/toolbox/springMassDamperDesign.m"

** Finished codeIssues

** Starting mex
Building with 'Xcode with Clang'.
MEX completed successfully.
** Finished mex

** Starting test
Setting up ProjectFixture
Done setting up ProjectFixture: Project 'msd' is already loaded. Set up is not required.
__________

Running convecTest
.
Done convecTest
__________

Running designTest
...
Done designTest
__________

Tearing down ProjectFixture
Done tearing down ProjectFixture: Teardown is not required.
__________

  1×4 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   4 Passed, 0 Failed, 0 Incomplete.
   0.22349 seconds testing time.

** Finished test

One last thing

Now we have you all setup nice and cozy with the protection that static analysis gives you. However, while we fail on static analysis errors, I am still uncomfortable with how easy it is to continue to add constructs to my code that result in static analysis warnings, which often point to real problems in your program. We could also fail the build on warnings if we'd like, but I didn't want to start with that idea out of the gate.

It is pretty clear that we want this protection with full-on-bonafide errors, which are almost always bugs. We run into a problem though when a code base already has an inventory of warnings. It would be fantastic to go through that inventory and fix all of those warnings as well. In fact, the new code analysis tooling makes that very easy in many cases! However, you may not be up for this right now. Your code base may be large, and you may want or need to invest a bit more time into this activity. So our first crack at this failed the build only for issues with an "error" Severity.

However, if you know me you know I like to sneak one last thing in. What if we accepted all of our current warnings in the codebase, but wanted to lock down our code base such that we are protected from introducing new warnings? To me this sounds like a great idea. We can then ratchet down our warnings by preventing inflow of new warnings and can remove existing warnings over time through interacting with the codebase. How can we do this? We can leverage the power of the codeIssues programmatic API!

We can do this by capturing and saving our existing warnings to a baseline of known issues. As MATLAB tables, theses issues are in a nice representation to save in a *.csv or *.xlsx file. Saving them in this format makes it really easy to tweak them, open them outside of MATLAB, or even remove issues that have been fixed.

To do this we just need to make a couple tweaks to the issues table. We need to overwrite the Location variable with relative paths to the files, remove the FullFilename variable, and make a quick datatype tweak to allow for nice CSV'ing. The relative filename adjustment is important because we want to be able to compare these results across different machines and the full path is likely to differ across environments. Such environments include the desktops of individual engineers as well as different build agents in a CI system.

That function looks as follows:

function theTable = preprocessIssues(theTable)
% Make an issues table conducive for baselining via a few small tweaks

% Overwrite the location field with relative paths, and remove absolute paths
basePath = string(pwd) + filesep;
theTable.Location = erase(theTable.FullFilename, basePath);
theTable.Properties.VariableNames{"Location"} = 'RelativeFilename';
theTable.FullFilename = [];

% Convert the Severity to categorical, which serializes nicely to string
theTable.Severity = categorical(theTable.Severity);

end

...and now with this function we can create a new task in the buildfile to generate a new baseline:

function captureWarningsBaselineTask(~)
% Captures the current codeIssues warnings and creates a baseline csv file
allIssues = codeIssues("toolbox");
warningIdx = allIssues.Issues.Severity == "warning";
warnings = allIssues.Issues(warningIdx,:);

warnings = preprocessIssues(warnings);
if ~isempty(warnings)
    disp("Saving a new ""knownIssues.csv"" baseline file for " + height(warnings) + " code warnings")
    writetable(warnings, "knownIssues.csv");
else
    disp("No warnings to create a baseline for")
end

end

Let's do it!

buildtool captureWarningsBaseline
** Starting captureWarningsBaseline
Saving a new "knownIssues.csv" baseline file for 2 code warnings
** Finished captureWarningsBaseline

Great I now see the csv files. We can take a peek:

type knownIssues.csv
RelativeFilename,Severity,Description,CheckID,LineStart,LineEnd,ColumnStart,ColumnEnd
toolbox/springMassDamperDesign.m,warning,Value assigned to variable might be unused.,NASGU,4,4,3,3
toolbox/springMassDamperDesign.m,warning,Value assigned to variable might be unused.,NASGU,6,6,3,3

Beautiful. In this case we just have two minor warnings that I don't want to look into quite yet. However, now we can adjust the "codeIssues" task to prevent me from introducing anything new:

function codeIssuesTask(~)
% Get all the issues under toolbox
allIssues = codeIssues("toolbox");

% Assert that no errors creep into the codebase
errorIdx = allIssues.Issues.Severity == "error";
errors = allIssues.Issues(errorIdx,:);
otherIssues = allIssues.Issues(~errorIdx,:);
if ~isempty(errors)
    disp("Failed! Found critical errors in code:");
    disp(errors);
else
    disp("No critical errors found.");
end

% Load the known warnings baseline
newWarnings = [];
if isfile("knownIssues.csv")
    otherIssues = preprocessIssues(otherIssues);
    
    % Load the baseline file
    opts = detectImportOptions("knownIssues.csv");
    types = varfun(@class, otherIssues,"OutputFormat","cell");
    opts.VariableTypes = types;
    knownIssues = readtable("knownIssues.csv",opts);

    % Find the new warnings by subtracting the known issues in the baseline
    otherIssues = setdiff(otherIssues, knownIssues);
    newWarningIdx = otherIssues.Severity == "warning";
    newWarnings = otherIssues(newWarningIdx,:);
    if ~isempty(newWarnings)
        disp("Failed! Found new warnings in code:");
        disp(newWarnings);
    else
        disp("No new warnings found.");
    end

    otherIssues = [knownIssues; otherIssues(~newWarningIdx,:)];
end

% Display all the other issues
if ~isempty(otherIssues)
    disp("Other Issues:")
    disp(otherIssues);
else
    disp("No other issues found either. (wow, good for you!)")
end

assert(isempty(errors));
assert(isempty(newWarnings));
end

This now loads the issues and does a setdiff to ignore those that are already known and captured in our baseline CSV file. This way, at least from now on I won't introduce any new warnings to the code base. It can only get better from here. Also, if I change some file that has an existing warning, there is a decent chance that my build tooling is going to yell at me because the existing warning is slightly different. For example it might be on a different line due to changes made in the file.

If this happens, great! Make me clean up or suppress the warning while I have the file open and modified. That's a feature not a bug. Worst case scenario, I can always capture a new baseline if I really can't look into it immediately, but I love this approach to help me clean my code through the process.

What does this look like? Let's add a file to the codebase with a new warning:

function codeWarning

anUnusedVariable = "unused";

...and invoke the build:

copyfile .changes/codeWarning.m toolbox/codeWarning.m
try
    buildtool
catch ex
    disp(ex.getReport("basic"));
end

delete toolbox/codeWarning.m
** Starting codeIssues
No critical errors found.
Failed! Found new warnings in code:
       RelativeFilename        Severity                     Description                     CheckID    LineStart    LineEnd    ColumnStart    ColumnEnd
    _______________________    ________    _____________________________________________    _______    _________    _______    ___________    _________

    "toolbox/codeWarning.m"    warning     "Value assigned to variable might be unused."     NASGU         3           3            1            16    

Other Issues:
             RelativeFilename             Severity                     Description                     CheckID    LineStart    LineEnd    ColumnStart    ColumnEnd
    __________________________________    ________    _____________________________________________    _______    _________    _______    ___________    _________

    "toolbox/springMassDamperDesign.m"    warning     "Value assigned to variable might be unused."     NASGU         4           4            3             3    
    "toolbox/springMassDamperDesign.m"    warning     "Value assigned to variable might be unused."     NASGU         6           6            3             3    

## -----------------------------------------------------------------------------
## Error using assert
## Assertion failed.
## 
## Error in buildfile>codeIssuesTask (line 68)
## assert(isempty(newWarnings));
## -----------------------------------------------------------------------------
** Failed codeIssues

Error using buildtool
Build failed.

Love it! I am now protected from myself. I can leverage this in my standard toolbox development process to help ensure that over time my code only gets better, not worse. You could also imagine tweaking this to fail or otherwise notify when a warning goes away from the known issues so that we have some pressure to help ensure the lockdown gets tighter and tighter as time goes on. For reference, here is the final buildfile used for this workflow discussed today:

function plan = buildfile
plan = buildplan(localfunctions);

plan("test").Dependencies = "mex";
plan.DefaultTasks = ["codeIssues" "test"];
end

function mexTask(~)
% Compile mex files
mex mex/convec.c -outdir toolbox/;
end

function testTask(~)
% Run the unit tests
results = runtests;
disp(results);
assertSuccess(results);
end

function codeIssuesTask(~)
% Get all the issues under toolbox
allIssues = codeIssues("toolbox");

% Assert that no errors creep into the codebase
errorIdx = allIssues.Issues.Severity == "error";
errors = allIssues.Issues(errorIdx,:);
otherIssues = allIssues.Issues(~errorIdx,:);
if ~isempty(errors)
    disp("Failed! Found critical errors in code:");
    disp(errors);
else
    disp("No critical errors found.");
end


% Load the known warnings baseline
newWarnings = [];
if isfile("knownIssues.csv")
    otherIssues = preprocessIssues(otherIssues);
    
    opts = detectImportOptions("knownIssues.csv");
    types = varfun(@class, otherIssues,"OutputFormat","cell");
    opts.VariableTypes = types;
    knownIssues = readtable("knownIssues.csv",opts);

    otherIssues = setdiff(otherIssues, knownIssues);
    newWarningIdx = otherIssues.Severity == "warning";
    newWarnings = otherIssues(newWarningIdx,:);
    if ~isempty(newWarnings)
        disp("Failed! Found new warnings in code:");
        disp(newWarnings);
    else
        disp("No new warnings found.");
    end

    otherIssues = [knownIssues; otherIssues(~newWarningIdx,:)];
end

% Display all the other issues
if ~isempty(otherIssues)
    disp("Other Issues:")
    disp(otherIssues);
else
    disp("No other issues found either. (wow, good for you!)")
end

assert(isempty(errors));
assert(isempty(newWarnings));
end

function captureWarningsBaselineTask(~)
% Captures the current codeIssues warnings and creates a baseline csv file
allIssues = codeIssues("toolbox");
warningIdx = allIssues.Issues.Severity == "warning";
warnings = allIssues.Issues(warningIdx,:);

warnings = preprocessIssues(warnings);
if ~isempty(warnings)
    disp("Saving a new ""knownIssues.csv"" baseline file for " + height(warnings) + " code warnings")
    writetable(warnings, "knownIssues.csv");
else
    disp("No warnings to create a baseline for")
end

end

function theTable = preprocessIssues(theTable)
% Make an issues table conducive for baselining via a few small tweaks

% Overwrite the location field with relative paths, and remove absolute paths
basePath = string(pwd) + filesep;
theTable.Location = erase(theTable.FullFilename, basePath);
theTable.Properties.VariableNames{"Location"} = 'RelativeFilename';
theTable.FullFilename = [];

% Convert the Severity to categorical, which serializes nicely to string
theTable.Severity = categorical(theTable.Severity);

end

There you have it, a clean API for MATLAB's code analysis and a standard way to include this in the development process using the build tool. I can practically feel the quality getting higher!

Folks, there is so much to blog about in the next little while. There's more to discuss here on how we can leverage new and improving tools to develop clean build and test pipelines for your MATLAB projects. Also, I am so excited for some really fantastic progress we will be able to share shortly. Buckle in, we are gonna be talking about high quality test and automation developments for a bit here on this blog. Chime in with your insights, tools, and workflows you use as you develop your professional MATLAB projects.


Get the MATLAB code

Published with MATLAB® R2022b

]]>
https://blogs.mathworks.com/developer/2023/03/15/static-analysis-code-checking-and-linting-with-codeissues/feed/ 0
Han Solo Revisited https://blogs.mathworks.com/developer/2022/12/14/the-power-of-encapsulation/?s_tid=feedtopost https://blogs.mathworks.com/developer/2022/12/14/the-power-of-encapsulation/#comments Wed, 14 Dec 2022 08:30:50 +0000 https://blogs.mathworks.com/developer/?p=2945 A long time ago in a blog post far, far away… Andy wrote about Han Solo Encapsulation – to keep Jabba’s “system working as designed he needed to encapsulate his system behind... read more >>

]]>

A long time ago in a blog post far, far away… Andy wrote about Han Solo Encapsulation – to keep Jabba’s “system working as designed he needed to encapsulate his system behind a rock solid interface”.

By a stroke of good fortune, or judicious design choices depending on your perspective, a recent refactoring job that I thought could have far reaching consequences turned into a few changes in a single class, ultimately saving much time. Let’s have a look.

My scenario is that I have a DataProvider that accepts a DataRequest and returns some data:

classdef DataProvider
    
    methods
        
        function data = getData(dataProvider,dataRequest)

            arguments
                dataProvider (1,1) DataProvider
                dataRequest (1,1) DataRequest
            end

            % Implementation continues...

        end
        
    end
    
end

The DataRequest looks like this:

classdef DataRequest
    
    properties
        Make (1,1) string = missing
        Model (1,1) string = missing
        ManufacturingYear (1,1) datetime = missing
    end
    
end

It defines relevant properties that the DataProvider needs to serve up the data. All the values are scalar and have default values of missing to represent the fact that they are “unset” by the user.

To form its query, the DataProvider must extract the values of each property. To do this, it needs to know that “unset” is represented by “missing” and therefore it should be ignored from the search condition.

Here’s one possible implementation in DataProvider:

classdef DataProvider
    
    methods
        
        function data = getData(dataProvider,dataRequest)

            arguments
                dataProvider (1,1) DataProvider
                dataRequest (1,1) DataRequest
            end
            
            searchTerms = {};
            propsToGet = ["Make" "ManufacturingYear" "Model"];
            
            for prop = propsToGet
                if ~ismissing(dataRequest.(prop))
                    searchTerms = [searchTerms {prop} {dataRequest.(prop)}];
                end
            end

            result = dataProvider.search(searchTerms{:});

            % Implementation continues...

        end
        
    end
    
end

The problem here is that DataProvider, and any other code that makes use of DataRequest, is coupled to how DataRequest represents its “unset” values. That’s something that should be encapsulated within the DataRequest. What my DataProvider really wants is an array of name-value pairs of non-missing property names and values.

(We could also describe this as an instance of the tell don’t ask principle since we’re not actually hiding DataRequest’s properties from the outside world.)

Let’s refactor the DataRequest to add a method that does just that. I’ve called it namedargs2cell due to its similarity to the built-in MATLAB function.

classdef DataRequest
    
    properties
        Make (1,1) string = missing
        Model (1,1) string = missing
        ManufacturingYear (1,1) datetime = missing
    end
    
    methods
        
        function paramCell = namedargs2cell(dataRequest)
            
            arguments
                dataRequest (1,1) DataRequest
            end
            
            paramCell = {};
            propsToGet = ["Make" "ManufacturingYear" "Model"];
            
            for prop = propsToGet
                if ~ismissing(dataRequest.(prop))
                    paramCell = [paramCell {prop} {dataRequest.(prop)}];
                end
            end
            
        end
        
    end
    
end

This makes our DataProvider much simpler:

classdef DataProvider
    
    methods
        
        function data = getData(dataProvider,request)

            arguments
                dataProvider (1,1) DataProvider
                request (1,1) DataRequest
            end

            searchTerms = namedargs2cell(request);
            result = dataProvider.search(searchTerms{:});

            % Implementation continues...

        end
        
    end
    
end

Returning to our encapsulation principle, what do we achieve? In my real-life case, I wanted to change the DataRequest to allow multiple values to be specified for a given property rather than just scalars. It therefore makes sense to represent “unset” with an empty rather that with a scalar missing. Since all knowledge of how “unset” is represented is contained within the DataRequest, the only changes I needed to make were within DataRequest itself. No other code had to be touched:

classdef DataRequest
    
    properties
        Make (1,:) string
        Model (1,:) string
        ManufacturingYear (1,:) datetime
    end
    
    methods
        
        function paramCell = namedargs2cell(dataRequest)
            
            arguments
                dataRequest (1,1) DataRequest
            end
            
            paramCell = {};
            propsToGet = ["Make" "ManufacturingYear" "Model"];
            
            for prop = propsToGet
                if ~all(isempty(dataRequest.(prop)))
                    paramCell = [paramCell {prop} {dataRequest.(prop)}];
                end
            end
            
        end
        
    end
    
end

In the above, note how ismissing has become all(isempty(…)).

In conclusion, by providing the right interface to your classes, code changes can become much more limited in scope and easier to implement.


Get the MATLAB code

Published with MATLAB® R2022b

]]>
https://blogs.mathworks.com/developer/2022/12/14/the-power-of-encapsulation/feed/ 13
Building Blocks https://blogs.mathworks.com/developer/2022/10/17/building-blocks-with-buildtool/?s_tid=feedtopost https://blogs.mathworks.com/developer/2022/10/17/building-blocks-with-buildtool/#comments Mon, 17 Oct 2022 16:34:02 +0000 https://blogs.mathworks.com/developer/?p=2835

My people! Oh how I have missed you. It has been such a long time since we have talked about some developer workflow goodness here on the blog. I have found it hard to sit down and write up more... read more >>

]]>

My people! Oh how I have missed you. It has been such a long time since we have talked about some developer workflow goodness here on the blog. I have found it hard to sit down and write up more thoughts and musings on these topics, but the silver lining here is a big reason for my lack of time is that we have been hard at work delivering development infrastructure for MATLAB.

One of those things is the new build tool for MATLAB that included in R2022b! We are super excited about this tool's rookie release, but even more excited for all the value that will come of it as you begin using it for your MATLAB projects.

What is this thing anyway? Well in short it is a standard interface for you to build and collaborate on your MATLAB projects. "Build?", you say?

Yes, "Build!", I say. Anyone developing serious, shareable, production grade MATLAB code knows that even though MATLAB is an "easy-to-leverage" language that typically doesn't require an actual "compile" step, it still requires a development process that includes tasks like testing, quality gates, and bumping a release number. Also it turns out that there are many ways in which MATLAB does indeed build something. Think mex files, p-code, code generation, toolbox packages, doc pages, or producing artifacts from MATLAB Compiler and Compiler SDK. These are all build steps.

The issue though, has been that there has been no standard API for MATLAB projects to organize these build steps. It usually ends up looking something like this:

Does this look familiar? It does to me. All of these scripts grow in a project or repo for doing these specific tasks. Each one looks a little different because one was written on Tuesday and the other the following Monday. If we are lucky, we remember how these all work when we need to interact with them. However, sometimes we are not lucky. Sometimes we go back to our code and haven't the foggiest idea how we built it, in what order, and with which scripts.

Also, know who is never so lucky? A new contributor. Someone who wants to contribute to your code and hasn't learned the system you have put in place to develop the project. Some projects are rigorous and do indeed have their own custom-authored build framework put in place. This is great for them, but requires more maintenance, and even in these cases a new developer on the project needs to learn this custom system, which is different than all the other systems to build MATLAB code.

Well, not anymore. Starting in R2022b we now have a standard interface and build framework that enables project owners to easily produce their build in a way that anyone else can consume, no matter how complicated the build pipeline is. We now can move from ad-hoc scripts and custom build systems to a known, structured, and standard framework.

Let's take my favorite simple Mass-Spring-Damper example (looks like I am still a mechanical engineer at heart). This is a simple example "toolbox" that has 3 components, a design script springMassDamperDesign.m that defines stiffness and damping constants for the system, a function simulateSystem.m that simulates the system from an initial condition outside of equilibrium to show a step response, and a mex file convec.c that convolves two arrays, which might be a useful utility for a dynamic system such as this. It also has a couple tests to ensure all is well and good as the code changes.

Hopefully the author of this code knows all about these components and why they were written as they were. However, if I am a contributor for the first time to this code base I have no idea. My workflow might look something like this:

  1. Get the code
  2. Use the toolbox
  3. See there is something I want to change about the toolbox, a feature to add or a tweak to the design
  4. Make the change
  5. Submit the change for the win!!

Seems like I am setting myself up for a solid contribution, and I am very proud of myself. After getting the code I see the initial design looks like so:

function design = springMassDamperDesign(mass)

if nargin
  m = mass;
else
  m = 1500; % Need to know the mass to determine critical damping
end

design.k = 5e6; % Spring Constant
design.c = 5e5; % Damping coefficient


...and when simulating using the included function:

function [x, t] = simulateSystem(design)

if ~isstruct(design) || ~all(isfield(design,{'c','k'}))
    error('simulateSystem:InvalidDesign:ShouldBeStruct', ...
        'The design should be a structure with fields "c" and "k"');
end

% Design variables
c = design.c;
k = design.k;

% Constant variables
z0 = [-0.1; 0];  % Initial Position and Velocity
m = 1500;        % Mass

odefun = @(t,z) [0 1; -k/m -c/m]*z;
[t, z] = ode45(odefun, [0, 1], z0);

% The first column is the position (displacement from equilibrium)
x = z(:, 1);

...it yields the following response:

[t,y] = simulateSystem(springMassDamperDesign);
plot(y,t)

Pretty decent, but I think that there is room for improvement. I think we can get back to equilibrium sooner, and I like a nice smooth shift that slightly overshoots. Because I am an excellent mechanical engineer, this is a clearly preferable design, we just need to have a little less damping:

addpath .changes/round1
function design = springMassDamperDesign(mass)

if nargin
  m = mass;
else
  m = 1500; % Need to know the mass to determine critical damping
end

design.k = 5e6; % Spring Constant
design.c = 1e5; % Damping coefficient


[t,y] = simulateSystem(springMassDamperDesign);
plot(y,t)

...and submit. This is when I get a dose of humility and experience a world of pain. The toolbox maintainer declines my submission because this design fails a test already put in place intended to limit the overshoot of the response. See?

runtests("tests/designTest.m")
Running designTest
.
================================================================================
Verification failed in designTest/testOvershoot.
    ----------------
    Test Diagnostic:
    ----------------
    Overshoot violation! Maximum overshoot is 0.01
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyLessThan failed.
    --> The value must be less than the maximum value.
    
    Actual Value:
       0.010846982843858
    Maximum Value (Exclusive):
       0.010000000000000
    ------------------
    Stack Information:
    ------------------
    In /Users/acampbel/Library/CloudStorage/OneDrive-MathWorks/repos/msd_blog1/tests/designTest.m (testOvershoot) at 23
================================================================================
..
Done designTest
__________

Failure Summary:

     Name                      Failed  Incomplete  Reason(s)
    =======================================================================
     designTest/testOvershoot    X                 Failed by verification.

ans = 

  1×3 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   2 Passed, 1 Failed, 0 Incomplete.
   0.052733 seconds testing time.

In retrospect this is easy to predict, there were tests after all. I should have run them before submitting. But there was nothing pointing me in their direction and I just missed it. For a simple example repo that might seem obvious, but for a "real" toolbox this can be hard to see.

Alright clearly there is more work to do after my contribution was declined tersely by an overworked toolbox author. But I am still up to the task. After learning that there is an overshoot requirement I can tweak my design to fit within these constraints:

addpath .changes/round2
function design = springMassDamperDesign(mass)

if nargin
  m = mass;
else
  m = 1500; % Need to know the mass to determine critical damping
end

design.k = 5e6; % Spring Constant
design.c = 1.1e5; % Damping coefficient


[t,y] = simulateSystem(springMassDamperDesign);
plot(y,t)

Looks good, does it pass the test?

runtests("tests/designTest.m")
Running designTest
...
Done designTest
__________


ans = 

  1×3 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   3 Passed, 0 Failed, 0 Incomplete.
   0.011899 seconds testing time.

Yes! Finally I must be done. However, when I submit this code I get another rejection because there is still a test failing for the mex file utility that I didn't even know about:

runtests("tests")
Running convecTest

================================================================================
Error occurred in convecTest/MatchesConvBaseline and it did not run to completion.
    ---------
    Error ID:
    ---------
    'MATLAB:UndefinedFunction'
    --------------
    Error Details:
    --------------
    Undefined function 'convec' for input arguments of type 'double'.
    
    Error in convecTest (line 6)
    assert(isequal(convec(x,y), conv(x,y)), ...
================================================================================
.
Done convecTest
__________

Running designTest
...
Done designTest
__________

Failure Summary:

     Name                            Failed  Incomplete  Reason(s)
    ===============================================================
     convecTest/MatchesConvBaseline    X         X       Errored.

ans = 

  1×4 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   3 Passed, 1 Failed, 1 Incomplete.
   0.036491 seconds testing time.

Alright, at this point I see that there is some utility that I wasn't changing, using, or even familiar with and it's test is failing. Furthermore, I realize that it is failing because it isn't compiled. I have no idea how to compile this mex file, and at this point I give up because I hadn't planned to invest this much time into this contribution. I don't have time to learn all the details of this repo (I just wanted to tweak the damping coefficient!). After giving up I leave with a bad taste in my mouth. I am probably done trying to contribute to this code base, and actually may even think twice before trying to contribute to some other code base. Not good. No buena. Nicht gut.

Enter buildtool

All of this pain can be addressed through using this new build tool. As a new contributor, all I need to know is that I need to invoke the build tool to go through the author's intended development workflow. I need to learn this the first time, but once I am familiar with this standard framework I can interact with any other project that is also using the build tool. Once I see that the root of the project has a file called buildfile.m I know I am in business and I can do anything the author intended, including things like running tests and compiling mex files, by simply invoking the tool. Let's try it:

buildtool
** Starting mex
Building with 'Xcode with Clang'.
MEX completed successfully.
** Finished mex

** Starting setup
** Finished setup

** Starting test
Running convecTest
.
Done convecTest
__________

Running designTest
...
Done designTest
__________

  1×4 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   4 Passed, 0 Failed, 0 Incomplete.
   0.16678 seconds testing time.

** Finished test

Isn't that beautiful? I didn't have to know anything about how the project is built and I could get rolling quickly. I can make my small change, everything that needs to happen (e.g. building a mex file) happens and then we can confirm it doesn't fail the tests. It makes baking in high quality easy(er).

How's it done?

I have been focusing on the perspective of the unfamiliar contributor. How can the author/owner use this to set up for success? Well this is super simple and leverages an easy to work with scriptable MATLAB interface as the fundamental framework. You start by creating your buildfile.m, which is a function that creates your build plan.

function plan = buildfile

plan = buildplan(localfunctions);

end

Passing all of the local functions when you create your build plan makes it easy to define simple tasks. This enables you to create tasks from any function that ends in the word Task (or task or _task or tAsK, etc). The first comment in the function (the H1 line) gives a task description. For this case we have 3 tasks we'd like to add.

A setup task

function setupTask(context)
% Setup path for the build
addpath(fullfile(context.Plan.RootFolder,"toolbox"));
end

This task ensures that the right paths are in place for the build. You might ask whether this should be done using a MATLAB Project, and the answer is yes absolutely! That is a better way. For now we are building this in but will projectify it in a later post.

A mex task

function mexTask(~)
% Compile mex files

mex mex/convec.c -outdir toolbox/;
end

This is a pretty simple compile in this example, but for many projects this step can be more involved. Simple or complex, here is where you can make it trivial for the newcomer.

A test task

function testTask(~)
% Run the unit tests

results = runtests("tests");
disp(results);
assertSuccess(results);
end

Straightforward. Now that those tasks are defined and automatically included in your build file anyone can see what tasks can be run:

buildtool -tasks
mex   - Compile mex files
setup - Setup path for the build
test  - Run the unit tests

Great, we can see our 3 tasks, but as you might predict that these tasks can't be run in just any order. The tests won't pass unless the proper code is on the path and the mex file is built. These task dependency relationships can be defined in the main function as you setup your plan. We need to add these dependencies, and while we are at it, let's setup a default task so that buildtool will work without even passing any arguments.

function plan = buildfile

plan = buildplan(localfunctions);

plan("test").Dependencies = ["mex", "setup"];

plan.DefaultTasks = "test";
end

Now we can invoke it by default by just calling buildtool (as we did above) or we can invoke a specific task we'd like to run such as mex and it will just run what is required for that task:

buildtool mex
** Starting mex
Building with 'Xcode with Clang'.
MEX completed successfully.
** Finished mex

Here is the full buildfile for your reference:

function plan = buildfile

plan = buildplan(localfunctions);

plan("test").Dependencies = ["mex", "setup"];

plan.DefaultTasks = "test";
end

function setupTask(context)
% Setup path for the build
addpath(fullfile(context.Plan.RootFolder,"toolbox"));
end

function mexTask(~)
% Compile mex files

mex mex/convec.c -outdir toolbox/;
end

function testTask(~)
% Run the unit tests

results = runtests("tests");
disp(results);
assertSuccess(results);
end


Alright with that I am going or send you off to begin your MATLAB project development adventures with the new build tool. We'd love to hear your feedback. Let's make this a series! I am going to blog a few more times on this so you can see this project grow in capabilities and really start to leverage this build framework. Also, we are working like crazy on future capabilities for this tool. So on multiple fronts this is just the beginning of much more to come.


Get the MATLAB code

Published with MATLAB® R2022b

]]>
https://blogs.mathworks.com/developer/2022/10/17/building-blocks-with-buildtool/feed/ 1
Failure is the first step to trying https://blogs.mathworks.com/developer/2022/04/12/always-start-with-a-failing-test/?s_tid=feedtopost https://blogs.mathworks.com/developer/2022/04/12/always-start-with-a-failing-test/#respond Tue, 12 Apr 2022 08:31:04 +0000 https://blogs.mathworks.com/developer/?p=2792

The official guidance on test-driven development is to follow the red-green-refactor cycle:... read more >>

]]>

The official guidance on test-driven development is to follow the red-green-refactor cycle:

  1. Write a test that fails.
  2. Make it pass.
  3. Refactor.

But what’s the point in starting with a test that fails? To make sure you’ve written the right test! I encountered some unexpected behaviour recently that highlighted this point.

Imagine you have a Library that aggregates Items. An Item can be either a Book or a Film, but not both at the same time. If we create a Library in "book mode", it should initially contain an empty Book. If we create it in "film mode", it should initially contain an empty Film. Let’s start by writing a test to capture the book mode behaviour:

classdef tLibrary < matlab.unittest.TestCase

    methods(Test)

        function bookModeInitialisesToEmptyBook(testCase)
            
            lib = Library(Mode="book");
            
            testCase.verifyEqual(lib.Items,Book.empty(1,0))
            
        end
        
    end

end

(The Name=value syntax for name-value pairs was introduced in R2021a. It’s interchangeable with the classic (…,"Name",value) syntax.)

Let’s run the test. We expect it to fail because Library doesn’t exist.

We’ll skip over the steps of creating a blank class definition, the subsequent test failures due to a missing constructor with input arguments and a public Items property, and iteratively adding them in.

Instead, let's jump to the implementation of Library that makes our test pass:

classdef Library
    
    properties
        Items (1,:) Item = Book.empty
    end
    
    methods
        
        function lib = Library(nvp)
            
            arguments
                nvp.Mode (1,1) string {mustBeMember(nvp.Mode,["book" "film"])} = "book"
            end
            
        end
        
    end
    
end

We run the test and see that it passes:

So far, so good. Now we write a test to capture film mode:

function filmModeInitialisesToEmptyFilm(testCase)

    lib = Library(Mode="film");

    testCase.verifyEqual(lib.Items,Film.empty(1,0))

end

We run the test:

And… it passes!?

Why is it passing? We can use the debugger to inspect lib.Items manually and see that it’s an empty Book and not an empty Film. After some investigation, we find that verifyEqual relies on isequal and isequal considers two empties of different classes to be equal under some circumstances.

Whether or not this behaviour of isequal is correct, the important point for us is that we’ve written the wrong test! Our implementation could have been wrong and we wouldn’t have known. We would have achieved full coverage but our test would not have been contributing useful information.

We therefore need to rewrite our test to catch this issue:

function filmModeInitialisesToEmptyFilm(testCase)

    lib = Library(Mode="film");

    testCase.verifyEmpty(lib.Items)
    testCase.verifyClass(lib.Items,?Film)

end

Let’s run the updated test:

The test now fails and we can continue with our implementation, confident that our test will only pass when our implementation is correct.


Get the MATLAB code

Published with MATLAB® R2022a

]]>
https://blogs.mathworks.com/developer/2022/04/12/always-start-with-a-failing-test/feed/ 0
You’ve Got Mail https://blogs.mathworks.com/developer/2022/03/17/using-custom-classes-to-convey-intent/?s_tid=feedtopost https://blogs.mathworks.com/developer/2022/03/17/using-custom-classes-to-convey-intent/#comments Thu, 17 Mar 2022 12:51:39 +0000 https://blogs.mathworks.com/developer/?p=2782 Let’s send some emails!Imagine you write a function that sends an email to a customer. Your initial function call might look like this: ... read more >>

]]>

Let’s send some emails!

Imagine you write a function that sends an email to a customer. Your initial function call might look like this:

  sendEmail(emailAddress,firstName,lastName,bodyText)

Having 4 inputs to our function is fine, but if we need to add more customer details it becomes difficult to manage. When working on software projects, we find that certain variables tend to be grouped together. In our case, “first name”, “last name”, and “email address” represent a customer. We might also have “name”, “description”, and “price” to represent a product, or “customer”, “product”, and “datetime” to represent an order.

So let’s group our first 3 inputs into a single variable. The simplest way to do this is with a struct:

  customer = struct("FirstName",firstName,"LastName",lastName,"Email",emailAddress);

Our function call then becomes:

  sendEmail(customer,bodyText)

The advantage of this is that we can now pass around all the information related to a customer as a single variable. If their postal address is added to the struct, it’s automatically available for us to use in our sendEmail function.

What’s in a customer?

A disadvantage arises when a new developer (or you several weeks from now!) looks at this function again. What exactly is in the “customer” struct? Is the field for their first name called “FirstName” or “Firstname” or “firstname”?

If you try to access a field that doesn’t exist, you’ll get an error, but if you set a field that didn’t previously exist, it will be automatically added to the struct. Maybe you can work out the correct names from the existing code, assuming it’s not too long or complex. Otherwise, you’ll have to run a test (because someone has written a test, right?) and see what is provided to the function at runtime.

With a customer, we can at least take an educated guess as to the contents. If a more generic name is adopted (“data”, “metdadata”, “info”), it can be impossible.

This lack of clarity wastes developer time over and over again, and can cause subtle bugs that are hard to track down.

What’s a valid customer?

To address this, we could define validation on the customer input: we want to make sure that we have a scalar struct with the correct field names, that the data types are correct, and that any other validation rules are met. For example, that the email address is always of the format “*@*.*” .

Managing this can be complex and laborious. If you’re not careful, it leads to scattered code that repeats the validation and that is difficult to read. Furthermore, can we guarantee that all such “customer structs” will always be valid? No – it’s possible for us, or another developer, to create an invalid customer struct and we won’t know about it until we run our validation again or the code errors.

This forms a second source of bugs – data that is not in the expected format. Common issues include variables that are empty but shouldn’t be, or a cellstr that should be a string array.

What do we need from a customer?

There are operations that we frequently need to perform on our customer struct. For example, constructing the customer’s full name from their first and last name:

  fullname = customer.FirstName + " " + customer.LastName;

Writing this code each time we need the full name either leads to code duplication (and a high likelihood of inconsistency), or a function that’s divorced from its data and potentially hard to find.

Define a class instead

Instead, we can define a class for our customer! Doing so will bring the following advantages:

Functions that use the customer class only need one line of validation in the arguments block – the required class. The validation line tells you exactly what the input is, and you can immediately hit Ctrl+D to go to the class definition. It forms a clear contract between the calling code and your class.

The properties block tells you exactly what properties all objects of that class will have. The validation for each property makes explicit what each property will contain, it can set default values, and it guarantees that all objects of the class will be valid.

Dependent properties can be added to give you derived information without having to poke around the internals of the object (hello encapsulation!); tell the object what you want it to do, don’t ask for the individual pieces of data. Other functionality related to the class can be added as methods.

Make your class work with arrays

Custom classes really begin to shine when you make them array compatible. Rather than having a customer or an order, you can have an array of customers or orders and perform operations on them in one go. This native array handling is one of the unique features of MATLAB and removes the need for a “collection” object like you might have in C.

An array-based method I almost always add is table. table transforms an array of objects into a standard MATLAB table. It allows you to easily see the entire contents of the object array and perhaps to write the data into a uitable for display or to a spreadsheet for reporting purposes.

So why go to all this trouble of creating a custom class just to turn it back into a generic data type? The crucial difference now is that the table is derived from our custom class which handles all the validation and calculations; the table is not the source of truth.

Code example

Below is example code showing what a customer might look like when implemented as a class in MATLAB:

  • Property names are fixed, always present, and cannot be changed at runtime.
  • Data types and sizes are fixed, and type conversion performed automatically where possible (e.g. char to string).
  • Email is validated whenever it is changed.
  • The FullName dependent property gives calling code direct access to what it actually wants.
  • The table method allows us to easily visualise the contents of a customer array and that data to be consumed outside of our application.
classdef Customer
    
    properties
        FirstName (1,1) string
        LastName (1,1) string
        Email (1,1) string {mustBeValidEmail} = "undefined@domain.com"
    end
    
    properties (Dependent,SetAccess = private)
        FullName (1,1) string
    end

    methods
        
        function cust = Customer(first,last,email)
            
            cust.FirstName = first;
            cust.LastName = last;
            cust.Email = email;
            
        end
        
        function str = get.FullName(customer)
            
            str = customer.FirstName + " " + customer.LastName;
            
        end
        
        function tbl = table(customers)
            
            arguments
                customers (1,:) Customer
            end
            
            fn = [customers.FirstName]';
            ln = [customers.LastName]';
            email = [customers.Email]';
            
            tbl = table(fn,ln,email,'VariableNames',["FirstName" "LastName" "Email"]);
            
        end
        
    end
    
end

function mustBeValidEmail(value)
    
    anyLetterOrNum = alphanumericsPattern();
    pat = anyLetterOrNum + "@" + anyLetterOrNum + "." + anyLetterOrNum;
    assert(matches(value,pat),"Customer:InvalidEmail","Invalid email")
    
end

Here’s how we might use it:

c(1) = Customer("Mitch","Docker","mitch@foo.com")
c(2) = Customer("Lachlan","Morton","lachlan@bah.com");
c(3) = Customer("Rigoberto","Uran","rigo@uran.com");
table(c)
c = 

  Customer with properties:

    FirstName: "Mitch"
     LastName: "Docker"
        Email: "mitch@foo.com"
     FullName: "Mitch Docker"


ans =

  3×3 table

     FirstName     LastName          Email      
    ___________    ________    _________________

    "Mitch"        "Docker"    "mitch@foo.com"  
    "Lachlan"      "Morton"    "lachlan@bah.com"
    "Rigoberto"    "Uran"      "rigo@uran.com"  

We can validate the inputs to our sendEmail function with a simple arguments block:

function sendEmail(customer,bodyText)
    
    arguments
        customer (1,1) Customer
        bodyText (1,1) string
    end
    
    % Other code...
    
end

When should I define a custom class?

You may be wondering at what point you should go to the effort and formality of creating custom classes. Much like making the decision to go from a script to a function or a function to a class, it’s when you hit the limits of what your current implementation can do. If you find that:

  • It’s difficult to understand what’s in your data structure.
  • You have problems with validation.
  • Code that’s closely related to the data is duplicated, scattered, or inconsistent.

It’s time to think about custom classes.


Get the MATLAB code

Published with MATLAB® R2022a

]]>
https://blogs.mathworks.com/developer/2022/03/17/using-custom-classes-to-convey-intent/feed/ 7