We’ve got you covered!
In the previous blog post of this series, I gave an overview of MATLAB Test and went into the details of the Test Manager. Today’s topic is code coverage.
Code coverage is a measure of “how much” of your code has been executed. By measuring and examining code coverage, you can gain insights into which parts of your code have not been fully tested, and therefore where you should direct your efforts in writing more test cases. Similarly, if the code coverage report is saying that some parts of your code are not being executed and you know you’re satisfying your requirements, chances are you can delete this dead logic.
There are a variety of ways of measuring code coverage. Base MATLAB provides function and statement coverage whilst MATLAB Test adds to that by providing decision, condition, and modified condition/decision coverage metrics. Here’s a summary of the different coverage metrics:
What level of coverage do I need?
Given the above coverage types, which one(s) should you use and what level of coverage do you need to achieve? The answer to is that it’s largely your choice based on your project’s requirements and constraints.
Statement coverage is the starting point for general purpose applications. However, you may have constraints imposed by regulations if you’re working in a safety critical environment such as medical (IEC 62304), aerospace (DO-178), or automotive (ISO 26262).
One of my colleagues likes to use the analogy of choosing an insurance policy – if the risk is low and the impact of anything going wrong is also low, you’ll likely choose the cheapest, most basic insurance. However, when the risk and consequences of something going wrong are high, you’ll want a comprehensive policy which comes at additional cost. In the world of testing, that more comprehensive insurance corresponds to achieving increasing levels of coverage with more advanced metrics. The cost is your time to write the additional tests.
Collecting coverage
How do we collect coverage? There are three ways!
Test Browser
If you’re using the Test Browser (MATLAB R2023a onwards), click the coverage button, check “Enable Coverage reporting”, and add the files or folders of code that you want to measure coverage for. Then run your tests and an easy-to-read HTML coverage report will open.
Collecting statement coverage via the Test Browser.
Note that if you want to record decision, condition, or MC/DC, you will still need a MATLAB Test licence.
Test Manager
If you’re using the Test Manager (MATLAB Test, R2023a onwards), click the coverage button and check “Enable Coverage” in the pop-out menu.
Collecting statement coverage via the Test Manager.
Run the tests and then view the report by clicking the adjacent menu button:
Opening the coverage report from the Test Manager.
Command Window
Finally, if you’re running your tests programmatically using runtests, you can use the ReportCoverageFor option to specify the files, folders, and packages that you want to record coverage for.
(This will give you the default statement and function coverage. If you want to use the more advanced coverage metrics, you’ll need to use the longhand method to create your own TestRunner and add the CodeCoveragePlugin here you can specify the coverage level.)
Once your tests have finished running, a hyperlink will appear in the Command Window taking you to the coverage report.
Interactive HTML coverage report
The code coverage report looks like this:
Example code coverage report.
The top section provides a summary of all the code you have measured coverage for, for all the different coverage types you have selected. Use the “Currently viewing” drop-down menu to choose which type of coverage is being shown in the rest of the report.
The middle section provides a breakdown on a per file basis. Click a line in the table to choose which file is displayed in the bottom section.
The bottom section provides coverage information on a line-by-line basis for the selected coverage type and the selected file. We’ll explore the details of this next.
Types of coverage
In the following, I expand on what’s covered in the documentation for types of coverage available with MATLAB Test. This is a great reference so keep it to hand!
Function coverage
Function and statement coverage are the default coverage metrics that MATLAB records. Function coverage indicates whether each function in your code gets executed at least once. (This is not to be confused with functionalcoverage which relates to the overall functionality being correct. This and requirements-based testing are topics for another day!)
In the following example, we have three functions – the top-level function myFunction and two nested functions fcnOne and fcnTwo. I’ve recorded coverage for when I call myFunction(0,0,0).
Example function coverage report with 67% coverage.
This executes myFunction itself and fcnTwo but not fcnOne. We therefore have 3 functions in total of which 2 have been executed, giving 2/3 = 67% function coverage. There are clearly holes on our testing so let’s move onto statement coverage.
Statement coverage
Statements are small chunks of code that MATLAB executes that are separated by a comma, semicolon, or newline character. To achieve 100% statement coverage, each statement much be executed at least once.
The following snippet from the coverage report shows statement coverage for when I call myFunction(0,0,0).
Example statement coverage report with 60% coverage.
The branch is not executed so lines 4 and 12 are not covered and show in red. We have 5 statements in total, 3 of which have been executed, so the statement coverage is 3/5 = 60%. By inspection we can see that the ifstatement is only ever false; we’re missing the case of it being true. Let’s look at how we can view this more formally.
Decision coverage
Decisions are expressions that alter the execution flow of the code in conjunction with the MATLAB keywords if, elseif, switch, for, or while. Decision coverage measures the extent to which all these decisions have been tested to be both true and false. It therefore provides you with a view of code branches with missing tests.
(In addition, the short-circuit operators || and && are also decisions since the value on the left-hand side of the expression determines whether the right-hand side is evaluated. However, to be consistent with Simulink Coverage, these expressions (“non-branch decisions”) are excluded from the metrics when recording decision coverage but are included when recording MC/DC.)
In the previous example we achieved full statement coverage for the if decision on line 3. In the following example, I’ve once again tested my code by calling myFunction(0,0,0) and have measured decision coverage.
Example decision coverage report with 50% coverage.
We now see that line 3 is only partially covered (orange) as it only evaluates to false. There is 1 decision with 2 possible values, so 2 combinations in total. We hit 1 of them so that’s 1/2 = 50% decision coverage.
I’ll now add in an additional test point so that I’m now calling myFunction(0,0,0) and myFunction(1,0,0). This achieves full decision coverage:
Example decision coverage report with 100% coverage.
You may also have noticed that the coverage recording level is cumulative – recording decision coverage also includes statement and function coverage.
Condition coverage
Decisions can be composed of two or more logical expressions separated by a short-circuit operator (|| or &&) – these are called conditions. Here is an example with 1 decision and 2 conditions:
if (x > -3) || (x == -5) …
The following contains one decision but no conditions as there are no expressions separated by a short-circuit operator:
if x == 0 …
Below is the condition coverage report for our test code when I call myFunction(0,0,0) and myFunction(1,0,0). Remember how we achieved full coverage previously? Not anymore! Condition coverage reveals yet more holes in our testing.
Example condition coverage report with 50% coverage.
The x condition is covered (green) as it takes values of both true and false. The y condition is partially covered (orange) as it only evaluates to false. The z condition is not covered (red) as it is never executed – why? y is always false and so the && short circuits and doesn’t bother evaluating z.
We have 3 conditions, each of which can take two values (true or false), so there are 6 combinations in total. We’re hitting 3/6 = 50% coverage.
Let’s modify our tests to fill in the missing coverage:
- myFunction(0,0,0)
- myFunction(1,0,0)
- myFunction(0,1,0)
- myFunction(0,1,1)
Example condition coverage report with 100% coverage.
MC/DC
MC/DC stands for Modified condition/decision coverage. It identifies how tests independently exercise conditions within decisions. To achieve full MC/DC, both of the following must be achieved:
- All conditions within decisions are evaluated to both true and false.
- Every condition within a decision independently affects the outcome of the decision.
MC/DC is both stricter than condition coverage whilst allowing full coverage to be achieved with fewer tests than would be required to test all combinations of conditions exhaustively. Consider our if statement of 1 decision with 3 conditions:
if x || (y && z) …
All the possible combinations of x, y, and z and the corresponding result are listed in the table below.
Looking at this table, we can identify test combinations that satisfy the requirements of MC/DC. Each condition must evaluate to true and false, and independently affect the result. For example:
- x – 2 and 6.
- y – 2 and 4.
- z – 3 and 4.
We can therefore achieve full MC/DC with test combinations 2, 3, 4, and 6. Here’s the corresponding coverage report when I run these test combinations:
Example MC/DC report with 100% coverage.
Under the “Test Pairs” heading, you can see the various combinations of input values used to achieve 100% MC/DC. “T” denotes true, “F” false, and “x” is don’t care.
By contrast, here’s an example of incomplete MC/DC when I only ran test combinations 2 & 6:
Example MC/DC report with 33% coverage.
As per the bulleted list above, test combinations 2 & 6 provide full MC/DC for x, and for y to independently produce a result of false. Note how the framework has used test condition 2 (FFx) in the first test pair as opposed to test condition 3 previously.
We have 1 out of our 3 conditions achieving MC/DC, so the overall coverage is 1/3 = 33%. By comparison, we have tested 3/6 conditions so have achieved 50% condition coverage. This demonstrates how MC/DC is a stricter metric than condition coverage.
Summary
We’ve looked at the following types of coverage:
- Function (MATLAB)
- Statement (MATLAB)
- Decision (MATLAB Test)
- Condition (MATLAB Test)
- Modified Condition / Decision Coverage (MCDC) (MATLAB Test).
These metrics can be used to guide your testing and therefore help you to produce high quality software whether that’s for everyday desktop use or for safety critical applications. Remember, getting 100% coverage is not the goal – the goal is better quality software consistent with the time budget of the project, the needs of the end deliverable, and where it will be deployed.
In the next post in this series, we’ll look at MATLAB Test’s Code Quality dashboard.
- Category:
- MATLAB Test,
- Testing
Comments
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.