{"id":3098,"date":"2023-08-30T04:04:22","date_gmt":"2023-08-30T08:04:22","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=3098"},"modified":"2023-08-30T04:04:22","modified_gmt":"2023-08-30T08:04:22","slug":"weve-got-you-covered","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2023\/08\/30\/weve-got-you-covered\/","title":{"rendered":"We\u2019ve got you covered!"},"content":{"rendered":"
\r\n
In the previous blog post<\/span><\/a> of this series, I gave an overview of MATLAB Test and went into the details of the Test Manager. Today\u2019s topic is code coverage.<\/div>\r\n
Code coverage is a measure of \u201chow much\u201d of your code has been executed. By measuring and examining code coverage, you can gain insights into which parts of your code have not been fully tested, and therefore where you should direct your efforts in writing more test cases. Similarly, if the code coverage report is saying that some parts of your code are not being executed and you know you\u2019re satisfying your requirements, chances are you can delete this dead logic.<\/div>\r\n
There are a variety of ways of measuring code coverage. Base MATLAB provides function and statement coverage whilst MATLAB Test adds to that by providing decision, condition, and modified condition\/decision coverage metrics. Here\u2019s a summary of the different coverage metrics:<\/div>\r\n
\"coverageSummary.png\"<\/div>\r\n

What level of coverage do I need?<\/h2>\r\n
Given the above coverage types, which one(s) should you use and what level of coverage do you need to achieve? The answer to is that it\u2019s largely your choice based on your project\u2019s requirements and constraints.<\/div>\r\n
Statement coverage is the starting point for general purpose applications. However, you may have constraints imposed by regulations if you\u2019re working in a safety critical environment such as medical (IEC 62304), aerospace (DO-178), or automotive (ISO 26262).<\/div>\r\n
One of my colleagues likes to use the analogy of choosing an insurance policy \u2013 if the risk is low and the impact of anything going wrong is also low, you\u2019ll likely choose the cheapest, most basic insurance. However, when the risk and consequences of something going wrong are high, you\u2019ll want a comprehensive policy which comes at additional cost. In the world of testing, that more comprehensive insurance corresponds to achieving increasing levels of coverage with more advanced metrics. The cost is your time to write the additional tests.<\/div>\r\n

Collecting coverage<\/h2>\r\n
How do we collect coverage? There are three ways!<\/div>\r\n

Test Browser<\/h3>\r\n
If you\u2019re using the Test Browser (MATLAB R2023a onwards), click the coverage button, check \u201cEnable Coverage reporting\u201d, and add the files or folders of code that you want to measure coverage for. Then run your tests and an easy-to-read HTML coverage report will open.<\/div>\r\n
\"testBrowser.png\"<\/div>\r\n
Collecting statement coverage via the Test Browser.<\/span><\/div>\r\n
Note that if you want to record decision, condition, or MC\/DC, you will still need a MATLAB Test licence.<\/div>\r\n

Test Manager<\/h3>\r\n
If you\u2019re using the Test Manager (MATLAB Test, R2023a onwards), click the coverage button and check \u201cEnable Coverage\u201d in the pop-out menu.<\/div>\r\n
\"testManager.png\"<\/div>\r\n
Collecting statement coverage via the Test Manager.<\/span><\/div>\r\n
Run the tests and then view the report by clicking the adjacent menu button:<\/div>\r\n
\"testManager_viewReport.png\"<\/div>\r\n
Opening the coverage report from the Test Manager.<\/span><\/div>\r\n

Command Window<\/h3>\r\n
Finally, if you\u2019re running your tests programmatically using runtests<\/span><\/a>, you can use the ReportCoverageFor<\/span><\/a> option to specify the files, folders, and packages that you want to record coverage for.<\/div>\r\n
(This will give you the default statement and function coverage. If you want to use the more advanced coverage metrics, you\u2019ll need to use the longhand method to create your own TestRunner<\/span><\/a> and add the CodeCoveragePlugin<\/span><\/a> here you can specify the coverage level.)<\/div>\r\n
Once your tests have finished running, a hyperlink will appear in the Command Window taking you to the coverage report.<\/div>\r\n

Interactive HTML coverage report<\/h3>\r\n
The code coverage report looks like this:<\/div>\r\n
\"coverageReportOverview.png\"<\/div>\r\n
Example code coverage report.<\/span><\/div>\r\n
The top section provides a summary of all the code you have measured coverage for, for all the different coverage types you have selected. Use the \u201cCurrently viewing\u201d drop-down menu to choose which type of coverage is being shown in the rest of the report.<\/div>\r\n
The middle section provides a breakdown on a per file basis. Click a line in the table to choose which file is displayed in the bottom section.<\/div>\r\n
The bottom section provides coverage information on a line-by-line basis for the selected coverage type and the selected file. We\u2019ll explore the details of this next.<\/div>\r\n

Types of coverage<\/h2>\r\n
In the following, I expand on what\u2019s covered in the documentation for types of coverage available with MATLAB Test<\/span><\/a>. This is a great reference so keep it to hand!<\/div>\r\n

Function coverage<\/h3>\r\n
Function and statement coverage are the default coverage metrics that MATLAB records. Function coverage indicates whether each function in your code gets executed at least once. (This is not to be confused with functional<\/span>coverage which relates to the overall functionality being correct. This and requirements-based testing are topics for another day!)<\/div>\r\n
In the following example, we have three functions \u2013 the top-level function myFunction<\/span> and two nested functions fcnOne<\/span> and fcnTwo<\/span>. I\u2019ve recorded coverage for when I call myFunction(0,0,0)<\/span>.<\/div>\r\n
\"functionCoverage.png\"<\/div>\r\n
Example function coverage report with 67% coverage.<\/span><\/div>\r\n
This executes myFunction<\/span> itself and fcnTwo<\/span> but not fcnOne<\/span>. We therefore have 3 functions in total of which 2 have been executed, giving 2\/3 = 67% function coverage. There are clearly holes on our testing so let\u2019s move onto statement coverage.<\/div>\r\n

Statement coverage<\/h3>\r\n
Statements are small chunks of code that MATLAB executes that are separated by a comma, semicolon, or newline character. To achieve 100% statement coverage, each statement much be executed at least once.<\/div>\r\n
The following snippet from the coverage report shows statement coverage for when I call myFunction(0,0,0)<\/span>.<\/div>\r\n
\"statementCoverage.png\"<\/div>\r\n
Example statement coverage report with 60% coverage.<\/span><\/div>\r\n
The branch is not executed so lines 4 and 12 are not covered and show in red. We have 5 statements in total, 3 of which have been executed, so the statement coverage is 3\/5 = 60%. By inspection we can see that the if<\/span>statement is only ever false; we\u2019re missing the case of it being true. Let\u2019s look at how we can view this more formally.<\/div>\r\n

Decision coverage<\/h3>\r\n
Decisions are expressions that alter the execution flow of the code in conjunction with the MATLAB keywords if<\/span>, elseif<\/span>, switch<\/span>, for<\/span>, or while<\/span>. Decision coverage measures the extent to which all these decisions have been tested to be both true and false. It therefore provides you with a view of code branches with missing tests.<\/div>\r\n
(In addition, the short-circuit operators<\/span><\/a> || and && are also decisions since the value on the left-hand side of the expression determines whether the right-hand side is evaluated. However, to be consistent with Simulink Coverage, these expressions (\u201cnon-branch decisions\u201d) are excluded from the metrics when recording decision coverage but are<\/span> included when recording MC\/DC.)<\/div>\r\n
In the previous example we achieved full statement coverage for the if<\/span> decision on line 3. In the following example, I\u2019ve once again tested my code by calling myFunction(0,0,0)<\/span> and have measured decision coverage.<\/div>\r\n
\"decisionCoverage_partial.png\"<\/div>\r\n
Example decision coverage report with 50% coverage.<\/span><\/div>\r\n
We now see that line 3 is only partially covered (orange) as it only evaluates to false. There is 1 decision with 2 possible values, so 2 combinations in total. We hit 1 of them so that\u2019s 1\/2 = 50% decision coverage.<\/div>\r\n
I\u2019ll now add in an additional test point so that I\u2019m now calling myFunction(0,0,0) <\/span>and myFunction(1,0,0).<\/span> This achieves full decision coverage:<\/div>\r\n
\"decisionCoverage_full.png\"<\/div>\r\n
Example decision coverage report with 100% coverage.<\/span><\/div>\r\n
You may also have noticed that the coverage recording level is cumulative \u2013 recording decision coverage also includes statement and function coverage.<\/div>\r\n

Condition coverage<\/h3>\r\n
Decisions can be composed of two or more logical expressions separated by a short-circuit operator (|| or &&) \u2013 these are called conditions. Here is an example with 1 decision and 2 conditions:<\/div>\r\n
\r\n
if <\/span>(x > -3) || (x == -5) \u2026<\/span><\/span><\/div>\r\n<\/div>\r\n
The following contains one decision but no conditions as there are no expressions separated by a short-circuit operator:<\/div>\r\n
\r\n
if <\/span>x == 0 \u2026<\/span><\/span><\/div>\r\n<\/div>\r\n
Below is the condition coverage report for our test code when I call myFunction(0,0,0) <\/span>and myFunction(1,0,0)<\/span>. Remember how we achieved full coverage previously? Not anymore! Condition coverage reveals yet more holes in our testing.<\/div>\r\n
\"conditionCoverage_partial.png\"<\/div>\r\n
Example condition coverage report with 50% coverage.<\/span><\/div>\r\n
The x condition is covered (green) as it takes values of both true and false. The y condition is partially covered (orange) as it only evaluates to false. The z condition is not covered (red) as it is never executed \u2013 why? y is always false and so the && short circuits and doesn\u2019t bother evaluating z.<\/div>\r\n
We have 3 conditions, each of which can take two values (true or false), so there are 6 combinations in total. We\u2019re hitting 3\/6 = 50% coverage.<\/div>\r\n
Let\u2019s modify our tests to fill in the missing coverage:<\/div>\r\n
    \r\n \t
  • myFunction(0,0,0)<\/li>\r\n \t
  • myFunction(1,0,0)<\/li>\r\n \t
  • myFunction(0,1,0)<\/li>\r\n \t
  • myFunction(0,1,1)<\/li>\r\n<\/ul>\r\n
    \"conditionCoverage_full.png\"<\/div>\r\n
    Example condition coverage report with 100% coverage.<\/span><\/div>\r\n

    MC\/DC<\/h3>\r\n
    MC\/DC stands for Modified condition\/decision coverage. It identifies how tests independently exercise conditions within decisions. To achieve full MC\/DC, both of the following must be achieved:<\/div>\r\n
      \r\n \t
    1. All conditions within decisions are evaluated to both true and false.<\/li>\r\n \t
    2. Every condition within a decision independently affects the outcome of the decision.<\/li>\r\n<\/ol>\r\n
      MC\/DC is both stricter than condition coverage whilst allowing full coverage to be achieved with fewer tests than would be required to test all combinations of conditions exhaustively. Consider our if<\/span> statement of 1 decision with 3 conditions:<\/div>\r\n
      \r\n
      if <\/span>x || (y && z) \u2026<\/span><\/span><\/div>\r\n<\/div>\r\n
      All the possible combinations of x, y, and z and the corresponding result are listed in the table below.<\/div>\r\n
      \"mcdcCombinations.png\"<\/div>\r\n
      Looking at this table, we can identify test combinations that satisfy the requirements of MC\/DC. Each condition must evaluate to true and false, and independently affect the result. For example:<\/div>\r\n
        \r\n \t
      • x \u2013 2 and 6.<\/li>\r\n \t
      • y \u2013 2 and 4.<\/li>\r\n \t
      • z \u2013 3 and 4.<\/li>\r\n<\/ul>\r\n
        We can therefore achieve full MC\/DC with test combinations 2, 3, 4, and 6. Here\u2019s the corresponding coverage report when I run these test combinations:<\/div>\r\n
        \"mcdc_full.png\"<\/div>\r\n
        Example MC\/DC report with 100% coverage.<\/span><\/div>\r\n
        Under the \u201cTest Pairs\u201d heading, you can see the various combinations of input values used to achieve 100% MC\/DC. \u201cT\u201d denotes true, \u201cF\u201d false, and \u201cx\u201d is don\u2019t care.<\/div>\r\n
        \"mcdc_explanation.png\"<\/div>\r\n
        By contrast, here\u2019s an example of incomplete MC\/DC when I only ran test combinations 2 & 6:<\/div>\r\n
        \"mcdc_partial.png\"<\/div>\r\n
        Example MC\/DC report with 33% coverage.<\/span><\/div>\r\n
        As per the bulleted list above, test combinations 2 & 6 provide full MC\/DC for x, and for y to independently produce a result of false. Note how the framework has used test condition 2 (FFx) in the first test pair as opposed to test condition 3 previously.<\/div>\r\n
        We have 1 out of our 3 conditions achieving MC\/DC, so the overall coverage is 1\/3 = 33%. By comparison, we have tested 3\/6 conditions so have achieved 50% condition coverage. This demonstrates how MC\/DC is a stricter metric than condition coverage.<\/div>\r\n

        Summary<\/h2>\r\n
        We\u2019ve looked at the following types of coverage:<\/div>\r\n
          \r\n \t
        • Function (MATLAB)<\/li>\r\n \t
        • Statement (MATLAB)<\/li>\r\n \t
        • Decision (MATLAB Test)<\/li>\r\n \t
        • Condition (MATLAB Test)<\/li>\r\n \t
        • Modified Condition \/ Decision Coverage (MCDC) (MATLAB Test).<\/li>\r\n<\/ul>\r\n
          These metrics can be used to guide your testing and therefore help you to produce high quality software whether that\u2019s for everyday desktop use or for safety critical applications. Remember, getting 100% coverage is not the goal \u2013 the goal is better quality software consistent with the time budget of the project, the needs of the end deliverable, and where it will be deployed.<\/div>\r\n
          In the next post in this series, we\u2019ll look at MATLAB Test\u2019s Code Quality dashboard.<\/div>\r\n<\/div>\r\n