Introducing MATLAB Test!
MATLAB R2023a was released back in March, and with it was a new product MATLAB Test. In a series of blog posts starting today, I’m going to show you how MATLAB Test can help you improve the quality of your code whether you’re developing a MATLAB Toolbox to share with your colleagues or writing code that will be deployed into a safety critical environment such as for medical devices.
Overview
There are five key features in the first release of MATLAB Test which I’m going to cover. They will be extended and added to over the coming releases:
- Test Manager – manage your tests and results (today’s topic).
- Extended coverage – measure condition, decision, and modified condition and decision coverage (MCDC) to help ensure your code is fully tested and meets regulatory requirements.
- Code quality dashboard – see an overview of your project’s quality status such as coverage, Code Analyzer warnings, and requirements links.
- Equivalence testing with MATLAB Coder and MATLAB Compiler SDK – easily compare the behaviour of generated C or C++ code, Java or Python packages, or .NET assemblies to that of desktop MATLAB.
- Dependency based test selection – execute only the tests in your suite that are impacting by changes in your code base.
In addition, tool qualification artifacts are available for MATLAB Test in our IEC Certification Kit making MATLAB Test suitable for use in the development of high integrity software.
What’s not changing?
Don’t worry! The MATLAB unit testing (including measuring statement coverage), mocking, app testing, and performance testing frameworks are not changing and remain in base MATLAB.
In addition, R2023a sees the introduction to base MATLAB of the Test Browser, a simple interface to run tests and browse results.
Test Browser
The Test Browser will launch automatically whenever you run tests from a test file using the “Run Tests” (F5) or “Run Current Test” (Ctrl + Enter) buttons. If you run your tests from the command line (runtests(“tDemoSmokeTest”)), the Test Browser will not launch and you will instead see the classic Command Window output.
Test Manager
The Test Manager allows you to create and manage test suites, view and search results, view requirements links, manage coverage settings, and persist results between MATLAB sessions.
So what’s the difference between the Test Browser and the Test Manager? The Test Browser is a simple viewer for iteratively developing code and tests within a MATLAB session. The Test Manager allows you to manage tests at scale, link to requirements, and persist results between sessions.
To use Test Manager, you will need to be using a Project. (For this blog post, I’m using the shipping shortest path example.) The Test Manager needs Projects for dependency information and for persisting test results. You can launch the Test Manager from the Project Toolstrip under the Apps section:
Launch the Test Manager from the Project toolstrip.
When you first launch the Test Manager, it will automatically find all tests in your project. You can add additional columns to the table view (click the green plus icon to the top right of the table) to view test tags and linked requirements (requires the Requirements Toolbox).
Test Manager with “Tags” and “Requirements” columns added.
Once you have run the tests (hit the green play button!), you can view your tests. You can filter the results by their pass/fail/incomplete/not run status, or search based on some text. Here I’ve searched for “input” which finds all the tests that have the tag “InputTests”.
Test results filtered to show those that include “input” in their metadata.
Collecting coverage information
Coverage can be configured using the option in the menu bar:
Coverage settings.
Whilst the definition of statement coverage is fairly obvious, the differences between the others can be subtle. We’ve added a page in the documentation to explain the meaning of each type of coverage as they apply to MATLAB source code. I will also explore it more in a future blog post.
The coverage recording level in MATLAB Test is cumulative, i.e. MC/DC also includes Statement, Decision, and Condition.
Coverage results can be viewed by clicking the adjacent menu item after all tests have been run:
Access the coverage report.
(As of R2023a, coverage information is only available when all tests in the current project are run; coverage is not recorded when running a custom test suite. This restriction will be removed in a future release.)
Coverage is recorded for all files in your project that have the classification of “Design”. By default, all files in a project will be classified as “Design” except those that use the unit testing framework which are classified as “Test”. If you want to remove a file from the coverage report, change the classification to something else.
Change a file’s classification to something other than “Design” to exclude it from coverage recording.
Creating custom test suites
The Test Manager provides a graphical interface for creating custom test suites by using this menu option:
Create a custom test suite.
Here, I’ve created a new test suite to select all tests that have the tag “InputTests”:
Creating a new test suite for all tests in the project with the tag “InputTests”.
I can then select my suite from the dropdown menu and just run that subset:
Viewing tests that are part of the custom suite.
Session persistence
Finally, since the Test Manager persists the results of tests from session to session, I can close MATLAB, open it up again later, reopen the Test Manager, and the results of my last test run will still be there.
Summary
MATLAB Test’s Test Manager helps you to manage your test suite, coverage, and requirements as you scale up your testing activities.
In the next post, I’ll take a look at the new coverage options in MATLAB Test.
- Category:
- MATLAB Test,
- Testing,
- Toolbox Development
Comments
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.