Hello all! I was reading up on Guy's blog the other day and remembered his great post showing a Simulink Test workflow with Jenkins and GitLab. One thing I thought was cool in his musings was how he set it up to store the results so that you didn't need to rerun the tests to begin investigating.

Well I am happy to report that may be even easier in R2018b in your CI workflows since the TestReportPlugin now includes all the Simulink Test specific information in the report. This means that you can even begin to debug the problem or get an idea of why the tests failed from directly in your browser by looking at your CI dashboard. This then allows you to perhaps identify the right person to further debug and fix the problem much more efficiently. Just doin our part to keep your workflows well-oiled!

This is really easy to do. To see it in action let's just look at one of the Simulink Test examples, the f14ParameterSweepTest. This test has a few failures that we can hopefully glean some quick insight into. To run this as part of your CI build, you'll need to make a few quick connections. First, for example, you'll want to ensure your Simulink Test file(s) are configured to generate a report. Really all you'll need to do is open the file in Test Manager and click the option to generate the report:

testFile = 'f14ParameterSweepTest.mldatx';
sltest.testmanager.view;
sltest.testmanager.load(testFile);

Once this is checked you'll see some more options. You should just be able to use the default selections to see the value here, but if you want you can tweak what is included in the report.

Alright, next you will need to create your runner with the right plugins. This is what you will want to run from your Jenkins build. Let's first set up the basic CI script. To include this information in your report, you will need two plugins on the TestRunner. First you will need the TestReportPlugin if you want a test report (makes sense I hope). Let's set this up to create an HTML report that we put into a "reports" subfolder of the Jenkins workspace:

import matlab.unittest.*;
import matlab.unittest.plugins.*;

reportLocation = fullfile(getenv('WORKSPACE'), 'reports');
if ~isfolder(reportLocation)
    mkdir(reportLocation);
end
runner = TestRunner.withTextOutput;
runner.addPlugin(TestReportPlugin.producingHTML(reportLocation));

Next, in order to get all of the Simulink Test specific details in the report, add the TestManagerResultsPlugin as well. Note, this plugin is included by default when using runtests, but it needs to be explicitly added here since we are configuring our own runner:

import sltest.plugins.*;
runner.addPlugin(TestManagerResultsPlugin);

runner.run(testsuite(testFile));

Running f14ParameterSweepTest > New Test Suite 1
.......... ........
================================================================================
Verification failed in f14ParameterSweepTest > New Test Suite 1/Iterations Parameter Sweep(ScriptedIteration=Scripted_Iteration19).

    ---------------------
    Framework Diagnostic:
    ---------------------
    Failed criteria: Baseline
    --> Logs:
            Inputs may not be compatible for simulation. Test results might not be accurate. Click here for more information on external input mapping.
    --> Simulink Test Manager Results:
            Results: 2018-Nov-13 13:07:03/f14ParameterSweepTest/New Test Suite 1/Iterations Parameter Sweep/Scripted_Iteration19
================================================================================
.. ...
================================================================================
Verification failed in f14ParameterSweepTest > New Test Suite 1/Iterations Parameter Sweep(ScriptedIteration=Scripted_Iteration24).

    ---------------------
    Framework Diagnostic:
    ---------------------
    Failed criteria: Baseline
    --> Logs:
            Inputs may not be compatible for simulation. Test results might not be accurate. Click here for more information on external input mapping.
    --> Simulink Test Manager Results:
            Results: 2018-Nov-13 13:07:03/f14ParameterSweepTest/New Test Suite 1/Iterations Parameter Sweep/Scripted_Iteration24
================================================================================
.
================================================================================
Verification failed in f14ParameterSweepTest > New Test Suite 1/Iterations Parameter Sweep(ScriptedIteration=Scripted_Iteration25).

    ---------------------
    Framework Diagnostic:
    ---------------------
    Failed criteria: Baseline
    --> Logs:
            Inputs may not be compatible for simulation. Test results might not be accurate. Click here for more information on external input mapping.
    --> Simulink Test Manager Results:
            Results: 2018-Nov-13 13:07:03/f14ParameterSweepTest/New Test Suite 1/Iterations Parameter Sweep/Scripted_Iteration25
================================================================================
.
Done f14ParameterSweepTest > New Test Suite 1
__________

Failure Summary:

     Name                                                                                                         Failed  Incomplete  Reason(s)
    ==========================================================================================================================================================
     f14ParameterSweepTest > New Test Suite 1/Iterations Parameter Sweep(ScriptedIteration=Scripted_Iteration19)    X                 Failed by verification.
    ----------------------------------------------------------------------------------------------------------------------------------------------------------
     f14ParameterSweepTest > New Test Suite 1/Iterations Parameter Sweep(ScriptedIteration=Scripted_Iteration24)    X                 Failed by verification.
    ----------------------------------------------------------------------------------------------------------------------------------------------------------
     f14ParameterSweepTest > New Test Suite 1/Iterations Parameter Sweep(ScriptedIteration=Scripted_Iteration25)    X                 Failed by verification.
Generating report. Please wait.
    Preparing content for the report.
    Adding content to the report.
    Writing report to file.
Report has been saved to: /private/var/folders/bm/6qgg87js1bb7fpr2p475bcwh0002wp/T/reports/index.html

Now when this test is run you will see the report generated with all its simulation data inspector based visual goodies, as well as other metadata specific to the Simulink Test. Here's one of the failures:

...zooming in you can see how the error between the actual and expected signals at around 20 seconds into the simulation was greater than the tolerance:

Now, you may want to save the resulting html in your Jenkins builds. You can, but you will either need to host your own web server to serve up the resulting html, or configure your Jenkins environment and leverage the HTML Publisher Plugin of Jenkins. The only issue with this is that you may need to make some changes to the Content Security Policy of your Jenkins server in order for these html reports to show correctly. You'll only want to do this if you have a good understanding of how your Jenkins server is configured and understand the security implications. A good overview of this is listed here. However, the html test report won't be rendered correctly with the default Jenkins policy due to its use of JavaScript and CSS.

If you are able to safely configure the policy, however, you can see the report directly from your Jenkins builds:

Isn't that lovely? Now you have a head start diagnosing the problem and getting the investigation into the right hands so you can focus on the amazing things you do!

Get the MATLAB code

Published with MATLAB® R2018b