Last post we explored connecting MATLAB to Jenkins, running tests, and ensuring the build fails upon any test failure. We leveraged the TestResult array returned from the test run in order to exit MATLAB with a non-zero exit status code to communicate to Jenkins when needed that the build should fail.

This will work but there is a better way if you have R2014a or later. To truly integrate with Jenkins, MATLAB and Jenkins need to be communicating more fully with each other. A great way to do that is to leverage the standard, language independent Test Anything Protocol (TAP). Using the TAP Plugin for Jenkins and the TAPPlugin for the MATLAB TestRunner, MATLAB can give more detailed information to Jenkins. This information includes which tests were run, which failed, and which were filtered. Furthermore, other Jenkins features, such as history tracking, are enabled.

Enough already! How is this done?

Set up the TAP Plugin in Jenkins

Firstly, Jenkins needs the TAP Plugin in order to interpret TAP streams. This is done from the Jenkins plugin manager and can be found easily by searching for "TAP":

Once installed a new "Publish TAP Results" option is available as a post-build action.

After pointing Jenkins to the location where the tap streams will be generated, we can further configure the plugin behavior as we see fit. For example, we can fail the build if no tests are run or if the number of tests differs from the TAP Plan. Importantly, we need to check the option to fail the build if there are any test failures. Here we are picking up any files with a ".tap" extension, and we are failing the build if there are any test failures, if no tests are run, or if the TAP Plan is not complete.

Set up the TAPPlugin in MATLAB

From the last post we are already connected and running the tests, now we just need that particular test run to output a TAP stream that will be picked up and analyzed by Jenkins. To do this we can create our own TestRunner and add the TAP plugin to it. One consideration, however, is that we don't want the test output produced in MATLAB to get in the way of the TAP stream and invalidate it so we need to ensure the tap stream is directed somewhere other than the Command Window. We also need the TAP stream to output to a file that the Jenkins plugin can find for analysis. Both of these issues are conveniently solved by sending the TAPPlugin output to a ToFile stream. The test running script becomes:

import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    suite = TestSuite.fromPackage('testcases','IncludingSubpackages',true);
    % Create a typical runner with text output
    runner = TestRunner.withTextOutput();
    % Add the TAP plugin and direct its output to a file
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));
    % Run the tests
    results = runner.run(suite);
    display(results);
catch e
    disp(getReport(e,'extended'));
    exit(1);
end
exit;

Running the Jenkins build now fails the build in the presence of these test failures:

Enjoy the fruit of your labor

Jenkins will now also give you in depth insight into which tests passed, failed, or were filtered to you can quickly navigate to the failure syndrome and begin investigating.

In addition, many Jenkins/TAP-Plugin features add improved visibility into the health of your software over time, such as a history of the passing tests, failed tests, and filtered tests over time listed on the project page:

Here you can see a history of the test suite growth and health as it pertains to failures and filtered test content. For example, if we fix the failures in this test suite and run a new job we see the red disappear.

Likewise if we then do the needed work to unfilter the test content not yet running...¡Hasta la vista al amarillo!

Note that there are many different ways to setup your CI system and this is just one quick example. That said, if you use this approach you will want to also make sure you clean up your workspace between builds to remove these TAP files produced so that each build starts with fresh results, otherwise the TAP files from previous jobs will wreak havoc with those from the current job.

What do you think? Have you used the language independent TAP format for test results before? How else do you fit MATLAB tests with the rest of your ecosystem? Let us know in the comments!

Get the MATLAB code

Published with MATLAB® R2014b