Developer Zone

Advanced Software Development with MATLAB

Just TAP it in…give it a little TAP-py TAP TAP TAP-aroo 17

Posted by Andy Campbell,

Last post we explored connecting MATLAB to Jenkins, running tests, and ensuring the build fails upon any test failure. We leveraged the TestResult array returned from the test run in order to exit MATLAB with a non-zero exit status code to communicate to Jenkins when needed that the build should fail.

This will work but there is a better way if you have R2014a or later. To truly integrate with Jenkins, MATLAB and Jenkins need to be communicating more fully with each other. A great way to do that is to leverage the standard, language independent Test Anything Protocol (TAP). Using the TAP Plugin for Jenkins and the TAPPlugin for the MATLAB TestRunner, MATLAB can give more detailed information to Jenkins. This information includes which tests were run, which failed, and which were filtered. Furthermore, other Jenkins features, such as history tracking, are enabled.

Enough already! How is this done?

Set up the TAP Plugin in Jenkins

Firstly, Jenkins needs the TAP Plugin in order to interpret TAP streams. This is done from the Jenkins plugin manager and can be found easily by searching for "TAP":

Once installed a new "Publish TAP Results" option is available as a post-build action.

After pointing Jenkins to the location where the tap streams will be generated, we can further configure the plugin behavior as we see fit. For example, we can fail the build if no tests are run or if the number of tests differs from the TAP Plan. Importantly, we need to check the option to fail the build if there are any test failures. Here we are picking up any files with a ".tap" extension, and we are failing the build if there are any test failures, if no tests are run, or if the TAP Plan is not complete.

Set up the TAPPlugin in MATLAB

From the last post we are already connected and running the tests, now we just need that particular test run to output a TAP stream that will be picked up and analyzed by Jenkins. To do this we can create our own TestRunner and add the TAP plugin to it. One consideration, however, is that we don't want the test output produced in MATLAB to get in the way of the TAP stream and invalidate it so we need to ensure the tap stream is directed somewhere other than the Command Window. We also need the TAP stream to output to a file that the Jenkins plugin can find for analysis. Both of these issues are conveniently solved by sending the TAPPlugin output to a ToFile stream. The test running script becomes:

import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    suite = TestSuite.fromPackage('testcases','IncludingSubpackages',true);
    % Create a typical runner with text output
    runner = TestRunner.withTextOutput();
    % Add the TAP plugin and direct its output to a file
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));
    % Run the tests
    results = runner.run(suite);
    display(results);
catch e
    disp(getReport(e,'extended'));
    exit(1);
end
exit;

Running the Jenkins build now fails the build in the presence of these test failures:

Enjoy the fruit of your labor

Jenkins will now also give you in depth insight into which tests passed, failed, or were filtered to you can quickly navigate to the failure syndrome and begin investigating.

In addition, many Jenkins/TAP-Plugin features add improved visibility into the health of your software over time, such as a history of the passing tests, failed tests, and filtered tests over time listed on the project page:

Here you can see a history of the test suite growth and health as it pertains to failures and filtered test content. For example, if we fix the failures in this test suite and run a new job we see the red disappear.

Likewise if we then do the needed work to unfilter the test content not yet running...¡Hasta la vista al amarillo!

Note that there are many different ways to setup your CI system and this is just one quick example. That said, if you use this approach you will want to also make sure you clean up your workspace between builds to remove these TAP files produced so that each build starts with fresh results, otherwise the TAP files from previous jobs will wreak havoc with those from the current job.

What do you think? Have you used the language independent TAP format for test results before? How else do you fit MATLAB tests with the rest of your ecosystem? Let us know in the comments!


Get the MATLAB code

Published with MATLAB® R2014b

17 CommentsOldest to Newest

Aditya Shah replied on : 1 of 17

Great post.
We’re using the TAP plugin with good success. However are there plans to add more capabilities to the TAP output? Such as diagnostic output (see http://testanything.org/tap-specification.html)? The reason I ask is that it would be nice to provide some more detail regarding tests that do fail with things like the test output that did not meet the criteria.

As always, thanks for considering the feedback.

Andy Campbell replied on : 2 of 17

That sounds great Aditya, I am glad you are already getting good results from it. Yes we are definitely considering additional TAP related enhancements for a future release such as including test diagnostics in the tap stream. It is good to hear your request for the feature, thanks!

Daniell Algar replied on : 3 of 17

The TAP format was new for me, great information!
I’d like to add another approach of how to present the test results. I work with Active Safety (in automotive) and we often a have a need to present much more information about each test than a pass/fail. For this we include a Publish-section in each test script that allows the user to do any kind of plotting and visualization. We then publish this with the Matlab publish() command, giving us a test report in .html that we can publish in Jenkins through the HTML Publisher plugin. We then add an hyperlink on the Description column of the Jenkins view (each test is a separate Jenkins job). We append the new report to the previous runs to get the report span over version history. It’s a neat way to keep track of the progress of each test.

Great new blog! Post often ;). Thank you.

Aditya Shah replied on : 4 of 17

Daniell,
That is very interesting – I may experiment and try something similar to include plots, tables, etc. in the output. I assume keeping it in html (versus a pdf from the report generator and then archiving it) makes it easy to integrate and view from within the Jenkins dashboard. Really good idea.

Aditya Shah replied on : 5 of 17

Andy,
Would you consider providing the end-user with capabilities to do that kind of extension in addition to adding it as a feature of a new release?

Andy Campbell replied on : 6 of 17

Daniell/Aditya,

Yes this sounds like a great use of MATLAB’s publishing technology. We can definitely look for ways to incorporate such type of visualization features into a test run. It would be nice to capture the figures/output/etc produced in such a way that we:

a) Don’t have to rewrite the test twice, once for visualization and once for the actual regression test
b) Can still ensure that the tests clean up after themselves correctly so as to avoid polluting the state of MATLAB for subsequent tests.

Food for thought, we can certainly consider building something for this workflow in the future.

Oleg Komarov replied on : 7 of 17

I wonder if TMW could set up a FREE service to run tests on e.g. Github pull requests.
My line of thought follows the direction that TMW is taking, i.e. integration with source control and github (for the FEX) and the creation of the Developer Zone blog. I think that serious collaborative software development cannot happen without web-based source code management like Github. Hence, such a free service would greatly promote Matlab among developers.

I am one of the contributors to a big FEX submission, and we are running continuous integration with Travis CI but on Octave. It would be great if we could take advantage of aforementioned service.

Andy Campbell replied on : 8 of 17

Hey Oleg,

Great suggestion! Something like that would indeed help promote collaborative use of MATLAB and as it seems you know, more collaboration always yields great benefits.

As you no doubt see we are very interested in this type of enabling technology for serious software development with MATLAB, and the hard part now is just making it happen and figuring out all of the nuts and bolts. In the meantime, you can certainly store the MATLAB code on a site like GitHub and set up your environment to listen to scm changes to run those tests. I realize you are asking for more but at least some of those building blocks are there for you. Also, there are services like CloudBees (https://www.cloudbees.com/) that may help in the meantime as well.

Narendra replied on : 9 of 17

Hi,
I also got the CI to work with matlab. THis is a great boon since we now have a way to ensure some quality of the code. Is there any way to autoindent the code after checkin? THis way I can ensure that all our code meets the standards.

Andy Campbell replied on : 10 of 17

Hello Narendra,

You may want to look into adding a build step that launches MATLAB and uses the editor services to smart indent all of the MATLAB files in the change. For example:

doc = matlab.desktop.editor.openDocument(‘foo.m’);
doc.smartIndentContents
doc.save

You just need to figure out which files are changing and do this operation on them all. I would suggest thinking it through though. For example, you’ll want be sure that you are OK with your CI system making such changes instead of a human. What if you _really_ want the specific indenting for a certain file for example? This might be a big hammer. Also, the modified file will need to be checked out and checked in by the CI system which may be problematic.

Another strategy is implement a test which looks at all the the files in the source tree and compares them against their smart indented counterparts. In this case you don’t even need to modify the files, you just can:

1) open them in the editor (and register them to close with a call to testCase.addTeardown)
2) get the text contents
3) smart indent
4) get the new text contents
5) Compare to ensure the files are correctly indented.

This is a bit better but still may result in fragile tests if you really want a particular file to be let go without the smart indenting. One more thought is that you can write these tests but keep them separate from your tests that actually fail the CI build. Then you can build into your CI system a warning system that highlights code that is not indented correctly but you don’t wish to actually fail the build.

Hopefully that helps give you some ideas, please do report back if you are able to use the editor services in some way to get this working!

Narendra replied on : 11 of 17

thank you for your suggestion. In the end I decided not to use it. I currently face couple of problems.
I would ideally like to run Jenkins on a separate box. But does this mean that we need to buy a new license of Matlab just to run tests? We dont want to spend that money just to run tests.
– Secondly is there a way to attach the tests that are actually broken in the email. Currently we need to go to the web page to check the actual tests that are broke, THis is not a big deal, but would be nice to have.

Andy Campbell replied on : 12 of 17

Hi Narendra,

For your first comment, I encourage you to connect with your account manager to see the licensing options you have with respect to working in a CI system. They are really the best person to talk to since they understand your needs and can help determine what fits your organization.

As for the second question, I don’t know of a way this is available out of the box, but it looks like the following Jenkins plugin can allow you to customize the email content:

https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin

Perhaps I can write a post showing this in action if that would help.

Denis replied on : 13 of 17

Thanks a lot for this great tool.
I was wondering if it is possible to expand the application to performance testing with the runperf commande in matlab?
Can Jenkins handle this type of results with TAP?
Thanks!

Andy Campbell replied on : 14 of 17

Hi @Denis,

Great question! There is currently no feature that helps to integrate your performance data & results directly with your jenkins build, but you can leverage ThingSpeak to store and view your performance trends pretty easily. The following post shows how you can get that started:

http://blogs.mathworks.com/developer/2016/06/30/performance-trends/

Note, you can even kick off a MATLAB as a build step that sends these messages to your ThingSpeak channel. Try that out and let me know how it goes. Perhaps I can show this as a blog post if you’d be interested in seeing the details. Let me know.

Also, we can look into whether we can produce the performance output in the form of something like the JMeter format, and then we could leverage the Jenkins Performance plugin to help in area. Is that something you would be interested in?

Narendra replied on : 15 of 17

Hi,
We have been using the tap files with great success. However what I would like to do is to parallelize the tests. However in running the tests in parallel breaks the tap generation process. Is there some way we could run the unit tests in parallel and still generate valid tap files?

Hi Narendra,

Great question! Really, we should provide a feature to make this workflow easier for you (I’ve taken note), but in the meantime you can create you’re own output stream to get you going so you don’t have to wait for us. Try out the following:

classdef ToParallelFiles < matlab.unittest.plugins.ToFile

    properties(Transient)
        FilenameModifier;
    end

    
    methods
        function stream = ToParallelFiles(baseFilename)
            %ToParallelFiles - Create an output stream to files in parallel
            %   STREAM = ToParallelFiles(BASEFILENAME) creates an OutputStream that
            %   writes text output to files with the root of BASEFILENAME. BASEFILENAME
            %   is a string or character vector containing the base name of the file to
            %   be written to. A unique identifier is added in order to faciliate many
            %   files in parallel.
            stream = stream@matlab.unittest.plugins.ToFile(baseFilename);
        end
        function print(stream, formatStr, varargin)
            if isempty(stream.FilenameModifier)
                [~, stream.FilenameModifier] = fileparts(tempname);
            end
            
            [folder, baseFilename, ext]  = fileparts(stream.Filename);
            filename = fullfile(folder, [baseFilename '_' stream.FilenameModifier, ext]);
                
            [fid, msg] = fopen(filename, 'a', 'n', 'UTF-8');
            assert(fid > 0, 'ToParallelFiles:OpenFailed', msg);
            cl = onCleanup(@() fclose(fid));
            fprintf(fid, formatStr, varargin{:});
        end
    end
end

I put it together quickly so I am not guaranteeing the results here, but that should help get you going. Let me know how it works for you!

Andy

Narendra replied on : 17 of 17

Can you please add this feature into Matlab? I think this would be useful to others as well.

Add A Comment

What is 6 + 10?

Preview: hide