# Just TAP it in…give it a little TAP-py TAP TAP TAP-aroo23

Posted by Andy Campbell,

Last post we explored connecting MATLAB to Jenkins, running tests, and ensuring the build fails upon any test failure. We leveraged the TestResult array returned from the test run in order to exit MATLAB with a non-zero exit status code to communicate to Jenkins when needed that the build should fail.

This will work but there is a better way if you have R2014a or later. To truly integrate with Jenkins, MATLAB and Jenkins need to be communicating more fully with each other. A great way to do that is to leverage the standard, language independent Test Anything Protocol (TAP). Using the TAP Plugin for Jenkins and the TAPPlugin for the MATLAB TestRunner, MATLAB can give more detailed information to Jenkins. This information includes which tests were run, which failed, and which were filtered. Furthermore, other Jenkins features, such as history tracking, are enabled.

Enough already! How is this done?

Set up the TAP Plugin in Jenkins

Firstly, Jenkins needs the TAP Plugin in order to interpret TAP streams. This is done from the Jenkins plugin manager and can be found easily by searching for "TAP":

Once installed a new "Publish TAP Results" option is available as a post-build action.

After pointing Jenkins to the location where the tap streams will be generated, we can further configure the plugin behavior as we see fit. For example, we can fail the build if no tests are run or if the number of tests differs from the TAP Plan. Importantly, we need to check the option to fail the build if there are any test failures. Here we are picking up any files with a ".tap" extension, and we are failing the build if there are any test failures, if no tests are run, or if the TAP Plan is not complete.

Set up the TAPPlugin in MATLAB

From the last post we are already connected and running the tests, now we just need that particular test run to output a TAP stream that will be picked up and analyzed by Jenkins. To do this we can create our own TestRunner and add the TAP plugin to it. One consideration, however, is that we don't want the test output produced in MATLAB to get in the way of the TAP stream and invalidate it so we need to ensure the tap stream is directed somewhere other than the Command Window. We also need the TAP stream to output to a file that the Jenkins plugin can find for analysis. Both of these issues are conveniently solved by sending the TAPPlugin output to a ToFile stream. The test running script becomes:

import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
suite = TestSuite.fromPackage('testcases','IncludingSubpackages',true);
% Create a typical runner with text output
runner = TestRunner.withTextOutput();
% Add the TAP plugin and direct its output to a file
tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
% Run the tests
results = runner.run(suite);
display(results);
catch e
disp(getReport(e,'extended'));
exit(1);
end
exit;


Running the Jenkins build now fails the build in the presence of these test failures:

Enjoy the fruit of your labor

Jenkins will now also give you in depth insight into which tests passed, failed, or were filtered to you can quickly navigate to the failure syndrome and begin investigating.

In addition, many Jenkins/TAP-Plugin features add improved visibility into the health of your software over time, such as a history of the passing tests, failed tests, and filtered tests over time listed on the project page:

Here you can see a history of the test suite growth and health as it pertains to failures and filtered test content. For example, if we fix the failures in this test suite and run a new job we see the red disappear.

Likewise if we then do the needed work to unfilter the test content not yet running...¡Hasta la vista al amarillo!

Note that there are many different ways to setup your CI system and this is just one quick example. That said, if you use this approach you will want to also make sure you clean up your workspace between builds to remove these TAP files produced so that each build starts with fresh results, otherwise the TAP files from previous jobs will wreak havoc with those from the current job.

What do you think? Have you used the language independent TAP format for test results before? How else do you fit MATLAB tests with the rest of your ecosystem? Let us know in the comments!

Get the MATLAB code

Published with MATLAB® R2014b

Aditya Shah replied on : 1 of 23

Great post.
We’re using the TAP plugin with good success. However are there plans to add more capabilities to the TAP output? Such as diagnostic output (see http://testanything.org/tap-specification.html)? The reason I ask is that it would be nice to provide some more detail regarding tests that do fail with things like the test output that did not meet the criteria.

As always, thanks for considering the feedback.

Andy Campbell replied on : 2 of 23

That sounds great Aditya, I am glad you are already getting good results from it. Yes we are definitely considering additional TAP related enhancements for a future release such as including test diagnostics in the tap stream. It is good to hear your request for the feature, thanks!

Daniell Algar replied on : 3 of 23

The TAP format was new for me, great information!
I’d like to add another approach of how to present the test results. I work with Active Safety (in automotive) and we often a have a need to present much more information about each test than a pass/fail. For this we include a Publish-section in each test script that allows the user to do any kind of plotting and visualization. We then publish this with the Matlab publish() command, giving us a test report in .html that we can publish in Jenkins through the HTML Publisher plugin. We then add an hyperlink on the Description column of the Jenkins view (each test is a separate Jenkins job). We append the new report to the previous runs to get the report span over version history. It’s a neat way to keep track of the progress of each test.

Great new blog! Post often ;). Thank you.

Aditya Shah replied on : 4 of 23

Daniell,
That is very interesting – I may experiment and try something similar to include plots, tables, etc. in the output. I assume keeping it in html (versus a pdf from the report generator and then archiving it) makes it easy to integrate and view from within the Jenkins dashboard. Really good idea.

Aditya Shah replied on : 5 of 23

Andy,
Would you consider providing the end-user with capabilities to do that kind of extension in addition to adding it as a feature of a new release?

Andy Campbell replied on : 6 of 23

Yes this sounds like a great use of MATLAB’s publishing technology. We can definitely look for ways to incorporate such type of visualization features into a test run. It would be nice to capture the figures/output/etc produced in such a way that we:

a) Don’t have to rewrite the test twice, once for visualization and once for the actual regression test
b) Can still ensure that the tests clean up after themselves correctly so as to avoid polluting the state of MATLAB for subsequent tests.

Food for thought, we can certainly consider building something for this workflow in the future.

Oleg Komarov replied on : 7 of 23

I wonder if TMW could set up a FREE service to run tests on e.g. Github pull requests.
My line of thought follows the direction that TMW is taking, i.e. integration with source control and github (for the FEX) and the creation of the Developer Zone blog. I think that serious collaborative software development cannot happen without web-based source code management like Github. Hence, such a free service would greatly promote Matlab among developers.

I am one of the contributors to a big FEX submission, and we are running continuous integration with Travis CI but on Octave. It would be great if we could take advantage of aforementioned service.

Andy Campbell replied on : 8 of 23

Hey Oleg,

Great suggestion! Something like that would indeed help promote collaborative use of MATLAB and as it seems you know, more collaboration always yields great benefits.

As you no doubt see we are very interested in this type of enabling technology for serious software development with MATLAB, and the hard part now is just making it happen and figuring out all of the nuts and bolts. In the meantime, you can certainly store the MATLAB code on a site like GitHub and set up your environment to listen to scm changes to run those tests. I realize you are asking for more but at least some of those building blocks are there for you. Also, there are services like CloudBees (https://www.cloudbees.com/) that may help in the meantime as well.

Narendra replied on : 9 of 23

Hi,
I also got the CI to work with matlab. THis is a great boon since we now have a way to ensure some quality of the code. Is there any way to autoindent the code after checkin? THis way I can ensure that all our code meets the standards.

Andy Campbell replied on : 10 of 23

Hello Narendra,

You may want to look into adding a build step that launches MATLAB and uses the editor services to smart indent all of the MATLAB files in the change. For example:

doc = matlab.desktop.editor.openDocument(‘foo.m’);
doc.smartIndentContents
doc.save

You just need to figure out which files are changing and do this operation on them all. I would suggest thinking it through though. For example, you’ll want be sure that you are OK with your CI system making such changes instead of a human. What if you _really_ want the specific indenting for a certain file for example? This might be a big hammer. Also, the modified file will need to be checked out and checked in by the CI system which may be problematic.

Another strategy is implement a test which looks at all the the files in the source tree and compares them against their smart indented counterparts. In this case you don’t even need to modify the files, you just can:

1) open them in the editor (and register them to close with a call to testCase.addTeardown)
2) get the text contents
3) smart indent
4) get the new text contents
5) Compare to ensure the files are correctly indented.

This is a bit better but still may result in fragile tests if you really want a particular file to be let go without the smart indenting. One more thought is that you can write these tests but keep them separate from your tests that actually fail the CI build. Then you can build into your CI system a warning system that highlights code that is not indented correctly but you don’t wish to actually fail the build.

Hopefully that helps give you some ideas, please do report back if you are able to use the editor services in some way to get this working!

Narendra replied on : 11 of 23

thank you for your suggestion. In the end I decided not to use it. I currently face couple of problems.
I would ideally like to run Jenkins on a separate box. But does this mean that we need to buy a new license of Matlab just to run tests? We dont want to spend that money just to run tests.
– Secondly is there a way to attach the tests that are actually broken in the email. Currently we need to go to the web page to check the actual tests that are broke, THis is not a big deal, but would be nice to have.

Andy Campbell replied on : 12 of 23

Hi Narendra,

For your first comment, I encourage you to connect with your account manager to see the licensing options you have with respect to working in a CI system. They are really the best person to talk to since they understand your needs and can help determine what fits your organization.

As for the second question, I don’t know of a way this is available out of the box, but it looks like the following Jenkins plugin can allow you to customize the email content:

https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin

Perhaps I can write a post showing this in action if that would help.

Denis replied on : 13 of 23

Thanks a lot for this great tool.
I was wondering if it is possible to expand the application to performance testing with the runperf commande in matlab?
Can Jenkins handle this type of results with TAP?
Thanks!

Andy Campbell replied on : 14 of 23

Hi @Denis,

Great question! There is currently no feature that helps to integrate your performance data & results directly with your jenkins build, but you can leverage ThingSpeak to store and view your performance trends pretty easily. The following post shows how you can get that started:

http://blogs.mathworks.com/developer/2016/06/30/performance-trends/

Note, you can even kick off a MATLAB as a build step that sends these messages to your ThingSpeak channel. Try that out and let me know how it goes. Perhaps I can show this as a blog post if you’d be interested in seeing the details. Let me know.

Also, we can look into whether we can produce the performance output in the form of something like the JMeter format, and then we could leverage the Jenkins Performance plugin to help in area. Is that something you would be interested in?

Narendra replied on : 15 of 23

Hi,
We have been using the tap files with great success. However what I would like to do is to parallelize the tests. However in running the tests in parallel breaks the tap generation process. Is there some way we could run the unit tests in parallel and still generate valid tap files?

Andy Campbell replied on : 16 of 23

Hi Narendra,

Great question! Really, we should provide a feature to make this workflow easier for you (I’ve taken note), but in the meantime you can create you’re own output stream to get you going so you don’t have to wait for us. Try out the following:

classdef ToParallelFiles < matlab.unittest.plugins.ToFile

properties(Transient)
FilenameModifier;
end

methods
function stream = ToParallelFiles(baseFilename)
%ToParallelFiles - Create an output stream to files in parallel
%   STREAM = ToParallelFiles(BASEFILENAME) creates an OutputStream that
%   writes text output to files with the root of BASEFILENAME. BASEFILENAME
%   is a string or character vector containing the base name of the file to
%   be written to. A unique identifier is added in order to faciliate many
%   files in parallel.
stream = stream@matlab.unittest.plugins.ToFile(baseFilename);
end
function print(stream, formatStr, varargin)
if isempty(stream.FilenameModifier)
[~, stream.FilenameModifier] = fileparts(tempname);
end

[folder, baseFilename, ext]  = fileparts(stream.Filename);
filename = fullfile(folder, [baseFilename '_' stream.FilenameModifier, ext]);

[fid, msg] = fopen(filename, 'a', 'n', 'UTF-8');
assert(fid > 0, 'ToParallelFiles:OpenFailed', msg);
cl = onCleanup(@() fclose(fid));
fprintf(fid, formatStr, varargin{:});
end
end
end


I put it together quickly so I am not guaranteeing the results here, but that should help get you going. Let me know how it works for you!

Andy

Narendra replied on : 17 of 23

Can you please add this feature into Matlab? I think this would be useful to others as well.

william replied on : 18 of 23

Hello Andy,
Thanks for the tutorial, I implemented a test, the first execution ran fine:

results =

1×71 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals:
53 Passed, 18 Failed, 18 Incomplete.
1805.4474 seconds testing time.

TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Saving reports…
Processing ‘/Users/Shared/Jenkins/Home/jobs/Run ALL THE TESTS/builds/14/tap-master-files/testResults.tap’
Parsing TAP test result [/Users/Shared/Jenkins/Home/jobs/Run ALL THE TESTS/builds/14/tap-master-files/testResults.tap].
There are failed test cases and the job is configured to mark the build as failure. Marking build as FAILURE
TAP Reports Processing: FINISH
Build step ‘Publish TAP Results’ changed build result to FAILURE
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: FAILURE

But then I get an error with duplicate TAP Plan found:
results =

1×71 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals:
53 Passed, 18 Failed, 18 Incomplete.
1852.955 seconds testing time.

TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Saving reports…
Processing ‘/Users/Shared/Jenkins/Home/jobs/Run ALL THE TESTS/builds/15/tap-master-files/testResults.tap’
Parsing TAP test result [/Users/Shared/Jenkins/Home/jobs/Run ALL THE TESTS/builds/15/tap-master-files/testResults.tap].
org.tap4j.parser.ParserException: Error parsing TAP Stream: Duplicated TAP Plan found.
at org.tap4j.parser.Tap13Parser.parseTapStream(Tap13Parser.java:230)
at org.tap4j.parser.Tap13Parser.parseFile(Tap13Parser.java:193)
at org.tap4j.plugin.TapParser.parse(TapParser.java:167)
at org.tap4j.plugin.TapPublisher.performImpl(TapPublisher.java:412)
at org.tap4j.plugin.TapPublisher.perform(TapPublisher.java:371)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676) at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild\$AbstractBuildExecution.post(AbstractBuild.java:621)
at hudson.model.Run.execute(Run.java:1760)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Caused by: org.tap4j.parser.ParserException: Duplicated TAP Plan found.
at org.tap4j.parser.Tap13Parser.parseLine(Tap13Parser.java:335)
at org.tap4j.parser.Tap13Parser.parseTapStream(Tap13Parser.java:224)
… 15 more
TAP parse errors found in the build. Marking build as UNSTABLE
TAP Reports Processing: FINISH
Build step ‘Publish TAP Results’ changed build result to UNSTABLE
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: UNSTABLE

Any idea how to fix that?
I’m using TAP plugin 2.1, Matlab 2016a

Andy Campbell replied on : 19 of 23

Hello @william,

It looks to me like you need to wipe out the TAP file from your previous build. What is happening is that the second build you are appending to the TAP file created from the prior build, which will result in the TAP exception that you are seeing. You can probably get going by using the Clean Workspace plugin.

Hope that helps!
Andy

Michael replied on : 20 of 23

Hello,
I have a question. In my continuous integration environment I would sometimes like to integrate not a test that is really testing something but I would just like to record some information. That could be e.g. hostname, RAM of hostname, Java heap Memory. I can achieve this by checking this.verifyLessThanOrEqual(javaHeapMemory,0) which will always fail and then I have the Information from the diagnostics. However it gives me a failed test which spoiles the blue-shining Jenkins :). Is there a way to do it without a failed test and still having the actual number of Java heap memeory ?

Thank you
Michael

William replied on : 21 of 23

Thanks @Andy, cleaning up after each execution the workspace removed the error.

Andy Campbell replied on : 22 of 23

Hi @Michael,

Couple thoughts:

1) This sounds somewhat similar to a performance test using the performance test framework. In this sense the tests are run not to produce pass/fail but to actually measure something being done in the test, in this case performance. What is interesting is not that this might solve your problem (at least not yet!) but that you are using the test writing apis to perform an experiment that returns some numerical measurements rather than pass/fail. It is just interesting, and it could possibly in the future be leveraged to get at what you are doing.

2) There are a few options today that you can get to what you are trying to do without causing a test failure in order to see the diagnostics. Probably the the method that is most immediately helpful is the

testCase.log

method (Here is the doc). This would allow you to control the diagnostic information you want to see based on a configuration of the TestRunner. For example, if you log these diagostics at the “Detailed” level and want these diagnostics to go to the MATLAB output then you can create a TestRunner with a LoggingPlugin at the Detailed level. You can also do this by using something like

runtests(...,'Verbosity','Detailed')

. What it important is that the diagnostics are shown based on a configuration of the TestRunner rather than a pass/fail result.

You can even keep the text output at exactly the same verbosity level (maybe you like the default), but record the diagnostics using the DiagnosticsRecordingPlugin at a higher level of verbosity and you will send this information to be include on the TestResults. From there you can save the results to a mat-file or something like that.

Another option is to write a plugin which displays the information you are looking for, but I wouldn’t be surprised if the log method gets the job done for you.

Hope that helps!
Andy

Michael replied on : 23 of 23

Hello Andy,
indeed it is working now. Thank you very much 🙂 This is my code (for any other interested) for the runner object

matlabRunner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingVersion13(matlab.unittest.plugins.ToFile(this.tapFile),'IncludingPassingDiagnostics',true,'ExcludingLoggedDiagnostics',false));

and this one of my test functions in a test class

function checkJavaHeapMemory(this)
jhm = java.lang.Runtime.getRuntime.maxMemory/1024/1024;
this.log(matlab.unittest.Verbosity.Terse,['Java Heap Memory is ' num2str(jhm)]);
end


The Information is now viewed in Jenkins.

Thanks again.

I will try now to view certain Information inside a table in Jenkins (e.g. maximal McCabe Complexity of single modules) as well as a trend graph of Number of ToDos in source code in Jenkins.