Developer Zone

Advanced Software Development with MATLAB

Turn on the YAML Channel! 11

Posted by Andy Campbell,

Diagnostics. They are the secret sauce of an effective test framework and we certainly take our job seriously at providing diagnostics that are not a little too hot or a little too cold, but juuuuusssst the right information to present, especially when encountering a failure. The reason this is so important is because good diagnostics can save human time, which is typically far more precious than computer time.

Indeed, this is one of the reasons why CI systems are so valuable. They take away the human time required to build, qualify, and deploy software and delegate that job to a machine rather than compromising your productivity. However, when a failure occurs you then rely on diagnostics from the failure to give you some hint as to what is going on.

Let me show you an example of this. I went and downloaded the digraph file exchange submission to highlight this. Note, this submission has now been superseded by the fantastic graph functionality included in R2015b and later, but if you are not yet on that release you can certainly check it out. However, one reason I am including that is because it has a nice test suite included with it that we can run from your CI system.

Not yet set up with a CI system? Take a look at some of these other posts to get you going. If you haven't read them yet go from the bottom up on that category page.

Anyway, let's continue with Jenkins and show how the diagnostics currently show up with these diagraph tests. Here is the script I am asking MATLAB to run in the Jenkins project:

import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    suite = testsuite('unittest');
    runner = TestRunner.withTextOutput('Verbosity',3);
    % Add the TAP plugin
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    
    runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));
    results = runner.run(suite)
catch e
    disp(getReport(e,'extended'));
    exit(1);
end
exit;

Does it pass? Well when we run this job on the digraph submission in Jenkins here is what we get:

Alright. Not bad but it looks like we have a failure. Do we know why it failed? Not yet. For that, we need to drill into the "TAP Test Results" instead of the "Extended TAP Test Results" of the Jenkins TAP Plugin.

Here, we can finally dig into the failing test by clicking on it and getting to this page, which shows the diagnostics:

Whew. Alright it took us a couple clicks, but we made it. We can see pretty quickly that that test is validating against an incorrect error message id. Looking at the two ids, you can take a pretty good guess that this is probably not actually a real problem in the software, but may simply require a trivial test update since perhaps the id has changed across versions of MATLAB. No problem. Quick to find, quick to fix.

However, I think we can improve on how we got here. For one, there were just too many clicks needed. Also, there seems to be some whitespace formatting lost when the Jenkins plugin displayed the diagnostic values. In this case it may have just been easier to look at the tap file output directly:

1..51
not ok 1 - digraphAPITest/digraph
# ================================================================================
# Verification failed in digraphAPITest/digraph.
# 
#     ---------------------
#     Framework Diagnostic:
#     ---------------------
#     verifyError failed.
#     --> The function threw the wrong exception.
#         
#         Actual Exception:
#             'MATLAB:TooManyInputs'
#         Expected Exception:
#             'MATLAB:maxrhs'
#     
#     Actual Error Report:
#         Error using digraph
#         Too many input arguments.
#         
#         Error in digraphAPITest>@()digraph('foo') (line 14)
#                     testCase.verifyError(@()digraph('foo'),'MATLAB:maxrhs');
#     Evaluated Function:
#             @()digraph('foo')
# 
#     ------------------
#     Stack Information:
#     ------------------
#     In H:\Documents\MATLAB\unittest\digraphAPITest.m (digraphAPITest.digraph) at 14
# ================================================================================
# 
ok 2 - digraphAPITest/isempty_
ok 3 - digraphAPITest/isEquivalent
ok 4 - digraphAPITest/addVertex
ok 5 - digraphAPITest/removeVertex
ok 6 - digraphAPITest/hasVertex
ok 7 - digraphAPITest/assertVertex
ok 8 - digraphAPITest/addEdge
ok 9 - digraphAPITest/removeEdge
ok 10 - digraphAPITest/hasEdge
ok 11 - digraphAPITest/findall
ok 12 - digraphAPITest/hasCycle
ok 13 - digraphAPITest/isComplete
ok 14 - digraphAPITest/shortestPath
ok 15 - digraphAPITest/subgraph
ok 16 - digraphAPITest/minimalEdges
ok 17 - digraphAPITest/transitiveClosure
ok 18 - digraphAPITest/union
ok 19 - digraphAPITest/intersect
ok 20 - digraphAPITest/sort
ok 21 - digraphAPITest/copy_
ok 22 - digraphAPITest/reset
ok 23 - digraphAPITest/spy
ok 24 - digraphFunctionalityTest/Vertex
ok 25 - digraphFunctionalityTest/Edge
ok 26 - digraphFunctionalityTest/unionBasic
ok 27 - digraphFunctionalityTest/unionAdvanced
ok 28 - digraphFunctionalityTest/intersectBasic
ok 29 - digraphFunctionalityTest/intersectAdvanced
ok 30 - digraphFunctionalityTest/hasPath
ok 31 - digraphFunctionalityTest/sort
ok 32 - digraphFunctionalityTest/shortestPathBasic
ok 33 - digraphFunctionalityTest/shortestPathAdvanced
ok 34 - digraphFunctionalityTest/hasCycle
ok 35 - digraphFunctionalityTest/CycleDoesNotTriggerInfiniteLoop
ok 36 - digraphFunctionalityTest/findall
ok 37 - digraphFunctionalityTest/subgraph
ok 38 - digraphFunctionalityTest/isEquivalent
ok 39 - digraphFunctionalityTest/copy_
ok 40 - digraphFunctionalityTest/isComplete
ok 41 - digraphFunctionalityTest/isempty_
ok 42 - digraphFunctionalityTest/transitiveClosureBasic
ok 43 - digraphFunctionalityTest/transitiveClosureAdvanced
ok 44 - digraphFunctionalityTest/transitiveClosureCycleAndComplete
ok 45 - digraphFunctionalityTest/minimalEdgesBasic
ok 46 - digraphFunctionalityTest/minimalEdgesAdvanced
ok 47 - digraphFunctionalityTest/minimalEdgesCycleAndComplete
ok 48 - digraphFunctionalityTest/trivialgraph
ok 49 - digraphFunctionalityTest/isEquivalentSubclass
ok 50 - digraphFunctionalityTest/subclassesArePreserved
ok 51 - digraphFunctionalityTest/spy

Well with R2016b there's an even better way. Did you ever wonder why the MATLAB TAPPlugin is created using the a call like the following?

import matlab.unittest.plugins.TAPPlugin;
plugin = TAPPlugin.producingOriginalFormat
plugin = 

  TAPOriginalFormatPlugin with properties:

    IncludePassingDiagnostics: 0
                    Verbosity: Terse
     ExcludeLoggedDiagnostics: 0

What is the deal with the original format? What other formats are there? Well, quick history lession, the TAP format was developed over a number of years by the Perl community, but the formal specification really began at what could be called version 12 (and what we have termed the original format since this version was the first with a documented specification). Since then, there has also arisen a version 13 format. Actually there is a group of people involved in the discussions who are discussing a version 14 as well but there has been no formal specification of that format yet. If you are interested, and if you want to join me in ensuring that any such format takes into account the needs of the MATLAB community, certainly do get involved. There is a github project and a slack channel discussing the format that you can take part in.

Anyway, the news is that R2016b includes the option to produce the version 13 TAP format!

plugin = TAPPlugin.producingVersion13
plugin = 

  TAPVersion13Plugin with properties:

    IncludePassingDiagnostics: 0
                    Verbosity: Terse
     ExcludeLoggedDiagnostics: 0

The largest takeaway in this format is the fact that diagnostics can be included in YAML(ish) blocks in the tap stream. You may or may not be aware, but:

Y AML

A in't

M arkup

L anguage

We all know how programmers love recursive acronyms (e.g, GNU, XNA). What does this mean for us? A better diagnostics experience. Let's see this same test failure in our Jenkins job but instead using version 13 of the TAP protocol.

import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    suite = testsuite('unittest');
    runner = TestRunner.withTextOutput('Verbosity',3);
    % Add the TAP plugin
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    
    runner.addPlugin(TAPPlugin.producingVersion13(ToFile(tapFile)));
    results = runner.run(suite)
catch e
    disp(getReport(e,'extended'));
    exit(1);
end
exit;

Using this produces the following TAP Stream

TAP version 13
1..51
not ok 1 - digraphAPITest/digraph
    ---
    Event:
        Event Name: 'VerificationFailed'
        Scope: 'digraphAPITest/digraph'
        Framework Diagnostic: |
            verifyError failed.
            --> The function threw the wrong exception.
                
                Actual Exception:
                    'MATLAB:TooManyInputs'
                Expected Exception:
                    'MATLAB:maxrhs'
            
            Actual Error Report:
                Error using digraph
                Too many input arguments.
                
                Error in digraphAPITest>@()digraph('foo') (line 14)
                            testCase.verifyError(@()digraph('foo'),'MATLAB:maxrhs');
            Evaluated Function:
                    @()digraph('foo')
        Stack: |
            In H:\Documents\MATLAB\unittest\digraphAPITest.m (digraphAPITest.digraph) at 14
    ...
ok 2 - digraphAPITest/isempty_
ok 3 - digraphAPITest/isEquivalent
ok 4 - digraphAPITest/addVertex
ok 5 - digraphAPITest/removeVertex
ok 6 - digraphAPITest/hasVertex
ok 7 - digraphAPITest/assertVertex
ok 8 - digraphAPITest/addEdge
ok 9 - digraphAPITest/removeEdge
ok 10 - digraphAPITest/hasEdge
ok 11 - digraphAPITest/findall
ok 12 - digraphAPITest/hasCycle
ok 13 - digraphAPITest/isComplete
ok 14 - digraphAPITest/shortestPath
ok 15 - digraphAPITest/subgraph
ok 16 - digraphAPITest/minimalEdges
ok 17 - digraphAPITest/transitiveClosure
ok 18 - digraphAPITest/union
ok 19 - digraphAPITest/intersect
ok 20 - digraphAPITest/sort
ok 21 - digraphAPITest/copy_
ok 22 - digraphAPITest/reset
ok 23 - digraphAPITest/spy
ok 24 - digraphFunctionalityTest/Vertex
ok 25 - digraphFunctionalityTest/Edge
ok 26 - digraphFunctionalityTest/unionBasic
ok 27 - digraphFunctionalityTest/unionAdvanced
ok 28 - digraphFunctionalityTest/intersectBasic
ok 29 - digraphFunctionalityTest/intersectAdvanced
ok 30 - digraphFunctionalityTest/hasPath
ok 31 - digraphFunctionalityTest/sort
ok 32 - digraphFunctionalityTest/shortestPathBasic
ok 33 - digraphFunctionalityTest/shortestPathAdvanced
ok 34 - digraphFunctionalityTest/hasCycle
ok 35 - digraphFunctionalityTest/CycleDoesNotTriggerInfiniteLoop
ok 36 - digraphFunctionalityTest/findall
ok 37 - digraphFunctionalityTest/subgraph
ok 38 - digraphFunctionalityTest/isEquivalent
ok 39 - digraphFunctionalityTest/copy_
ok 40 - digraphFunctionalityTest/isComplete
ok 41 - digraphFunctionalityTest/isempty_
ok 42 - digraphFunctionalityTest/transitiveClosureBasic
ok 43 - digraphFunctionalityTest/transitiveClosureAdvanced
ok 44 - digraphFunctionalityTest/transitiveClosureCycleAndComplete
ok 45 - digraphFunctionalityTest/minimalEdgesBasic
ok 46 - digraphFunctionalityTest/minimalEdgesAdvanced
ok 47 - digraphFunctionalityTest/minimalEdgesCycleAndComplete
ok 48 - digraphFunctionalityTest/trivialgraph
ok 49 - digraphFunctionalityTest/isEquivalentSubclass
ok 50 - digraphFunctionalityTest/subclassesArePreserved
ok 51 - digraphFunctionalityTest/spy

Very nice, we are in business with roughly the same failure diagnostics as the original format. However, the big win is that now that the diagnostics are in a structured format, other tools (like Jenkins) can represent them with much more flavor. This is now what the "TAP Extended Test Results" pages looks like in Jenkins.:

Now we can see all of the failure diagnostics on a single test results page with much more clarity and structure. Pretty sweet.

Do you see yourself benefitting from the more structured diagnostics that we get with version 13 of the TAP format? Do you have any stories to tell where good diagnostics saved you a bunch of debugging time? Do tell!


Get the MATLAB code

Published with MATLAB® R2016b

11 CommentsOldest to Newest

Additional Ways to Consume the Secret Sauce » Developer Zone replied on : 2 of 11

[…] 29 Sep Turn on the YAML Channel! […]

Rafael Sanchez replied on : 3 of 11

Hello, Andy

I am using the TAPPlugin in 15B.
I tried to use TAPPlugin.producingVersion13, without realising this was undocumented.
The result I got was

1..532
ok 1 – AbsTimeSignalSetTest/constructWithDT
ok 2 – AbsTimeSignalSetTest/constructWithDTAndStartTime


ok 425 – TimeSignalSetTest/hasSignalFalse
not ok 426 – TimeSignalSetTest/expectedToFail
# ================================================================================
# Error occurred in TimeSignalSetTest/expectedToFail and it did not run to completion.
#
# ————–
# Error Details:
# ————–
# Error using TimeSignalSetTest/expectedToFail (line 83)
# oops
#
# ================================================================================
#
ok 427 – TimeSignalSetTest/constructEmpty
etc

Like it would be with usingOriginalFormat.

Is this what is expected to happen, since it is, well, undocumented?

Thanks

Andy Campbell replied on : 4 of 11

Yes good point. It was indeed an undocumented method of the TAP plugin in 15b, and strictly speaking it does produce valid version 13 output. You should have one slight difference in the tap output, which is there should be a “TAP version 13” statement at the top of the stream.

Anyway, even though technically this is a valid tap version 13 stream, it doesn’t really give you anything over the original format because the YAML diagnostics are not produced by the framework. In 16b however, we did indeed produce these more structured YAML diagnostics so now it is actually a meaningful feature worth documenting.

Great question!

Rafael Sanchez replied on : 5 of 11

Thanks for the quick response, Andy, I saw that line, but it got lost on the copy-paste…

I guess we’ll have to do with this for now.

Michael replied on : 6 of 11

Thanks for your article – thats the first thing that I will do when I go back to work in 2017. Keep going concerning CI. I was so impressed how easy the whole thing is with the TAP files and how fruitful this is for us.

Andy Campbell replied on : 7 of 11

Sounds good @Michael, I am glad to hear that it is working well for you and improving your workflows!

Christian Heigele replied on : 8 of 11

Hi there

We are also using Matlabs CI functionality to assure our code quality and your blog posts are quite helpful. However I still have the problem that in my CI system (we are using team city) I always have a runtime of <1ms which is just wrong. Is there any possibility to expose this information with the TAP Plugins? On your screenshot it also looks like there is no information provided to jenkins as well, so I doubt that its a problem in team citys TAP plugin.

Andy Campbell replied on : 9 of 11

Hi Christian,

The TAP stream does not actually contain any duration information. Interestingly, the Jenkins TAPPlugin has built into it a way to show this information if you include a special field in the YAML tap stream. I think it is something like duration_ms, which sounds like duration in milliseconds, but actually is duration in seconds. Investigating this approach of the Jenkins plugin showed that it is not really a production grade feature, and actually it would only apply to Jenkins anyway. TeamCity actually doesn’t have any such support in its TAP plugin at all, and our tap stream doesn’t include the information.

One option you have is to use the JUnit format instead of TAP. There are tradeoffs here. For example, with the JUnit format you don’t get:

1) Diagnostics for tests filtered by assumptions
2) Passing diagnostics
3) Diagnostics logged using testCase.log
4) Distinction between the “type” of the failure (e.g. assertion vs verification failures)

However, you do indeed get the duration. Check out this post if you want to try it out:

http://blogs.mathworks.com/developer/2015/11/03/tap-diagnostics-and-junit-xml/

Add A Comment

What is 5 + 8?

Preview: hide