Turn on the YAML Channel!

Posted by Andy Campbell, September 29, 2016

3 views (last 30 days) | 0 Likes | 12 comments

Diagnostics. They are the secret sauce of an effective test framework and we certainly take our job seriously at providing diagnostics that are not a little too hot or a little too cold, but juuuuusssst the right information to present, especially when encountering a failure. The reason this is so important is because good diagnostics can save human time, which is typically far more precious than computer time.

Indeed, this is one of the reasons why CI systems are so valuable. They take away the human time required to build, qualify, and deploy software and delegate that job to a machine rather than compromising your productivity. However, when a failure occurs you then rely on diagnostics from the failure to give you some hint as to what is going on.

Let me show you an example of this. I went and downloaded the digraph file exchange submission to highlight this. Note, this submission has now been superseded by the fantastic graph functionality included in R2015b and later, but if you are not yet on that release you can certainly check it out. However, one reason I am including that is because it has a nice test suite included with it that we can run from your CI system.

Not yet set up with a CI system? Take a look at some of these other posts to get you going. If you haven't read them yet go from the bottom up on that category page.

Anyway, let's continue with Jenkins and show how the diagnostics currently show up with these diagraph tests. Here is the script I am asking MATLAB to run in the Jenkins project:

import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    suite = testsuite('unittest');
    runner = TestRunner.withTextOutput('Verbosity',3);
    % Add the TAP plugin
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    
    runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));
    results = runner.run(suite)
catch e
    disp(getReport(e,'extended'));
    exit(1);
end
exit;

Does it pass? Well when we run this job on the digraph submission in Jenkins here is what we get:

Alright. Not bad but it looks like we have a failure. Do we know why it failed? Not yet. For that, we need to drill into the "TAP Test Results" instead of the "Extended TAP Test Results" of the Jenkins TAP Plugin.

Here, we can finally dig into the failing test by clicking on it and getting to this page, which shows the diagnostics:

Whew. Alright it took us a couple clicks, but we made it. We can see pretty quickly that that test is validating against an incorrect error message id. Looking at the two ids, you can take a pretty good guess that this is probably not actually a real problem in the software, but may simply require a trivial test update since perhaps the id has changed across versions of MATLAB. No problem. Quick to find, quick to fix.

However, I think we can improve on how we got here. For one, there were just too many clicks needed. Also, there seems to be some whitespace formatting lost when the Jenkins plugin displayed the diagnostic values. In this case it may have just been easier to look at the tap file output directly:

1..51
not ok 1 - digraphAPITest/digraph
# ================================================================================
# Verification failed in digraphAPITest/digraph.
# 
#     ---------------------
#     Framework Diagnostic:
#     ---------------------
#     verifyError failed.
#     --> The function threw the wrong exception.
#         
#         Actual Exception:
#             'MATLAB:TooManyInputs'
#         Expected Exception:
#             'MATLAB:maxrhs'
#     
#     Actual Error Report:
#         Error using digraph
#         Too many input arguments.
#         
#         Error in digraphAPITest>@()digraph('foo') (line 14)
#                     testCase.verifyError(@()digraph('foo'),'MATLAB:maxrhs');
#     Evaluated Function:
#             @()digraph('foo')
# 
#     ------------------
#     Stack Information:
#     ------------------
#     In H:\Documents\MATLAB\unittest\digraphAPITest.m (digraphAPITest.digraph) at 14
# ================================================================================
# 
ok 2 - digraphAPITest/isempty_
ok 3 - digraphAPITest/isEquivalent
ok 4 - digraphAPITest/addVertex
ok 5 - digraphAPITest/removeVertex
ok 6 - digraphAPITest/hasVertex
ok 7 - digraphAPITest/assertVertex
ok 8 - digraphAPITest/addEdge
ok 9 - digraphAPITest/removeEdge
ok 10 - digraphAPITest/hasEdge
ok 11 - digraphAPITest/findall
ok 12 - digraphAPITest/hasCycle
ok 13 - digraphAPITest/isComplete
ok 14 - digraphAPITest/shortestPath
ok 15 - digraphAPITest/subgraph
ok 16 - digraphAPITest/minimalEdges
ok 17 - digraphAPITest/transitiveClosure
ok 18 - digraphAPITest/union
ok 19 - digraphAPITest/intersect
ok 20 - digraphAPITest/sort
ok 21 - digraphAPITest/copy_
ok 22 - digraphAPITest/reset
ok 23 - digraphAPITest/spy
ok 24 - digraphFunctionalityTest/Vertex
ok 25 - digraphFunctionalityTest/Edge
ok 26 - digraphFunctionalityTest/unionBasic
ok 27 - digraphFunctionalityTest/unionAdvanced
ok 28 - digraphFunctionalityTest/intersectBasic
ok 29 - digraphFunctionalityTest/intersectAdvanced
ok 30 - digraphFunctionalityTest/hasPath
ok 31 - digraphFunctionalityTest/sort
ok 32 - digraphFunctionalityTest/shortestPathBasic
ok 33 - digraphFunctionalityTest/shortestPathAdvanced
ok 34 - digraphFunctionalityTest/hasCycle
ok 35 - digraphFunctionalityTest/CycleDoesNotTriggerInfiniteLoop
ok 36 - digraphFunctionalityTest/findall
ok 37 - digraphFunctionalityTest/subgraph
ok 38 - digraphFunctionalityTest/isEquivalent
ok 39 - digraphFunctionalityTest/copy_
ok 40 - digraphFunctionalityTest/isComplete
ok 41 - digraphFunctionalityTest/isempty_
ok 42 - digraphFunctionalityTest/transitiveClosureBasic
ok 43 - digraphFunctionalityTest/transitiveClosureAdvanced
ok 44 - digraphFunctionalityTest/transitiveClosureCycleAndComplete
ok 45 - digraphFunctionalityTest/minimalEdgesBasic
ok 46 - digraphFunctionalityTest/minimalEdgesAdvanced
ok 47 - digraphFunctionalityTest/minimalEdgesCycleAndComplete
ok 48 - digraphFunctionalityTest/trivialgraph
ok 49 - digraphFunctionalityTest/isEquivalentSubclass
ok 50 - digraphFunctionalityTest/subclassesArePreserved
ok 51 - digraphFunctionalityTest/spy

Well with R2016b there's an even better way. Did you ever wonder why the MATLAB TAPPlugin is created using the a call like the following?

import matlab.unittest.plugins.TAPPlugin;
plugin = TAPPlugin.producingOriginalFormat

plugin = 

  TAPOriginalFormatPlugin with properties:

    IncludePassingDiagnostics: 0
                    Verbosity: Terse
     ExcludeLoggedDiagnostics: 0

What is the deal with the original format? What other formats are there? Well, quick history lession, the TAP format was developed over a number of years by the Perl community, but the formal specification really began at what could be called version 12 (and what we have termed the original format since this version was the first with a documented specification). Since then, there has also arisen a version 13 format. Actually there is a group of people involved in the discussions who are discussing a version 14 as well but there has been no formal specification of that format yet. If you are interested, and if you want to join me in ensuring that any such format takes into account the needs of the MATLAB community, certainly do get involved. There is a github project and a slack channel discussing the format that you can take part in.

Anyway, the news is that R2016b includes the option to produce the version 13 TAP format!

plugin = TAPPlugin.producingVersion13

plugin = 

  TAPVersion13Plugin with properties:

    IncludePassingDiagnostics: 0
                    Verbosity: Terse
     ExcludeLoggedDiagnostics: 0

The largest takeaway in this format is the fact that diagnostics can be included in YAML(ish) blocks in the tap stream. You may or may not be aware, but:

Y AML

A in't

M arkup

L anguage

We all know how programmers love recursive acronyms (e.g, GNU, XNA). What does this mean for us? A better diagnostics experience. Let's see this same test failure in our Jenkins job but instead using version 13 of the TAP protocol.

import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;

try
    suite = testsuite('unittest');
    runner = TestRunner.withTextOutput('Verbosity',3);
    % Add the TAP plugin
    tapFile = fullfile(getenv('WORKSPACE'), 'testResults.tap');
    
    runner.addPlugin(TAPPlugin.producingVersion13(ToFile(tapFile)));
    results = runner.run(suite)
catch e
    disp(getReport(e,'extended'));
    exit(1);
end
exit;

Using this produces the following TAP Stream

TAP version 13
1..51
not ok 1 - digraphAPITest/digraph
    ---
    Event:
        Event Name: 'VerificationFailed'
        Scope: 'digraphAPITest/digraph'
        Framework Diagnostic: |
            verifyError failed.
            --> The function threw the wrong exception.
                
                Actual Exception:
                    'MATLAB:TooManyInputs'
                Expected Exception:
                    'MATLAB:maxrhs'
            
            Actual Error Report:
                Error using digraph
                Too many input arguments.
                
                Error in digraphAPITest>@()digraph('foo') (line 14)
                            testCase.verifyError(@()digraph('foo'),'MATLAB:maxrhs');
            Evaluated Function:
                    @()digraph('foo')
        Stack: |
            In H:\Documents\MATLAB\unittest\digraphAPITest.m (digraphAPITest.digraph) at 14
    ...
ok 2 - digraphAPITest/isempty_
ok 3 - digraphAPITest/isEquivalent
ok 4 - digraphAPITest/addVertex
ok 5 - digraphAPITest/removeVertex
ok 6 - digraphAPITest/hasVertex
ok 7 - digraphAPITest/assertVertex
ok 8 - digraphAPITest/addEdge
ok 9 - digraphAPITest/removeEdge
ok 10 - digraphAPITest/hasEdge
ok 11 - digraphAPITest/findall
ok 12 - digraphAPITest/hasCycle
ok 13 - digraphAPITest/isComplete
ok 14 - digraphAPITest/shortestPath
ok 15 - digraphAPITest/subgraph
ok 16 - digraphAPITest/minimalEdges
ok 17 - digraphAPITest/transitiveClosure
ok 18 - digraphAPITest/union
ok 19 - digraphAPITest/intersect
ok 20 - digraphAPITest/sort
ok 21 - digraphAPITest/copy_
ok 22 - digraphAPITest/reset
ok 23 - digraphAPITest/spy
ok 24 - digraphFunctionalityTest/Vertex
ok 25 - digraphFunctionalityTest/Edge
ok 26 - digraphFunctionalityTest/unionBasic
ok 27 - digraphFunctionalityTest/unionAdvanced
ok 28 - digraphFunctionalityTest/intersectBasic
ok 29 - digraphFunctionalityTest/intersectAdvanced
ok 30 - digraphFunctionalityTest/hasPath
ok 31 - digraphFunctionalityTest/sort
ok 32 - digraphFunctionalityTest/shortestPathBasic
ok 33 - digraphFunctionalityTest/shortestPathAdvanced
ok 34 - digraphFunctionalityTest/hasCycle
ok 35 - digraphFunctionalityTest/CycleDoesNotTriggerInfiniteLoop
ok 36 - digraphFunctionalityTest/findall
ok 37 - digraphFunctionalityTest/subgraph
ok 38 - digraphFunctionalityTest/isEquivalent
ok 39 - digraphFunctionalityTest/copy_
ok 40 - digraphFunctionalityTest/isComplete
ok 41 - digraphFunctionalityTest/isempty_
ok 42 - digraphFunctionalityTest/transitiveClosureBasic
ok 43 - digraphFunctionalityTest/transitiveClosureAdvanced
ok 44 - digraphFunctionalityTest/transitiveClosureCycleAndComplete
ok 45 - digraphFunctionalityTest/minimalEdgesBasic
ok 46 - digraphFunctionalityTest/minimalEdgesAdvanced
ok 47 - digraphFunctionalityTest/minimalEdgesCycleAndComplete
ok 48 - digraphFunctionalityTest/trivialgraph
ok 49 - digraphFunctionalityTest/isEquivalentSubclass
ok 50 - digraphFunctionalityTest/subclassesArePreserved
ok 51 - digraphFunctionalityTest/spy

Very nice, we are in business with roughly the same failure diagnostics as the original format. However, the big win is that now that the diagnostics are in a structured format, other tools (like Jenkins) can represent them with much more flavor. This is now what the "TAP Extended Test Results" pages looks like in Jenkins.:

Now we can see all of the failure diagnostics on a single test results page with much more clarity and structure. Pretty sweet.

Do you see yourself benefitting from the more structured diagnostics that we get with version 13 of the TAP format? Do you have any stories to tell where good diagnostics saved you a bunch of debugging time? Do tell!

Published with MATLAB® R2016b