{"id":739,"date":"2016-09-29T16:42:35","date_gmt":"2016-09-29T16:42:35","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=739"},"modified":"2016-09-29T16:42:35","modified_gmt":"2016-09-29T16:42:35","slug":"tap-with-yaml","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2016\/09\/29\/tap-with-yaml\/","title":{"rendered":"Turn on the YAML Channel!"},"content":{"rendered":"\r\n

Diagnostics. They are the secret sauce of an effective test framework and we certainly take our job seriously at providing diagnostics that are not a little too hot or a little too cold, but juuuuusssst the right information to present, especially when encountering a failure. The reason this is so important is because good diagnostics can save human time, which is typically far more precious than computer time.<\/p>

Indeed, this is one of the reasons why CI systems are so valuable. They take away the human time required to build, qualify, and deploy software and delegate that job to a machine rather than compromising your productivity. However, when a failure occurs you then rely on diagnostics from the failure to give you some hint as to what is going on.<\/p>

Let me show you an example of this. I went and downloaded the digraph<\/a> file exchange submission to highlight this. Note, this submission has now been superseded by the fantastic graph functionality<\/a> included in R2015b and later, but if you are not yet on that release you can certainly check it out. However, one reason I am including that is because it has a nice test suite included with it that we can run from your CI system.<\/p>

Not yet set up with a CI system? Take a look at some of these other posts<\/a> to get you going. If you haven't read them yet go from the bottom up on that category page.<\/p>

Anyway, let's continue with Jenkins and show how the diagnostics currently show up with these diagraph tests. Here is the script I am asking MATLAB to run in the Jenkins project:<\/p>

\r\nimport matlab.unittest.TestRunner<\/span>;\r\nimport matlab.unittest.plugins.TAPPlugin<\/span>;\r\nimport matlab.unittest.plugins.ToFile<\/span>;\r\n\r\ntry<\/span>\r\n    suite = testsuite('unittest'<\/span>);\r\n    runner = TestRunner.withTextOutput('Verbosity'<\/span>,3);\r\n    % Add the TAP plugin<\/span>\r\n    tapFile = fullfile(getenv('WORKSPACE'<\/span>), 'testResults.tap'<\/span>);\r\n    \r\n    runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile)));\r\n    results = runner.run(suite)\r\ncatch<\/span> e\r\n    disp(getReport(e,'extended'<\/span>));\r\n    exit(1);\r\nend<\/span>\r\nexit;\r\n\r\n<\/pre>
Does it pass? Well when we run this job on the digraph submission in Jenkins here is what we get:<\/p>
\"\" <\/p>
Alright. Not bad but it looks like we have a failure. Do we know why it failed? Not yet. For that, we need to drill into the \"TAP Test Results\"<\/b> instead of the \"Extended TAP Test Results\"<\/b> of the Jenkins TAP Plugin.<\/p>
\"\" <\/p>
Here, we can finally dig into the failing test by clicking on it and getting to this page, which shows the diagnostics:<\/p>
\"\" <\/p>
Whew. Alright it took us a couple clicks, but we made it. We can see pretty quickly that that test is validating against an incorrect error message id. Looking at the two ids, you can take a pretty good guess that this is probably not actually a real problem in the software, but may simply require a trivial test update since perhaps the id has changed across versions of MATLAB. No problem. Quick to find, quick to fix.<\/p>
However, I think we can improve on how we got here. For one, there were just too many clicks needed. Also, there seems to be some whitespace formatting lost when the Jenkins plugin displayed the diagnostic values. In this case it may have just been easier to look at the tap file output directly:<\/p>
\r\n1..51\r\nnot ok 1 - digraphAPITest\/digraph\r\n# ================================================================================\r\n# Verification failed in digraphAPITest\/digraph.\r\n# \r\n#     ---------------------\r\n#     Framework Diagnostic:\r\n#     ---------------------\r\n#     verifyError failed.\r\n#     --> The function threw the wrong exception.\r\n#         \r\n#         Actual Exception:\r\n#             'MATLAB:TooManyInputs'\r\n#         Expected Exception:\r\n#             'MATLAB:maxrhs'\r\n#     \r\n#     Actual Error Report:\r\n#         Error using digraph\r\n#         Too many input arguments.\r\n#         \r\n#         Error in digraphAPITest>@()digraph('foo') (line 14)\r\n#                     testCase.verifyError(@()digraph('foo'),'MATLAB:maxrhs');\r\n#     Evaluated Function:\r\n#             @()digraph('foo')\r\n# \r\n#     ------------------\r\n#     Stack Information:\r\n#     ------------------\r\n#     In H:\\Documents\\MATLAB\\unittest\\digraphAPITest.m (digraphAPITest.digraph) at 14\r\n# ================================================================================\r\n# \r\nok 2 - digraphAPITest\/isempty_\r\nok 3 - digraphAPITest\/isEquivalent\r\nok 4 - digraphAPITest\/addVertex\r\nok 5 - digraphAPITest\/removeVertex\r\nok 6 - digraphAPITest\/hasVertex\r\nok 7 - digraphAPITest\/assertVertex\r\nok 8 - digraphAPITest\/addEdge\r\nok 9 - digraphAPITest\/removeEdge\r\nok 10 - digraphAPITest\/hasEdge\r\nok 11 - digraphAPITest\/findall\r\nok 12 - digraphAPITest\/hasCycle\r\nok 13 - digraphAPITest\/isComplete\r\nok 14 - digraphAPITest\/shortestPath\r\nok 15 - digraphAPITest\/subgraph\r\nok 16 - digraphAPITest\/minimalEdges\r\nok 17 - digraphAPITest\/transitiveClosure\r\nok 18 - digraphAPITest\/union\r\nok 19 - digraphAPITest\/intersect\r\nok 20 - digraphAPITest\/sort\r\nok 21 - digraphAPITest\/copy_\r\nok 22 - digraphAPITest\/reset\r\nok 23 - digraphAPITest\/spy\r\nok 24 - digraphFunctionalityTest\/Vertex\r\nok 25 - digraphFunctionalityTest\/Edge\r\nok 26 - digraphFunctionalityTest\/unionBasic\r\nok 27 - digraphFunctionalityTest\/unionAdvanced\r\nok 28 - digraphFunctionalityTest\/intersectBasic\r\nok 29 - digraphFunctionalityTest\/intersectAdvanced\r\nok 30 - digraphFunctionalityTest\/hasPath\r\nok 31 - digraphFunctionalityTest\/sort\r\nok 32 - digraphFunctionalityTest\/shortestPathBasic\r\nok 33 - digraphFunctionalityTest\/shortestPathAdvanced\r\nok 34 - digraphFunctionalityTest\/hasCycle\r\nok 35 - digraphFunctionalityTest\/CycleDoesNotTriggerInfiniteLoop\r\nok 36 - digraphFunctionalityTest\/findall\r\nok 37 - digraphFunctionalityTest\/subgraph\r\nok 38 - digraphFunctionalityTest\/isEquivalent\r\nok 39 - digraphFunctionalityTest\/copy_\r\nok 40 - digraphFunctionalityTest\/isComplete\r\nok 41 - digraphFunctionalityTest\/isempty_\r\nok 42 - digraphFunctionalityTest\/transitiveClosureBasic\r\nok 43 - digraphFunctionalityTest\/transitiveClosureAdvanced\r\nok 44 - digraphFunctionalityTest\/transitiveClosureCycleAndComplete\r\nok 45 - digraphFunctionalityTest\/minimalEdgesBasic\r\nok 46 - digraphFunctionalityTest\/minimalEdgesAdvanced\r\nok 47 - digraphFunctionalityTest\/minimalEdgesCycleAndComplete\r\nok 48 - digraphFunctionalityTest\/trivialgraph\r\nok 49 - digraphFunctionalityTest\/isEquivalentSubclass\r\nok 50 - digraphFunctionalityTest\/subclassesArePreserved\r\nok 51 - digraphFunctionalityTest\/spy\r\n<\/pre>
Well with R2016b there's an even better way. Did you ever wonder why the MATLAB TAPPlugin is created using the a call like the following?<\/p>
import matlab.unittest.plugins.TAPPlugin<\/span>;\r\nplugin = TAPPlugin.producingOriginalFormat\r\n<\/pre>
\r\nplugin = \r\n\r\n  TAPOriginalFormatPlugin with properties:\r\n\r\n    IncludePassingDiagnostics: 0\r\n                    Verbosity: Terse\r\n     ExcludeLoggedDiagnostics: 0\r\n\r\n<\/pre>
What is the deal with the original format? What other formats are there? Well, quick history lession, the TAP format<\/a> was developed over a number of years by the Perl community, but the formal specification really began at what could be called version 12 (and what we have termed the original format since this version was the first with a documented specification<\/a>). Since then, there has also arisen a version 13 format<\/a>. Actually there is a group of people involved in the discussions who are discussing a version 14 as well but there has been no formal specification of that format yet. If you are interested, and if you want to join me in ensuring that any such format takes into account the needs of the MATLAB community, certainly do get involved. There is a github project<\/a> and a slack channel<\/a> discussing the format that you can take part in.<\/p>
Anyway, the news is that R2016b includes the option to produce the version 13 TAP format!<\/p>
plugin = TAPPlugin.producingVersion13\r\n<\/pre>
\r\nplugin = \r\n\r\n  TAPVersion13Plugin with properties:\r\n\r\n    IncludePassingDiagnostics: 0\r\n                    Verbosity: Terse\r\n     ExcludeLoggedDiagnostics: 0\r\n\r\n<\/pre>
The largest takeaway in this format is the fact that diagnostics can be included in YAML(ish) blocks in the tap stream. You may or may not be aware, but:<\/p>
Y<\/b> AML<\/p>
A<\/b> in't<\/p>
M<\/b> arkup<\/p>
L<\/b> anguage<\/p>
We all know how programmers love recursive acronyms (e.g, GNU<\/a>, XNA<\/a>). What does this mean for us? A better diagnostics experience. Let's see this same test failure in our Jenkins job but instead using version 13 of the TAP protocol.<\/p>
\r\nimport matlab.unittest.TestRunner<\/span>;\r\nimport matlab.unittest.plugins.TAPPlugin<\/span>;\r\nimport matlab.unittest.plugins.ToFile<\/span>;\r\n\r\ntry<\/span>\r\n    suite = testsuite('unittest'<\/span>);\r\n    runner = TestRunner.withTextOutput('Verbosity'<\/span>,3);\r\n    % Add the TAP plugin<\/span>\r\n    tapFile = fullfile(getenv('WORKSPACE'<\/span>), 'testResults.tap'<\/span>);\r\n    \r\n    runner.addPlugin(TAPPlugin.producingVersion13(ToFile(tapFile)));\r\n    results = runner.run(suite)\r\ncatch<\/span> e\r\n    disp(getReport(e,'extended'<\/span>));\r\n    exit(1);\r\nend<\/span>\r\nexit;\r\n\r\n<\/pre>
Using this produces the following TAP Stream<\/p>
\r\nTAP version 13\r\n1..51\r\nnot ok 1 - digraphAPITest\/digraph\r\n    ---\r\n    Event:\r\n        Event Name: 'VerificationFailed'\r\n        Scope: 'digraphAPITest\/digraph'\r\n        Framework Diagnostic: |\r\n            verifyError failed.\r\n            --> The function threw the wrong exception.\r\n                \r\n                Actual Exception:\r\n                    'MATLAB:TooManyInputs'\r\n                Expected Exception:\r\n                    'MATLAB:maxrhs'\r\n            \r\n            Actual Error Report:\r\n                Error using digraph\r\n                Too many input arguments.\r\n                \r\n                Error in digraphAPITest>@()digraph('foo') (line 14)\r\n                            testCase.verifyError(@()digraph('foo'),'MATLAB:maxrhs');\r\n            Evaluated Function:\r\n                    @()digraph('foo')\r\n        Stack: |\r\n            In H:\\Documents\\MATLAB\\unittest\\digraphAPITest.m (digraphAPITest.digraph) at 14\r\n    ...\r\nok 2 - digraphAPITest\/isempty_\r\nok 3 - digraphAPITest\/isEquivalent\r\nok 4 - digraphAPITest\/addVertex\r\nok 5 - digraphAPITest\/removeVertex\r\nok 6 - digraphAPITest\/hasVertex\r\nok 7 - digraphAPITest\/assertVertex\r\nok 8 - digraphAPITest\/addEdge\r\nok 9 - digraphAPITest\/removeEdge\r\nok 10 - digraphAPITest\/hasEdge\r\nok 11 - digraphAPITest\/findall\r\nok 12 - digraphAPITest\/hasCycle\r\nok 13 - digraphAPITest\/isComplete\r\nok 14 - digraphAPITest\/shortestPath\r\nok 15 - digraphAPITest\/subgraph\r\nok 16 - digraphAPITest\/minimalEdges\r\nok 17 - digraphAPITest\/transitiveClosure\r\nok 18 - digraphAPITest\/union\r\nok 19 - digraphAPITest\/intersect\r\nok 20 - digraphAPITest\/sort\r\nok 21 - digraphAPITest\/copy_\r\nok 22 - digraphAPITest\/reset\r\nok 23 - digraphAPITest\/spy\r\nok 24 - digraphFunctionalityTest\/Vertex\r\nok 25 - digraphFunctionalityTest\/Edge\r\nok 26 - digraphFunctionalityTest\/unionBasic\r\nok 27 - digraphFunctionalityTest\/unionAdvanced\r\nok 28 - digraphFunctionalityTest\/intersectBasic\r\nok 29 - digraphFunctionalityTest\/intersectAdvanced\r\nok 30 - digraphFunctionalityTest\/hasPath\r\nok 31 - digraphFunctionalityTest\/sort\r\nok 32 - digraphFunctionalityTest\/shortestPathBasic\r\nok 33 - digraphFunctionalityTest\/shortestPathAdvanced\r\nok 34 - digraphFunctionalityTest\/hasCycle\r\nok 35 - digraphFunctionalityTest\/CycleDoesNotTriggerInfiniteLoop\r\nok 36 - digraphFunctionalityTest\/findall\r\nok 37 - digraphFunctionalityTest\/subgraph\r\nok 38 - digraphFunctionalityTest\/isEquivalent\r\nok 39 - digraphFunctionalityTest\/copy_\r\nok 40 - digraphFunctionalityTest\/isComplete\r\nok 41 - digraphFunctionalityTest\/isempty_\r\nok 42 - digraphFunctionalityTest\/transitiveClosureBasic\r\nok 43 - digraphFunctionalityTest\/transitiveClosureAdvanced\r\nok 44 - digraphFunctionalityTest\/transitiveClosureCycleAndComplete\r\nok 45 - digraphFunctionalityTest\/minimalEdgesBasic\r\nok 46 - digraphFunctionalityTest\/minimalEdgesAdvanced\r\nok 47 - digraphFunctionalityTest\/minimalEdgesCycleAndComplete\r\nok 48 - digraphFunctionalityTest\/trivialgraph\r\nok 49 - digraphFunctionalityTest\/isEquivalentSubclass\r\nok 50 - digraphFunctionalityTest\/subclassesArePreserved\r\nok 51 - digraphFunctionalityTest\/spy\r\n<\/pre>
Very nice, we are in business with roughly the same failure diagnostics as the original format. However, the big win is that now that the diagnostics are in a structured format, other tools (like Jenkins) can represent them with much more flavor. This is now what the \"TAP Extended Test Results\"<\/b> pages looks like in Jenkins.:<\/p>
\"\" <\/p>
Now we can see all of the failure diagnostics on a single test results page with much more clarity and structure. Pretty sweet.<\/p>
Do you see yourself benefitting from the more structured diagnostics that we get with version 13 of the TAP format? Do you have any stories to tell where good diagnostics saved you a bunch of debugging time? Do tell!<\/p>