{"id":1747,"date":"2018-07-30T15:50:23","date_gmt":"2018-07-30T19:50:23","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=1747"},"modified":"2018-07-31T07:18:52","modified_gmt":"2018-07-31T11:18:52","slug":"semi-automated-testing","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2018\/07\/30\/semi-automated-testing\/","title":{"rendered":"Semi-Automated Testing"},"content":{"rendered":"

I've been doing a bit of spelunking around the File Exchange and GitHub lately, and I've seen a little pattern emerge in the tests of surprisingly many projects. It looks like this:<\/p>

\r\nclassdef<\/span> testDisk < matlab.unittest.TestCase\r\n    \r\n    properties<\/span>\r\n        map\r\n    end<\/span>\r\n    \r\n    methods<\/span> (TestClassSetup)\r\n        function<\/span> createMap(testCase)\r\n            opt = sctool.scmapopt('trace'<\/span>,0,'tol'<\/span>,1e-12);\r\n            p = polygon([4 2i -2+4i -3 -3-1i 2-2i]);\r\n            testCase.map = diskmap(p,opt);\r\n        end<\/span>\r\n    end<\/span>\r\n    \r\n    methods<\/span> (Test)\r\n        \r\n        function<\/span> testPlot(testCase)\r\n            fig = figure;\r\n            plot(testCase.map,4,3) % <======= RIGHT HERE!<\/span>\r\n            close(fig);\r\n        end<\/span>\r\n        \r\n    end<\/span>\r\n    \r\nend<\/span>\r\n\r\n<\/pre>
The plot command shows and exercises the graphical features of this toolbox. If we just run this outside of test form we can see it produces a cool result.<\/p>
opt = sctool.scmapopt('trace'<\/span>,0,'tol'<\/span>,1e-12);\r\np = polygon([4 2i -2+4i -3 -3-1i 2-2i]);\r\nmap = diskmap(p,opt);\r\nfig = figure;\r\nplot(map,4,3)\r\n<\/pre>\"\" 
By the way, in this case I am pulling from the Schwarz-Christoffel Toolbox<\/a>, which by my eye looks to be quite a nice package! Check out the User's Guide<\/a>.<\/p>
The idea here is great, right? The developer of the project is looking to get coverage on one of the key capabilities of the package, the visualization. At a minimum, the test is indeed confirming that the plot code executes without error, which is a great step. However, I feel like this might speak to a common pain point. How do I verify things that are very hard to verify, like graphics? Before we throw our hands into the air and flip over any tables<\/a> its worth noting that we may have a few options. We certainly can get access to the data in the plot and numerically confirm that it is plotted as expected. We can also check the properties of the graphics primitives and so on and so forth. This is all true, but I think it risks missing the point. Sometimes you just want to look at the dang plot!<\/p>
You might know exactly when the plot is right and when it is wrong. You might see subtle visual problems right away looking at it that would take forever to try to encode in a test covering every single property of every single graphics primitive you are working with.<\/p>
Just let me look at the plot.<\/p>
This test does just that, but it flashes the figure up on the screen and you have to look very closely (and quickly) or use a debugging workflow to get real insight and confirm the visualization is working correctly. A worse alternative is just to leave figures open and never close them. This litters your MATLAB environment every time you run the tests and is really hard to determine how each figure was produced and for what test. It doesn't work in a CI system workflow. In short, it makes it hard to verify the plots are correct, which means that we won't verify the plots are correct.<\/p>
Know what we can do though? We can log<\/b><\/tt>! We can testCase.log<\/b><\/tt>! We've already gone through the hard work of creating these figures and visualizations. Why don't we log them and see them later? We can do that pretty easily because we have a FigureDiagnostic<\/a> class that takes a figure handle and saves it away as both a .fig<\/b><\/tt> file and a .png<\/b><\/tt> file. That way we can log it away and open it up after the test run. If we were verifying anything (like the plot data or graphics attributes) we could also just use these diagnostics as the diagnostics input on the verification or assertion methods we are using. For the test above, let's log it:<\/p>
\r\nclassdef<\/span> testDisk < matlab.unittest.TestCase\r\n    \r\n    properties<\/span>\r\n        map\r\n    end<\/span>\r\n    \r\n    \r\n    methods<\/span> (TestClassSetup)\r\n        function<\/span> createMap(testCase)\r\n            opt = sctool.scmapopt('trace'<\/span>,0,'tol'<\/span>,1e-12);\r\n            p = polygon([4 2i -2+4i -3 -3-1i 2-2i]);\r\n            testCase.map = diskmap(p,opt);\r\n        end<\/span>\r\n    end<\/span>\r\n    \r\n    \r\n    methods<\/span> (Test)\r\n        \r\n        function<\/span> testPlot(testCase)\r\n            import matlab.unittest.diagnostics.Diagnostic<\/span>;\r\n            import matlab.unittest.diagnostics.FigureDiagnostic<\/span>;\r\n            \r\n            fig = figure;\r\n            % Let's use addTeardown instead because https:\/\/blogs.mathworks.com\/developer\/2015\/07\/27\/addteardown\/<\/span>\r\n            testCase.addTeardown(@close, fig);\r\n            plot(testCase.map,4,3);\r\n            \r\n            % Now we log it for fun and for profit.<\/span>\r\n            testCase.log(3, ...<\/span>\r\n                Diagnostic.join('Please confirm there are concentric convex sets in the lower left.'<\/span>, ...<\/span>\r\n                FigureDiagnostic(fig)));\r\n            \r\n        end<\/span>\r\n        \r\n    end<\/span>\r\n    \r\nend<\/span>\r\n\r\n<\/pre>
I've put a nice description on there so we know what we are looking for in the figure. I did this by joining a string description with our FigureDiagnostic<\/b><\/tt> using Diagnostic.join<\/b><\/tt>. Also, I've logged it at level 3, which corresponds to the Detailed<\/b><\/tt> level of the Verbosity<\/b><\/tt><\/a> enumeration.   This means it won't show up if I just run the standard runtests<\/b><\/tt> call:<\/p>
runtests('tests\/testDisk.m'<\/span>)\r\n<\/pre>
Running testDisk\r\n....\r\nDone testDisk\r\n__________\r\n\r\n\r\nans = \r\n\r\n  1×4 TestResult array with properties:\r\n\r\n    Name\r\n    Passed\r\n    Failed\r\n    Incomplete\r\n    Duration\r\n    Details\r\n\r\nTotals:\r\n   4 Passed, 0 Failed, 0 Incomplete.\r\n   0.80408 seconds testing time.\r\n\r\n<\/pre>
...but it will if I run at a higher level of logging:<\/p>
runtests('tests\/testDisk.m'<\/span>,'Verbosity'<\/span>,'Detailed'<\/span>)\r\n<\/pre>
 Running testDisk\r\n  Setting up testDisk\r\n  Done setting up testDisk in 0.01131 seconds\r\n   Running testDisk\/testForwardMap\r\n   Done testDisk\/testForwardMap in 0.0076177 seconds\r\n   Running testDisk\/testInverseMap\r\n   Done testDisk\/testInverseMap in 0.0071096 seconds\r\n   Running testDisk\/testCenter\r\n   Done testDisk\/testCenter in 0.0082754 seconds\r\n   Running testDisk\/testPlot\r\n[Detailed] Diagnostic logged (2018-07-30T15:42:18): \r\nPlease confirm there are concentric convex sets in the lower left.\r\nFigure saved to:\r\n--> \/private\/var\/folders\/bm\/6qgg87js1bb7fpr2p475bcwh0002wp\/T\/094eb448-615a-4667-95e2-0a6b62b81eae\/Figure_2d16d47d-a44a-4425-9507-84bb27afcf26.fig<\/a>\r\n--> \/private\/var\/folders\/bm\/6qgg87js1bb7fpr2p475bcwh0002wp\/T\/094eb448-615a-4667-95e2-0a6b62b81eae\/Figure_2d16d47d-a44a-4425-9507-84bb27afcf26.png<\/a>\r\n\r\n   Done testDisk\/testPlot in 1.3447 seconds\r\n  Tearing down testDisk\r\n  Done tearing down testDisk in 0 seconds\r\n Done testDisk in 1.379 seconds\r\n__________\r\n\r\n\r\nans = \r\n\r\n  1×4 TestResult array with properties:\r\n\r\n    Name\r\n    Passed\r\n    Failed\r\n    Incomplete\r\n    Duration\r\n    Details\r\n\r\nTotals:\r\n   4 Passed, 0 Failed, 0 Incomplete.\r\n   1.379 seconds testing time.\r\n\r\n<\/pre>
Great! Now we can see links in the test log pointing to images of the plot as well as a figure file. This is nice, but I am just getting started. Let's see this workflow when we generate a test report:<\/p>
import matlab.unittest.plugins.TestReportPlugin<\/span>;\r\n\r\nrunner = matlab.unittest.TestRunner.withTextOutput;\r\nrunner.addPlugin(TestReportPlugin.producingHTML('Verbosity'<\/span>,3))\r\nrunner.run(testsuite('tests'<\/span>))\r\n<\/pre>
Running testAnnulus\r\n \r\nNumber of iterations: 32\r\nNumber of function evaluations: 91\r\nFinal norm(F(x)): 1.27486e-09\r\nNumber of restarts for secant methods: 1\r\n...\r\nDone testAnnulus\r\n__________\r\n\r\nRunning testDisk\r\n....\r\nDone testDisk\r\n__________\r\n\r\nRunning testExterior\r\n...\r\nDone testExterior\r\n__________\r\n\r\nRunning testHalfplane\r\n...\r\nDone testHalfplane\r\n__________\r\n\r\nRunning testRectangle\r\n...\r\nDone testRectangle\r\n__________\r\n\r\nRunning testStrip\r\n...\r\nDone testStrip\r\n__________\r\n\r\nGenerating report. Please wait.\r\n    Preparing content for the report.\r\n    Adding content to the report.\r\n    Writing report to file.\r\nReport has been saved to: \/private\/var\/folders\/bm\/6qgg87js1bb7fpr2p475bcwh0002wp\/T\/tp86d8e3a7_aedb_45fa_a82e_0ceb6430ee87\/index.html<\/a>\r\n\r\nans = \r\n\r\n  1×19 TestResult array with properties:\r\n\r\n    Name\r\n    Passed\r\n    Failed\r\n    Incomplete\r\n    Duration\r\n    Details\r\n\r\nTotals:\r\n   19 Passed, 0 Failed, 0 Incomplete.\r\n   8.7504 seconds testing time.\r\n\r\n<\/pre>
This is where it really starts to get beautiful. Now we have a full report that we can view at our leisure and confirm that all the visualizations are correct<\/p>
\"\" <\/p>
We've run the whole test suite and have captured the figures for all the tests<\/b><\/i> not just this one. We are now in the realm of semi-automated<\/i> testing. There are some things that really need a human to take a look at to confirm correctness. However, the entirety of the test run and test setup can still be automated! This can still be done via a CI system so you don't have to remember to run the tests and look over the plots every time you change the code. You simply let the automation do it. For things that need manual verification you can always log away the artifacts in a pdf or html report and confirm periodically, or prior to release. If there is a bug, you can mine the artifacts from all your CI builds to see where and when it was introduced.<\/p>
You can even extend this approach to add an expected image to the report. So if you log a known good expected image and then use the test code to generate the image for each software change you can look at the actual image and the expected image right next to each other and confirm that they match. Beautiful. Full test automation is clearly the ideal to strive for, but in those cases that you really need to look at a picture, let the framework and your CI system do all the work for you in setting it up and you can just quickly and efficiently verify that it is correct.<\/p>
Happy semi-automated testing!<\/p>
P.S. Take a look at the full report generated in PDF form here<\/a><\/p>