{"id":78,"date":"2015-02-20T21:09:26","date_gmt":"2015-02-20T21:09:26","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=78"},"modified":"2015-02-20T21:46:35","modified_gmt":"2015-02-20T21:46:35","slug":"encouragingly-parallel-part-1","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2015\/02\/20\/encouragingly-parallel-part-1\/","title":{"rendered":"Encouragingly Parallel"},"content":{"rendered":"\r\n\r\n<div class=\"content\"><!--introduction--><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#05de8e27-8e92-4a4c-a34b-674bb46d8ba2\">The Test Suite<\/a><\/li><li><a href=\"#6a11abe3-9caa-4f4d-b247-3eb08341d59d\">Into the Pool<\/a><\/li><\/ul><\/div><p>When I am neck deep in hardcore <a href=\"http:\/\/en.wikipedia.org\/wiki\/Test-driven_development\">TDD Red-Green-Refactor<\/a> cycles I am constantly looking for ways to ensure that my feedback loop is as quick as possible. If the testing feedback takes too long I am liable to start daydreaming and I lose my context in the design process (because of course <a href=\"https:\/\/twitter.com\/KentBeck\/status\/323865584695640064\">TDD is about design not testing<\/a>). Often this means that I run a fast, small test suite which is focused on just the change at hand. However, sometimes the refactor step touches a few different areas, and it requires running a more substantial set of tests. How do we minimize the \"distractable\" time and maximize the design time in these cycles?<\/p><p>Encouragingly, well written test suites are (almost) <a href=\"https:\/\/en.wikipedia.org\/wiki\/Embarrassingly_parallel\">embarrassingly parallel<\/a>. As a principle each test should be completely independent of the next and thus can be run in any order, on any machine, at any time. Furthermore test suites in the MATLAB&reg; test framework are just arrays of <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.unittest.test-class.html\"><b><tt>Test<\/tt><\/b><\/a> objects where each element can be independently run. If you have the <a href=\"https:\/\/www.mathworks.com\/products\/parallel-computing\/\">Parallel Computing Toolbox&#8482;<\/a> there exists a variety of ways these tests can be run in parallel. Awesome, let's dig into the details of how this is done.<\/p><h4>The Test Suite<a name=\"05de8e27-8e92-4a4c-a34b-674bb46d8ba2\"><\/a><\/h4><p>We need to establish what the test suite is as we explore this topic. Of course the test suite can be anything written using the unit test framework. Typically the time taken to execute the tests corresponds to the time taken actually setting up, exercising, verifying, and tearing down the software under test. However, for this example why don't we just add some calls to the <b><tt>pause<\/tt><\/b> function in order to mimic a real test? We can create 3 simple tests that we can use to build a demonstrative test suite.<\/p><p>Let's use one script-based test with just a couple simple tests:<\/p><pre class=\"language-matlab\"><span class=\"comment\">%% The feature should do something<\/span>\r\npause(rand) <span class=\"comment\">% 0-1 seconds<\/span>\r\n\r\n<span class=\"comment\">%% The feature should do another thing<\/span>\r\npause(rand); <span class=\"comment\">% 0-1 seconds<\/span>\r\n<\/pre><p>...a function-based test with a couple tests and a relatively long <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/matlab_prog\/write-test-using-setup-and-teardown-functions.html#zmw57dd0e57154\">file fixture function<\/a>:<\/p><pre class=\"language-matlab\"><span class=\"keyword\">function<\/span> tests = aFunctionBasedTest\r\ntests = functiontests(localfunctions);\r\n\r\n<span class=\"keyword\">function<\/span> setupOnce(~)\r\n<span class=\"comment\">% Create a fixture that takes a while to build<\/span>\r\npause(rand*10); <span class=\"comment\">% 0-10 seconds<\/span>\r\n\r\n<span class=\"keyword\">function<\/span> testSomeFeature(~)\r\npause(rand); <span class=\"comment\">% 0-1 seconds<\/span>\r\n\r\n<span class=\"keyword\">function<\/span> testAnotherFeature(~)\r\npause(rand); <span class=\"comment\">% 0-1 seconds<\/span>\r\n<\/pre><p>...and finally a class-based test with one simple test and one relatively long system test:<\/p><pre class=\"language-matlab\"><span class=\"keyword\">classdef<\/span> aClassBasedTest &lt; matlab.unittest.TestCase\r\n    <span class=\"keyword\">methods<\/span>(Test)\r\n        <span class=\"keyword\">function<\/span> testLongRunningEndToEndWorkflow(~)\r\n            pause(rand*10); <span class=\"comment\">% 0-10 seconds<\/span>\r\n        <span class=\"keyword\">end<\/span>\r\n        <span class=\"keyword\">function<\/span> testANormalFeature(~)\r\n            pause(rand); <span class=\"comment\">% 0-1 seconds<\/span>\r\n        <span class=\"keyword\">end<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\n<\/pre><p>Using these simple dummy tests we can create a large representative suite by just using <b><tt>repmat<\/tt><\/b> and concatenation:<\/p><pre class=\"codeinput\">import <span class=\"string\">matlab.unittest.TestSuite<\/span>;\r\nclassSuite = TestSuite.fromFile(<span class=\"string\">'aClassBasedTest.m'<\/span>);\r\nfcnSuite = TestSuite.fromFile(<span class=\"string\">'aFunctionBasedTest.m'<\/span>);\r\nscriptSuite = TestSuite.fromFile(<span class=\"string\">'aScriptBasedTest.m'<\/span>);\r\n\r\nsuite = [repmat(classSuite, 1, 50), repmat(fcnSuite, 1, 50), repmat(scriptSuite, 1, 50)];\r\n\r\n<span class=\"comment\">% Let's run this suite serially to see how long it takes:<\/span>\r\ntic;\r\nresult = run(suite)\r\ntoc;\r\n<\/pre><pre class=\"codeoutput\">Running aClassBasedTest\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\nDone aClassBasedTest\r\n__________\r\n\r\nRunning aFunctionBasedTest\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\nDone aFunctionBasedTest\r\n__________\r\n\r\nRunning aScriptBasedTest\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\n..........\r\nDone aScriptBasedTest\r\n__________\r\n\r\n\r\nresult = \r\n\r\n  1x300 TestResult array with properties:\r\n\r\n    Name\r\n    Passed\r\n    Failed\r\n    Incomplete\r\n    Duration\r\n\r\nTotals:\r\n   300 Passed, 0 Failed, 0 Incomplete.\r\n   380.6043 seconds testing time.\r\n\r\nElapsed time is 385.422383 seconds.\r\n<\/pre><h4>Into the Pool<a name=\"6a11abe3-9caa-4f4d-b247-3eb08341d59d\"><\/a><\/h4><p>Somewhere in the neighborhood of 6 and a half minutes delay time will definitely lose my attention, hopefully to something more productive than cat videos (but no guarantee!). What is the simplest way I can run these in parallel? Let's try a simple <a href=\"https:\/\/www.mathworks.com\/help\/distcomp\/parfor.html\"><b><tt>parfor<\/tt><\/b><\/a> loop using a parallel pool containing 16 workers:<\/p><pre class=\"codeinput\">tic;\r\n<span class=\"keyword\">parfor<\/span> idx = 1:numel(suite)\r\n    results(idx) = run(suite(idx));\r\n<span class=\"keyword\">end<\/span>\r\nresults\r\ntoc;\r\n<\/pre><pre class=\"codeoutput\">Running aClassBasedTest\r\nRunning aClassBasedTest\r\nRunning aFunctionBasedTest\r\nRunning aClassBasedTest\r\n.\r\nDone aFunctionBasedTest\r\n__________\r\n\r\nRunning aFunctionBasedTest\r\n.\r\nDone aClassBasedTest\r\n__________\r\n\r\nRunning aClassBasedTest\r\n.\r\nDone aClassBasedTest\r\n__________\r\n\r\nRunning aClassBasedTest\r\nRunning aFunctionBasedTest\r\nRunning aFunctionBasedTest\r\n.\r\nDone aFunctionBasedTest\r\n__________\r\n\r\nRunning aFunctionBasedTest\r\nRunning aClassBasedTest\r\n.\r\nDone aClassBasedTest\r\n__________\r\n\r\nRunning aClassBasedTest\r\nRunning aFunctionBasedTest\r\nRunning aClassBasedTest\r\nRunning aClassBasedTest\r\n.\r\nDone aClassBasedTest\r\n__________\r\n\r\nRunning aClassBasedTest\r\n.\r\nDone aFunctionBasedTest\r\n__________\r\n\r\nRunning aFunctionBasedTest\r\nRunning aClassBasedTest\r\n.\r\nDone aClassBasedTest\r\n__________\r\n\r\n\r\n\r\n\r\n&lt;SNIP: Lengthy output removed to save your scrollwheel finger.&gt;\r\n\r\n\r\n\r\n\r\nRunning aFunctionBasedTest\r\n.\r\nDone aFunctionBasedTest\r\n__________\r\n\r\n\r\nresults = \r\n\r\n  1x300 TestResult array with properties:\r\n\r\n    Name\r\n    Passed\r\n    Failed\r\n    Incomplete\r\n    Duration\r\n\r\nTotals:\r\n   300 Passed, 0 Failed, 0 Incomplete.\r\n   838.4606 seconds testing time.\r\n\r\nElapsed time is 81.866555 seconds.\r\n<\/pre><p><a href=\"https:\/\/blogs.mathworks.com\/developer\/files\/2015ParallelismFTW.jpg\">Parallelism FTW!<\/a> Now we have the suite running on the order of a minute and a half. That is much better time, but it's still not good enough for me. Also, what is the deal with the humongous (and unparsable) output? Note, I spared you from excessive browser scrolling by actually removing (SNIP!) most of the produced output. You can see, however, that each test element got its own start\/end lines and different workers all printed their output to the command window without any grouping or understanding of what ran where. Do you see the lines that look like we start aClassBasedTest and finish aFunctionBasedTest? Theres no magic test conversion going on here, were are just getting garbled output from the workers.<\/p><p>Another interesting tidbit you can see is that the overall testing time actually increased significantly. This is not explained by the test framework time or the client\/worker communication overhead, because the <b><tt>Duration<\/tt><\/b> property of <b><tt>TestResult<\/tt><\/b> only includes the time taken by the actual test content. What is actually happening here is that the function-based test, which has an expensive <b><tt>setupOnce<\/tt><\/b> function, is not enjoying the efficiency benefits of only setting up that fixture once and sharing it across multiple tests. Instead this <b><tt>setupOnce<\/tt><\/b> function is executed on every element of the function-based test on every worker. The benefits of sharing the fixture only apply when a MATLAB runs more than one test using that fixture. In this case, we are setting it up for every new <b><tt>Test<\/tt><\/b> element that we send to each parallel worker because we are sending each suite element one at a time to the pool.<\/p><p>Let's talk next time about how we can improve on this further and tackle these problems. In the meantime, have you used parallelism in your testing workflow? What works for you?<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_973bc21b31e34f5e9f751162e24b05a3() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='973bc21b31e34f5e9f751162e24b05a3 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 973bc21b31e34f5e9f751162e24b05a3';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2015 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_973bc21b31e34f5e9f751162e24b05a3()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2014b<br><\/p><\/div><!--\r\n973bc21b31e34f5e9f751162e24b05a3 ##### SOURCE BEGIN #####\r\n%% Encouragingly Parallel \r\n\r\n%%\r\n% When I am neck deep in hardcore\r\n% <http:\/\/en.wikipedia.org\/wiki\/Test-driven_development TDD\r\n% Red-Green-Refactor> cycles I am constantly looking for ways to ensure\r\n% that my feedback loop is as quick as possible. If the testing feedback\r\n% takes too long I am liable to start daydreaming and I lose my context in\r\n% the design process (because of course\r\n% <https:\/\/twitter.com\/KentBeck\/status\/323865584695640064 TDD is about\r\n% design not testing>). Often this means that I run a fast, small test\r\n% suite which is focused on just the change at hand. However, sometimes the\r\n% refactor step touches a few different areas, and it requires running a\r\n% more substantial set of tests. How do we minimize the \"distractable\" time\r\n% and maximize the design time in these cycles?\r\n%\r\n%%\r\n% Encouragingly, well written test suites are (almost)\r\n% <https:\/\/en.wikipedia.org\/wiki\/Embarrassingly_parallel embarrassingly\r\n% parallel>. As a principle each test should be completely independent of\r\n% the next and thus can be run in any order, on any machine, at any time.\r\n% Furthermore test suites in the MATLAB(R) test framework are just arrays\r\n% of\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.unittest.test-class.html\r\n% *|Test|*> objects where each element can be independently run. If you have\r\n% the <https:\/\/www.mathworks.com\/products\/parallel-computing\/ Parallel\r\n% Computing Toolbox(TM)> there exists a variety of ways these tests can be\r\n% run in parallel. Awesome, let's dig into the details of how this is done.\r\n%\r\n%% The Test Suite\r\n%\r\n% We need to establish what the test suite is as we explore this topic. Of\r\n% course the test suite can be anything written using the unit test\r\n% framework. Typically the time taken to execute the tests corresponds to\r\n% the time taken actually setting up, exercising, verifying, and tearing\r\n% down the software under test. However, for this example why don't we just\r\n% add some calls to the *|pause|* function in order to mimic a real test? We\r\n% can create 3 simple tests that we can use to build a demonstrative test\r\n% suite.\r\n%\r\n% Let's use one script-based test with just a couple simple tests:\r\n% \r\n%   %% The feature should do something\r\n%   pause(rand) % 0-1 seconds\r\n%   \r\n%   %% The feature should do another thing\r\n%   pause(rand); % 0-1 seconds\r\n%\r\n% ...a function-based test with a couple tests and a relatively long\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/matlab_prog\/write-test-using-setup-and-teardown-functions.html#zmw57dd0e57154\r\n% file fixture function>:\r\n% \r\n%   function tests = aFunctionBasedTest\r\n%   tests = functiontests(localfunctions);\r\n%   \r\n%   function setupOnce(~)\r\n%   % Create a fixture that takes a while to build\r\n%   pause(rand*10); % 0-10 seconds\r\n%   \r\n%   function testSomeFeature(~)\r\n%   pause(rand); % 0-1 seconds\r\n%   \r\n%   function testAnotherFeature(~)\r\n%   pause(rand); % 0-1 seconds\r\n%\r\n% ...and finally a class-based test with one simple test and one relatively\r\n% long system test:\r\n% \r\n%   classdef aClassBasedTest < matlab.unittest.TestCase\r\n%       methods(Test)\r\n%           function testLongRunningEndToEndWorkflow(~)\r\n%               pause(rand*10); % 0-10 seconds\r\n%           end\r\n%           function testANormalFeature(~)\r\n%               pause(rand); % 0-1 seconds\r\n%           end\r\n%       end\r\n%   end\r\n%\r\n% Using these simple dummy tests we can create a large representative suite\r\n% by just using *|repmat|* and concatenation:\r\n%\r\nimport matlab.unittest.TestSuite;\r\nclassSuite = TestSuite.fromFile('aClassBasedTest.m');\r\nfcnSuite = TestSuite.fromFile('aFunctionBasedTest.m');\r\nscriptSuite = TestSuite.fromFile('aScriptBasedTest.m');\r\n\r\nsuite = [repmat(classSuite, 1, 50), repmat(fcnSuite, 1, 50), repmat(scriptSuite, 1, 50)];\r\n\r\n% Let's run this suite serially to see how long it takes:\r\ntic;\r\nresult = run(suite)\r\ntoc;\r\n\r\n%% Into the Pool\r\n%\r\n% Somewhere in the neighborhood of 6 and a half minutes delay time will\r\n% definitely lose my attention, hopefully to something more productive than\r\n% cat videos (but no guarantee!). What is the simplest way I can run these\r\n% in parallel? Let's try a simple\r\n% <https:\/\/www.mathworks.com\/help\/distcomp\/parfor.html *|parfor|*> loop using\r\n% a parallel pool containing 16 workers:\r\ntic;\r\nparfor idx = 1:numel(suite)\r\n    results(idx) = run(suite(idx));\r\nend\r\nresults\r\ntoc;\r\n\r\n%%\r\n%\r\n% <https:\/\/blogs.mathworks.com\/developer\/files\/2015ParallelismFTW.jpg\r\n% Parallelism FTW!> Now we have the suite running on the order of a minute\r\n% and a half. That is much better time, but it's still not good enough for\r\n% me. Also, what is the deal with the humongous (and unparsable) output?\r\n% Note, I spared you from excessive browser scrolling by actually removing\r\n% (SNIP!) most of the produced output. You can see, however, that\r\n% each test element got its own start\/end lines and different workers all\r\n% printed their output to the command window without any grouping or\r\n% understanding of what ran where. Do you see the lines that look like we\r\n% start aClassBasedTest and finish aFunctionBasedTest? Theres no magic\r\n% test conversion going on here, were are just getting garbled output from\r\n% the workers.\r\n%\r\n%%\r\n% Another interesting tidbit you can see is that the overall testing time\r\n% actually increased significantly. This is not explained by the test\r\n% framework time or the client\/worker communication overhead, because the\r\n% *|Duration|* property of *|TestResult|* only includes the time taken by the\r\n% actual test content. What is actually happening here is that the\r\n% function-based test, which has an expensive *|setupOnce|* function, is not\r\n% enjoying the efficiency benefits of only setting up that fixture once and\r\n% sharing it across multiple tests. Instead this *|setupOnce|* function is\r\n% executed on every element of the function-based test on every worker. The\r\n% benefits of sharing the fixture only apply when a MATLAB runs more than\r\n% one test using that fixture. In this case, we are setting it up for every\r\n% new *|Test|* element that we send to each parallel worker because we are\r\n% sending each suite element one at a time to the pool.\r\n%\r\n%%\r\n% Let's talk next time about how we can improve on this further and tackle\r\n% these problems. In the meantime, have you used parallelism in your\r\n% testing workflow? What works for you?\r\n\r\n##### SOURCE END ##### 973bc21b31e34f5e9f751162e24b05a3\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img src=\"https:\/\/blogs.mathworks.com\/developer\/files\/2015ParallelismFTW.jpg\" class=\"img-responsive attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"\" decoding=\"async\" loading=\"lazy\" \/><\/div><p>\r\n\r\nContentsThe Test SuiteInto the PoolWhen I am neck deep in hardcore TDD Red-Green-Refactor cycles I am constantly looking for ways to ensure that my feedback loop is as quick as possible. If the... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/developer\/2015\/02\/20\/encouragingly-parallel-part-1\/\">read more >><\/a><\/p>","protected":false},"author":90,"featured_media":81,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8,7],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/78"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/users\/90"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/comments?post=78"}],"version-history":[{"count":16,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/78\/revisions"}],"predecessor-version":[{"id":95,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/78\/revisions\/95"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media\/81"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media?parent=78"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/categories?post=78"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/tags?post=78"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}