{"id":508,"date":"2016-05-06T21:36:40","date_gmt":"2016-05-06T21:36:40","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=508"},"modified":"2016-06-01T22:10:47","modified_gmt":"2016-06-01T22:10:47","slug":"performance-principles","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2016\/05\/06\/performance-principles\/","title":{"rendered":"Performance Review Criteria 2: Sticks to Solid Principles"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p>We saw <a href=\"https:\/\/blogs.mathworks.com\/developer\/2016\/04\/11\/performance-ab-testing\/\">last time<\/a> how we can use the performance testing framework to easily see the runtime differences between multiple competing algorithms. This can be done without needing to learn about the philosophies and guiding principles of the framework, indeed you don't even need to know that you are using a performance test framework or that the script you are writing is actually even considered a \"test\".<\/p><p>However (believe you me!), those principles and philosophies are there to be sure. I'd like to walk through some of these principles so that we can all be on the same page and have a good understanding of the design points behind the new framework.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#b2e50e0a-88a1-4582-9fc2-357899d63d83\">Principle 1: Focus on Precision, Accuracy, and Robustness<\/a><\/li><li><a href=\"#2c2fec16-4dec-4277-9c75-0841ddafef3a\">Principle 2: A Performance Test is a Test<\/a><\/li><li><a href=\"#dd840726-d549-4a90-8272-84e1a55992e9\">Principle 3: Report Don't Qualify<\/a><\/li><li><a href=\"#fa55b5d7-afde-4e8e-a2f0-f6d7236c06a7\">Principle 4: Multiple Observations<\/a><\/li><li><a href=\"#f498d86d-8c9e-412d-ad73-f3a4cbbfb55c\">Your principles<\/a><\/li><\/ul><\/div><h4>Principle 1: Focus on Precision, Accuracy, and Robustness<a name=\"b2e50e0a-88a1-4582-9fc2-357899d63d83\"><\/a><\/h4><p>Perhaps this should go without saying. You are building production grade software here and we have a production grade framework to match. The first principle and underlying goal is that the performance measurement is as precise and accurate as possible.<\/p><p>This means, for example, that we include features such as \"warming up\" the code by executing the code first a few times so that initialization effects don't negatively impact the measurement. This allows a better idea of the typical execution time of the code. Features exist to measure the first time performance as well. Also, the measurement boundary is as tightly scoped as possible in order to just measure the relevant code. For example look at the following simple performance test:<\/p><pre class=\"language-matlab\">\r\n<span class=\"keyword\">function<\/span> tests = MyFirstMagicSquarePerfTest\r\ntests = functiontests(localfunctions);  \r\n\r\n<span class=\"keyword\">function<\/span> setup(~)\r\n<span class=\"comment\">% Do something that takes half a second.<\/span>\r\npause(.5);\r\n\r\n<span class=\"keyword\">function<\/span> testCreatingMagicSquare(~)\r\n<span class=\"comment\">% Measure the performance of creating a large(ish) magic square.<\/span>\r\nmagic(1000);\r\n\r\n<\/pre><p>Here I have added a pause to ensure that the TestMethodSetup takes much longer than the code we'd like to measure. However, we can see when we measure its execution time with <b>runperf<\/b> that this content is not included in the measurement:<\/p><pre class=\"codeinput\">measResult = runperf(<span class=\"string\">'MyFirstMagicSquarePerfTest'<\/span>);\r\nsamples = measResult.Samples\r\n<\/pre><pre class=\"codeoutput\">Running MyFirstMagicSquarePerfTest\r\n........\r\nDone MyFirstMagicSquarePerfTest\r\n__________\r\n\r\n\r\nsamples = \r\n\r\n                           Name                           MeasuredTime         Timestamp              Host         Platform           Version                      RunIdentifier            \r\n    __________________________________________________    ____________    ____________________    _____________    ________    _____________________    ____________________________________\r\n\r\n    MyFirstMagicSquarePerfTest\/testCreatingMagicSquare    0.0050525       06-May-2016 17:31:56    MyMachineName    maci64      9.0.0.341360 (R2016a)    b96e12cd-18cf-4f1f-a03e-a0e4e7da8f2d\r\n    MyFirstMagicSquarePerfTest\/testCreatingMagicSquare    0.0048567       06-May-2016 17:31:56    MyMachineName    maci64      9.0.0.341360 (R2016a)    b96e12cd-18cf-4f1f-a03e-a0e4e7da8f2d\r\n    MyFirstMagicSquarePerfTest\/testCreatingMagicSquare    0.0050932       06-May-2016 17:31:57    MyMachineName    maci64      9.0.0.341360 (R2016a)    b96e12cd-18cf-4f1f-a03e-a0e4e7da8f2d\r\n    MyFirstMagicSquarePerfTest\/testCreatingMagicSquare    0.0047803       06-May-2016 17:31:57    MyMachineName    maci64      9.0.0.341360 (R2016a)    b96e12cd-18cf-4f1f-a03e-a0e4e7da8f2d\r\n\r\n<\/pre><p>You can see we ran the test multiple times and each time stayed true to running the test correctly by running the code in setup to provide the <a href=\"http:\/\/xunitpatterns.com\/Fresh%20Fixture.html\">fresh fixture<\/a>. However, you'll notice the half second is not included in the measurement. This is because there is a specific <i>measurement boundary<\/i>. In functions and scripts this boundary is simply the test boundary (test function or test cell). In classes this boundary is the test method boundary by default but can also be directly specified. This is useful for the following test, which poses a problem:<\/p><pre class=\"language-matlab\">\r\n<span class=\"keyword\">classdef<\/span> MySecondMagicSquarePerfTest &lt; matlab.perftest.TestCase\r\n\r\n    <span class=\"keyword\">methods<\/span>(Test)\r\n        <span class=\"keyword\">function<\/span> testCreatingMagicSquare(~)\r\n        \r\n            <span class=\"comment\">% What about some work I need to do in this method?<\/span>\r\n            pause(0.5);\r\n            \r\n            <span class=\"comment\">% Measure the performance of creating a large(ish) magic square.<\/span>\r\n            magic(1000);\r\n        <span class=\"keyword\">end<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\n\r\n<\/pre><pre class=\"codeinput\">measResult = runperf(<span class=\"string\">'MySecondMagicSquarePerfTest'<\/span>);\r\nsamples = measResult.Samples\r\n<\/pre><pre class=\"codeoutput\">Running MySecondMagicSquarePerfTest\r\n........\r\nDone MySecondMagicSquarePerfTest\r\n__________\r\n\r\n\r\nsamples = \r\n\r\n                           Name                            MeasuredTime         Timestamp              Host         Platform           Version                      RunIdentifier            \r\n    ___________________________________________________    ____________    ____________________    _____________    ________    _____________________    ____________________________________\r\n\r\n    MySecondMagicSquarePerfTest\/testCreatingMagicSquare    0.50935         06-May-2016 17:32:00    MyMachineName    maci64      9.0.0.341360 (R2016a)    99746631-7a34-482b-86b7-94e914ae2649\r\n    MySecondMagicSquarePerfTest\/testCreatingMagicSquare    0.50829         06-May-2016 17:32:00    MyMachineName    maci64      9.0.0.341360 (R2016a)    99746631-7a34-482b-86b7-94e914ae2649\r\n    MySecondMagicSquarePerfTest\/testCreatingMagicSquare    0.50571         06-May-2016 17:32:01    MyMachineName    maci64      9.0.0.341360 (R2016a)    99746631-7a34-482b-86b7-94e914ae2649\r\n    MySecondMagicSquarePerfTest\/testCreatingMagicSquare     0.5083         06-May-2016 17:32:01    MyMachineName    maci64      9.0.0.341360 (R2016a)    99746631-7a34-482b-86b7-94e914ae2649\r\n\r\n<\/pre><p>Unfortunately you can see the method boundary includes some extra work and this prevents us from gathering our intended measurement. Never fear however, we've got ya taken care of. In this case you can specifically scope your measurement boundary to exactly the code you'd like to measure in the test method by using the <b>startMeasuring<\/b> and <b>stopMeasuring<\/b> methods to explicitly define your boundary:<\/p><pre class=\"language-matlab\">\r\n<span class=\"keyword\">classdef<\/span> MyThirdMagicSquarePerfTest &lt; matlab.perftest.TestCase\r\n\r\n    <span class=\"keyword\">methods<\/span>(Test)\r\n        <span class=\"keyword\">function<\/span> testCreatingMagicSquare(testCase)\r\n        \r\n            <span class=\"comment\">% What about some work I need to do in this method?<\/span>\r\n            pause(0.5);\r\n            \r\n            <span class=\"comment\">% Measure the performance of creating a large(ish) magic square.<\/span>\r\n            testCase.startMeasuring;\r\n            magic(1000);\r\n            testCase.stopMeasuring;\r\n        <span class=\"keyword\">end<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\n\r\n<\/pre><pre class=\"codeinput\">measResult = runperf(<span class=\"string\">'MyThirdMagicSquarePerfTest'<\/span>);\r\nsamples = measResult.Samples\r\n<\/pre><pre class=\"codeoutput\">Running MyThirdMagicSquarePerfTest\r\n..........\r\n..........\r\n..........\r\n..\r\nDone MyThirdMagicSquarePerfTest\r\n__________\r\n\r\n\r\nsamples = \r\n\r\n                           Name                           MeasuredTime         Timestamp              Host         Platform           Version                      RunIdentifier            \r\n    __________________________________________________    ____________    ____________________    _____________    ________    _____________________    ____________________________________\r\n\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare     0.005019       06-May-2016 17:32:04    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0091478       06-May-2016 17:32:05    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0051195       06-May-2016 17:32:05    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0049833       06-May-2016 17:32:06    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050415       06-May-2016 17:32:06    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050852       06-May-2016 17:32:07    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050112       06-May-2016 17:32:07    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0051059       06-May-2016 17:32:08    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0048692       06-May-2016 17:32:08    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0051603       06-May-2016 17:32:09    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare     0.004889       06-May-2016 17:32:10    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare     0.004917       06-May-2016 17:32:10    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0052781       06-May-2016 17:32:11    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0049215       06-May-2016 17:32:11    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0049284       06-May-2016 17:32:12    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0048642       06-May-2016 17:32:12    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050289       06-May-2016 17:32:13    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0048802       06-May-2016 17:32:13    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare     0.005147       06-May-2016 17:32:14    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050095       06-May-2016 17:32:14    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare     0.005138       06-May-2016 17:32:15    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050985       06-May-2016 17:32:15    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0051649       06-May-2016 17:32:16    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0050265       06-May-2016 17:32:16    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0048852       06-May-2016 17:32:17    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0051176       06-May-2016 17:32:17    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0049506       06-May-2016 17:32:18    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n    MyThirdMagicSquarePerfTest\/testCreatingMagicSquare    0.0051294       06-May-2016 17:32:18    MyMachineName    maci64      9.0.0.341360 (R2016a)    7f8e4608-c8f6-4444-a4b5-4a4c8b2139e8\r\n\r\n<\/pre><p>Voila! Boundary scoped appropriately.<\/p><p>However, even with the proper tight measurement scoping there still is some overhead of the framework such as function call overhead and other overhead involved in actually executing the code and taking the measurement. This is where another feature of the framework comes in. The framework calibrates (or \"tares\" if you want to think of it that way) the measurement by actually measuring an empty test. This gives us a concrete measurement of the overhead involved and this overhead benefits us in two ways. Firstly, we can actually subtract off the measurement overhead to get a more accurate result. Secondly, this calibration measurement gives us an idea of the framework precision, allowing us to alert you to the fact that you are trying to measure something within the framework precision and that you shouldn't trust the results. While this may seem limiting when encountered, it actually is an important feature that prevents you from interpreting the result incorrectly. For example, let's look at a performance test you should never implement:<\/p><pre class=\"language-matlab\">\r\n<span class=\"keyword\">classdef<\/span> BadPerfTest &lt; matlab.perftest.TestCase\r\n\r\n    <span class=\"keyword\">methods<\/span>(Test)\r\n        <span class=\"keyword\">function<\/span> pleaseDoNotTryToMeasureOnePlusOne(~)\r\n            <span class=\"comment\">% One plus one is too fast to measure. Rather than giving you a<\/span>\r\n            <span class=\"comment\">% bunk result we will alert you and the result will be invalid.<\/span>\r\n            1+1;\r\n        <span class=\"keyword\">end<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n<span class=\"keyword\">end<\/span>\r\n\r\n<\/pre><pre class=\"codeinput\">measResult = runperf(<span class=\"string\">'BadPerfTest'<\/span>);\r\nsamples = measResult.Samples\r\n<\/pre><pre class=\"codeoutput\">Running BadPerfTest\r\n..\r\n================================================================================\r\nBadPerfTest\/pleaseDoNotTryToMeasureOnePlusOne was filtered.\r\n    Test Diagnostic: The MeasuredTime should not be too close to the precision of the framework.\r\n================================================================================\r\n\r\nDone BadPerfTest\r\n__________\r\n\r\nFailure Summary:\r\n\r\n     Name                                           Failed  Incomplete  Reason(s)\r\n    ============================================================================================\r\n     BadPerfTest\/pleaseDoNotTryToMeasureOnePlusOne              X       Filtered by assumption.\r\n    \r\n\r\nsamples = \r\n\r\n   empty 0-by-7 table\r\n\r\n<\/pre><p>We don't even let you run it. The result you get back is invalid. To see what we would be trying to measure we have to do this manually with the poor man's version:<\/p><pre class=\"codeinput\">results = zeros(1,1e6);\r\n<span class=\"keyword\">for<\/span> idx=1:1e6\r\n    t = tic;\r\n    1+1;\r\n    results(idx) = toc(t);\r\n<span class=\"keyword\">end<\/span>\r\nax = axes;\r\nplot(ax, results)\r\nax.YLim(2) = 3*std(results); <span class=\"comment\">% Cap the y limit to 3 standard deviations<\/span>\r\n\r\ntitle(ax, <span class=\"string\">'The Garbage Results When Trying to Measure 1+1'<\/span>);\r\nylabel(ax, <span class=\"string\">'Execution time up to 3 standard deviations.'<\/span>);\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/developer\/files\/2016GarbageMeasurement.png\" alt=\"\"> <p>Garbage. Noise. Not what you want the framework to hand back, so we don't. We recognize a low confidence result and we let you know.<\/p><h4>Principle 2: A Performance Test is a Test<a name=\"2c2fec16-4dec-4277-9c75-0841ddafef3a\"><\/a><\/h4><p>The next principle is that a performance test is simply a test. It is a test that uses the familiar <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/script-based-unit-tests.html\">script<\/a>, <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/function-based-unit-tests.html\">function<\/a>, and <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/class-based-unit-tests.html\">class<\/a> based testing APIs, and as such can be run as normal tests to validate that the software is not broken. However, the same test content can also be run differently in order to measure the performance of the measured code. The behavior is driven by <i>how it is run<\/i>, not <i>what is in the test<\/i>!<\/p><p>If you have already used the unit test framework, there is remarkably little you need to learn when diving into performance testing! Please use the extra mental energy toward putting mankind on mars or something amazing like that.<\/p><p>This also means that your performance tests can be checked into your CI system and will fail the build if a change somehow breaks the performance tests.<\/p><p>Look, here's the deal, I can make all my code go really fast if it doesn't have to do the right thing. To prevent this, we can put a spot check in to ensure the software is not only fast but also correct. Let's see this in action with another magic square test:<\/p><pre class=\"language-matlab\">\r\n<span class=\"keyword\">classdef<\/span> MyFourthMagicSquarePerfTest &lt; matlab.perftest.TestCase\r\n    \r\n   \r\n    <span class=\"keyword\">methods<\/span>(Test)\r\n        <span class=\"keyword\">function<\/span> createMagicSquare(testCase)\r\n            import <span class=\"string\">matlab.unittest.constraints.IsEqualTo<\/span>;\r\n            import <span class=\"string\">matlab.unittest.constraints.EveryElementOf<\/span>;\r\n            \r\n            testCase.startMeasuring;\r\n            m = magic(100);\r\n            testCase.stopMeasuring;\r\n            \r\n            columnSum = sum(m);\r\n            rowSum = sum(m,2);\r\n            \r\n            testCase.verifyThat(EveryElementOf(columnSum), IsEqualTo(columnSum(1)), <span class=\"keyword\">...<\/span>\r\n                <span class=\"string\">'All the values of column sum should be equal.'<\/span>); \r\n            testCase.verifyThat(rowSum, IsEqualTo(columnSum), <span class=\"keyword\">...<\/span>\r\n                <span class=\"string\">'The column sum should equal the transpose of the row sum.'<\/span>); \r\n        <span class=\"keyword\">end<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n    \r\n<span class=\"keyword\">end<\/span>\r\n\r\n<\/pre><p>Now let's run it using <b><i>runtests<\/i><\/b> instead of <b><i>runperf<\/i><\/b> :<\/p><pre class=\"codeinput\">result = runtests(<span class=\"string\">'MyFourthMagicSquarePerfTest'<\/span>)\r\n<\/pre><pre class=\"codeoutput\">Running MyFourthMagicSquarePerfTest\r\n\r\n================================================================================\r\nVerification failed in MyFourthMagicSquarePerfTest\/createMagicSquare.\r\n\r\n    ----------------\r\n    Test Diagnostic:\r\n    ----------------\r\n    The column sum should equal the transpose of the row sum.\r\n\r\n    ---------------------\r\n    Framework Diagnostic:\r\n    ---------------------\r\n    IsEqualTo failed.\r\n    --&gt; NumericComparator failed.\r\n        --&gt; Size check failed.\r\n            --&gt; Sizes do not match.\r\n                \r\n                Actual size:\r\n                       100     1\r\n                Expected size:\r\n                         1   100\r\n    \r\n    Actual double:\r\n        100x1 double\r\n    Expected double:\r\n        1x100 double\r\n\r\n    ------------------\r\n    Stack Information:\r\n    ------------------\r\n    In \/mathworks\/inside\/files\/dev\/tools\/tia\/mdd\/performancePrinciples\/MyFourthMagicSquarePerfTest.m (MyFourthMagicSquarePerfTest.createMagicSquare) at 18\r\n================================================================================\r\n.\r\nDone MyFourthMagicSquarePerfTest\r\n__________\r\n\r\nFailure Summary:\r\n\r\n     Name                                           Failed  Incomplete  Reason(s)\r\n    ============================================================================================\r\n     MyFourthMagicSquarePerfTest\/createMagicSquare    X                 Failed by verification.\r\n    \r\n\r\nresult = \r\n\r\n  TestResult with properties:\r\n\r\n          Name: 'MyFourthMagicSquarePerfTest\/createMagicSquare'\r\n        Passed: 0\r\n        Failed: 1\r\n    Incomplete: 0\r\n      Duration: 0.0238\r\n       Details: [1x1 struct]\r\n\r\nTotals:\r\n   0 Passed, 1 Failed, 0 Incomplete.\r\n   0.023783 seconds testing time.\r\n\r\n<\/pre><p>Look at that, we can run the performance test like a functional regressions test and it fails! In this case the problem is that we should be verifying that the row sum is the transpose of the column sum but the test is comparing them directly. Here the test itself is the source of the problem not the code under test, but bugs in source code would fail it just the same.<\/p><h4>Principle 3: Report Don't Qualify<a name=\"dd840726-d549-4a90-8272-84e1a55992e9\"><\/a><\/h4><p>With the example above we ran this test using <b>runtests<\/b> to ensure it was correct. Note, that the result was indeed binary. The test fails or passes. In this case it failed and you can throw that back to where it came from and fix it. However, when measuring performance we don't have such luxury. When observing performance we are running an experiment and gathering a measurement and we need some interpretation of the results to determine whether it is good or bad. Often times this interpretation is only possible in some context, such as whether the code was faster or slower than last check-in, or last month. This difference in how the results are to be interpreted affects the resulting output data structure. Let's take a look using the same buggy test but running it as a performance test instead:<\/p><pre class=\"codeinput\">measResult = runperf(<span class=\"string\">'MyFourthMagicSquarePerfTest'<\/span>)\r\n<\/pre><pre class=\"codeoutput\">Running MyFourthMagicSquarePerfTest\r\n\r\n================================================================================\r\nVerification failed in MyFourthMagicSquarePerfTest\/createMagicSquare.\r\n\r\n    ----------------\r\n    Test Diagnostic:\r\n    ----------------\r\n    The column sum should equal the transpose of the row sum.\r\n\r\n    ---------------------\r\n    Framework Diagnostic:\r\n    ---------------------\r\n    IsEqualTo failed.\r\n    --&gt; NumericComparator failed.\r\n        --&gt; Size check failed.\r\n            --&gt; Sizes do not match.\r\n                \r\n                Actual size:\r\n                       100     1\r\n                Expected size:\r\n                         1   100\r\n    \r\n    Actual double:\r\n        100x1 double\r\n    Expected double:\r\n        1x100 double\r\n\r\n    ------------------\r\n    Stack Information:\r\n    ------------------\r\n    In \/mathworks\/inside\/files\/dev\/tools\/tia\/mdd\/performancePrinciples\/MyFourthMagicSquarePerfTest.m (MyFourthMagicSquarePerfTest.createMagicSquare) at 18\r\n================================================================================\r\n.\r\n================================================================================\r\nMyFourthMagicSquarePerfTest\/createMagicSquare was filtered.\r\n    Test Diagnostic: The MeasuredTime should not be too close to the precision of the framework.\r\n================================================================================\r\n\r\nDone MyFourthMagicSquarePerfTest\r\n__________\r\n\r\nFailure Summary:\r\n\r\n     Name                                           Failed  Incomplete  Reason(s)\r\n    ============================================================================================\r\n     MyFourthMagicSquarePerfTest\/createMagicSquare    X         X       Filtered by assumption.\r\n                                                                        Failed by verification.\r\n    \r\n\r\nmeasResult = \r\n\r\n  MeasurementResult with properties:\r\n\r\n            Name: 'MyFourthMagicSquarePerfTest\/createMagicSquare'\r\n           Valid: 0\r\n         Samples: [0x7 table]\r\n    TestActivity: [1x12 table]\r\n\r\nTotals:\r\n   0 Valid, 1 Invalid.\r\n\r\n<\/pre><p>Note the absence of a Passed\/Failed bit on the resulting data structure. Even for this test which is clearly problematic, we are in the context of performance testing and the focus is on measurement and reporting rather than qualification. What we do say is whether the measurement result is <b><i>Valid<\/i><\/b> or not. The valid property is simply a logical property that specifies whether you even want to begin analyzing the measurement results. The test failed? Invalid. The test was filtered? Invalid. In order to be valid the test needs to complete and pass. That is the deterministic state that we can trust provides us with the correct performance measurement.<\/p><p>Let's fix the bug and look a bit more:<\/p><pre class=\"language-matlab\">\r\n<span class=\"keyword\">classdef<\/span> MyFifthMagicSquarePerfTest &lt; matlab.perftest.TestCase\r\n    \r\n   \r\n    <span class=\"keyword\">methods<\/span>(Test)\r\n        <span class=\"keyword\">function<\/span> createMagicSquare(testCase)\r\n            import <span class=\"string\">matlab.unittest.constraints.IsEqualTo<\/span>;\r\n            import <span class=\"string\">matlab.unittest.constraints.EveryElementOf<\/span>;\r\n            \r\n            testCase.startMeasuring;\r\n            m = magic(1000);\r\n            testCase.stopMeasuring;\r\n            \r\n            columnSum = sum(m);\r\n            rowSum = sum(m,2);\r\n            \r\n            testCase.verifyThat(EveryElementOf(columnSum), IsEqualTo(columnSum(1)), <span class=\"keyword\">...<\/span>\r\n                <span class=\"string\">'All the values of column sum should be equal.'<\/span>); \r\n            testCase.verifyThat(rowSum, IsEqualTo(columnSum.'), <span class=\"keyword\">...<\/span>\r\n                <span class=\"string\">'The column sum should equal the transpose of the row sum.'<\/span>); \r\n        <span class=\"keyword\">end<\/span>\r\n    <span class=\"keyword\">end<\/span>\r\n    \r\n<span class=\"keyword\">end<\/span>\r\n\r\n<\/pre><pre class=\"codeinput\">measResult = runperf(<span class=\"string\">'MyFifthMagicSquarePerfTest'<\/span>)\r\n<\/pre><pre class=\"codeoutput\">Running MyFifthMagicSquarePerfTest\r\n........\r\nDone MyFifthMagicSquarePerfTest\r\n__________\r\n\r\n\r\nmeasResult = \r\n\r\n  MeasurementResult with properties:\r\n\r\n            Name: 'MyFifthMagicSquarePerfTest\/createMagicSquare'\r\n           Valid: 1\r\n         Samples: [4x7 table]\r\n    TestActivity: [8x12 table]\r\n\r\nTotals:\r\n   1 Valid, 0 Invalid.\r\n\r\n<\/pre><p>Alright, we have a valid result. Now we can look at the Samples to see that everything is working normally:<\/p><pre class=\"codeinput\">measResult.Samples\r\n<\/pre><pre class=\"codeoutput\">\r\nans = \r\n\r\n                        Name                        MeasuredTime         Timestamp              Host         Platform           Version                      RunIdentifier            \r\n    ____________________________________________    ____________    ____________________    _____________    ________    _____________________    ____________________________________\r\n\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    0.0049167       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    0.0049236       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    0.0049038       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare     0.004911       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n\r\n<\/pre><p>...and now we can apply whatever statistic we are interested in on the resulting data:<\/p><pre class=\"codeinput\">median(measResult.Samples.MeasuredTime)\r\n<\/pre><pre class=\"codeoutput\">\r\nans =\r\n\r\n    0.0049\r\n\r\n<\/pre><p>Note that you can also see all of the information, including the warmup runs and the actual TestResult data by looking at the TestActivity table.<\/p><pre class=\"codeinput\">measResult.TestActivity\r\n<\/pre><pre class=\"codeoutput\">\r\nans = \r\n\r\n                        Name                        Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp              Host         Platform           Version                      TestResult                          RunIdentifier            \r\n    ____________________________________________    ______    ______    __________    ____________    _________    ____________________    _____________    ________    _____________________    ________________________________    ____________________________________\r\n\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0048825       warmup       06-May-2016 17:32:20    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0050761       warmup       06-May-2016 17:32:20    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0049342       warmup       06-May-2016 17:32:20    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0050149       warmup       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0049167       sample       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0049236       sample       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false         0.0049038       sample       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n    MyFifthMagicSquarePerfTest\/createMagicSquare    true      false     false          0.004911       sample       06-May-2016 17:32:21    MyMachineName    maci64      9.0.0.341360 (R2016a)    [1x1 matlab.unittest.TestResult]    b7b651ff-1c19-40a4-b323-e5afd9b0d77b\r\n\r\n<\/pre><h4>Principle 4: Multiple Observations<a name=\"fa55b5d7-afde-4e8e-a2f0-f6d7236c06a7\"><\/a><\/h4><p>If you've made it this far I think you can already see that each test is run multiple times and multiple measurements were taken. Actually, a key piece of the technology that enabled the performance framework was extending the underlying test framework to support running each test repeatedly.<\/p><p>Repeated measurements are important of course because we are taking measurements in the presence of noise, and it follows that we need to measure a representative sample in order to arrive at the distribution and\/or a reasonable deduction as to the performance of the code. The principle here is that we measure multiple executions and provide you with the data you need to analyze the behavior. Whether you use classical statistical measures, Bayesian approaches, or something else, the resulting data set is at your command.<\/p><h4>Your principles<a name=\"f498d86d-8c9e-412d-ad73-f3a4cbbfb55c\"><\/a><\/h4><p>So that's a solid overview of the principles and philosophies of the new MATLAB performance framework. Do you have any other principles you have encountered when measuring the performance of your code?<\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_dd9eada0109942d889fe4f82d21af546() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='dd9eada0109942d889fe4f82d21af546 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' dd9eada0109942d889fe4f82d21af546';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2016 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_dd9eada0109942d889fe4f82d21af546()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2016a<br><\/p><\/div><!--\r\ndd9eada0109942d889fe4f82d21af546 ##### SOURCE BEGIN #####\r\n%%\r\n% We saw\r\n% <https:\/\/blogs.mathworks.com\/developer\/2016\/04\/11\/performance-ab-testing\/\r\n% last time> how we can use the performance testing framework to easily see\r\n% the runtime differences between multiple competing algorithms. This can\r\n% be done without needing to learn about the philosophies and guiding\r\n% principles of the framework, indeed you don't even need to know that you\r\n% are using a performance test framework or that the script you are writing\r\n% is actually even considered a \"test\".\r\n%\r\n% However (believe you me!), those principles and philosophies are there to be\r\n% sure. I'd like to walk through some of these principles so that we can\r\n% all be on the same page and have a good understanding of the design\r\n% points behind the new framework.\r\n\r\n%% Principle 1: Focus on Precision, Accuracy, and Robustness\r\n% Perhaps this should go without saying. You are building production\r\n% grade software here and we have a production grade framework to match.\r\n% The first principle and underlying goal is that the performance\r\n% measurement is as precise and accurate as possible.\r\n%\r\n% This means, for example, that we include features such as \"warming up\"\r\n% the code by executing the code first a few times so that initialization\r\n% effects don't negatively impact the measurement. This allows a better\r\n% idea of the typical execution time of the code. Features exist to measure\r\n% the first time performance as well. Also, the measurement boundary is as\r\n% tightly scoped as possible in order to just measure the relevant code.\r\n% For example look at the following simple performance test:\r\n%  \r\n% <include>MyFirstMagicSquarePerfTest.m<\/include>\r\n%\r\n% Here I have added a pause to ensure that the TestMethodSetup takes much\r\n% longer than the code we'd like to measure. However, we can see when we\r\n% measure its execution time with *runperf* that this content is not\r\n% included in the measurement:\r\nmeasResult = runperf('MyFirstMagicSquarePerfTest');\r\nsamples = measResult.Samples\r\n\r\n%%\r\n% You can see we ran the test multiple times and each time stayed true to\r\n% running the test correctly by running the code in setup to provide the\r\n% <http:\/\/xunitpatterns.com\/Fresh%20Fixture.html fresh fixture>. However,\r\n% you'll notice the half second is not included in the measurement. This is\r\n% because there is a specific _measurement boundary_. In functions and\r\n% scripts this boundary is simply the test boundary (test function or test\r\n% cell). In classes this boundary is the test method boundary by default\r\n% but can also be directly specified. This is useful for the following\r\n% test, which poses a problem:\r\n%\r\n% <include>MySecondMagicSquarePerfTest<\/include>\r\nmeasResult = runperf('MySecondMagicSquarePerfTest');\r\nsamples = measResult.Samples\r\n\r\n%%\r\n% Unfortunately you can see the method boundary includes some extra work\r\n% and this prevents us from gathering our intended measurement. Never fear\r\n% however, we've got ya taken care of. In this case you can specifically\r\n% scope your measurement boundary to exactly the code you'd like to measure\r\n% in the test method by using the *startMeasuring* and *stopMeasuring* methods\r\n% to explicitly define your boundary:\r\n%\r\n% <include>MyThirdMagicSquarePerfTest<\/include>\r\nmeasResult = runperf('MyThirdMagicSquarePerfTest');\r\nsamples = measResult.Samples\r\n\r\n%%\r\n% Voila! Boundary scoped appropriately. \r\n%\r\n% However, even with the proper tight measurement scoping there still is\r\n% some overhead of the framework such as function call overhead and other\r\n% overhead involved in actually executing the code and taking the\r\n% measurement. This is where another feature of the framework comes in. The\r\n% framework calibrates (or \"tares\" if you want to think of it that way) the\r\n% measurement by actually measuring an empty test. This gives us a concrete\r\n% measurement of the overhead involved and this overhead benefits us in two\r\n% ways. Firstly, we can actually subtract off the measurement overhead to\r\n% get a more accurate result. Secondly, this calibration measurement gives\r\n% us an idea of the framework precision, allowing us to alert you to the\r\n% fact that you are trying to measure something within the framework\r\n% precision and that you shouldn't trust the results. While this may seem\r\n% limiting when encountered, it actually is an important feature that\r\n% prevents you from interpreting the result incorrectly. For example, let's\r\n% look at a performance test you should never implement:\r\n%\r\n% <include>BadPerfTest<\/include>\r\n%\r\nmeasResult = runperf('BadPerfTest');\r\nsamples = measResult.Samples\r\n\r\n%% \r\n% We don't even let you run it. The result you get back is invalid. To see\r\n% what we would be trying to measure we have to do this manually with the\r\n% poor man's version:\r\nresults = zeros(1,1e6);\r\nfor idx=1:1e6\r\n    t = tic;\r\n    1+1;\r\n    results(idx) = toc(t);\r\nend\r\nax = axes;\r\nplot(ax, results)\r\nax.YLim(2) = 3*std(results); % Cap the y limit to 3 standard deviations\r\n\r\ntitle(ax, 'The Garbage Results When Trying to Measure 1+1');\r\nylabel(ax, 'Execution time up to 3 standard deviations.');\r\n\r\n%%\r\n% Garbage. Noise. Not what you want the framework to hand back, so we\r\n% don't. We recognize a low confidence result and we let you know.\r\n\r\n%% Principle 2: A Performance Test is a Test\r\n% The next principle is that a performance test is simply a test. It is a\r\n% test that uses the familiar\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/script-based-unit-tests.html\r\n% script>,\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/function-based-unit-tests.html\r\n% function>, and\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/class-based-unit-tests.html class>\r\n% based testing APIs, and as such can be run as normal tests to validate\r\n% that the software is not broken. However, the same test content can also\r\n% be run differently in order to measure the performance of the measured\r\n% code. The behavior is driven by _how it is run_, not _what is in the\r\n% test_!\r\n% \r\n% If you have already used the unit test framework, there is remarkably\r\n% little you need to learn when diving into performance testing! Please use\r\n% the extra mental energy toward putting mankind on mars or something\r\n% amazing like that.\r\n%\r\n% This also means that your performance tests can be checked into your CI\r\n% system and will fail the build if a change somehow breaks the performance\r\n% tests.\r\n%\r\n% Look, here's the deal, I can make all my code go really fast if it\r\n% doesn't have to do the right thing. To prevent this, we can put a spot\r\n% check in to ensure the software is not only fast but also correct. Let's\r\n% see this in action with another magic square test:\r\n%\r\n% <include>MyFourthMagicSquarePerfTest.m<\/include>\r\n%\r\n% Now let's run it using *_runtests_* instead of *_runperf_* :\r\nresult = runtests('MyFourthMagicSquarePerfTest')\r\n\r\n%%\r\n% Look at that, we can run the performance test like a functional\r\n% regressions test and it fails! In this case the problem is that we should\r\n% be verifying that the row sum is the transpose of the column sum but the\r\n% test is comparing them directly. Here the test itself is the source of\r\n% the problem not the code under test, but bugs in source code would fail\r\n% it just the same.\r\n\r\n%% Principle 3: Report Don't Qualify\r\n% With the example above we ran this test using *runtests* to ensure it was\r\n% correct. Note, that the result was indeed binary. The test fails or\r\n% passes. In this case it failed and you can throw that back to where it\r\n% came from and fix it. However, when measuring performance we don't have\r\n% such luxury. When observing performance we are running an experiment and\r\n% gathering a measurement and we need some interpretation of the results to\r\n% determine whether it is good or bad. Often times this interpretation is\r\n% only possible in some context, such as whether the code was faster or\r\n% slower than last check-in, or last month. This difference in how the\r\n% results are to be interpreted affects the resulting output data\r\n% structure. Let's take a look using the same buggy test but running it as\r\n% a performance test instead:\r\nmeasResult = runperf('MyFourthMagicSquarePerfTest')\r\n\r\n%% \r\n% Note the absence of a Passed\/Failed bit on the resulting data structure.\r\n% Even for this test which is clearly problematic, we are in the context of\r\n% performance testing and the focus is on measurement and reporting rather\r\n% than qualification. What we do say is whether the measurement result is\r\n% *_Valid_* or not. The valid property is simply a logical property that\r\n% specifies whether you even want to begin analyzing the measurement\r\n% results. The test failed? Invalid. The test was filtered? Invalid. In\r\n% order to be valid the test needs to complete and pass. That is the\r\n% deterministic state that we can trust provides us with the correct\r\n% performance measurement.\r\n%\r\n% Let's fix the bug and look a bit more:\r\n%\r\n% <include>MyFifthMagicSquarePerfTest<\/include>\r\nmeasResult = runperf('MyFifthMagicSquarePerfTest')\r\n\r\n%%\r\n% Alright, we have a valid result. Now we can look at the Samples to see\r\n% that everything is working normally:\r\nmeasResult.Samples\r\n\r\n%%\r\n% ...and now we can apply whatever statistic we are interested in on the\r\n% resulting data:\r\nmedian(measResult.Samples.MeasuredTime)\r\n\r\n%% \r\n% Note that you can also see all of the information, including the warmup\r\n% runs and the actual TestResult data by looking at the TestActivity table.\r\nmeasResult.TestActivity\r\n\r\n%% Principle 4: Multiple Observations\r\n% If you've made it this far I think you can already see that each test is\r\n% run multiple times and multiple measurements were taken. Actually, a key\r\n% piece of the technology that enabled the performance framework was\r\n% extending the underlying test framework to support running each test\r\n% repeatedly. \r\n%\r\n% Repeated measurements are important of course because we are taking\r\n% measurements in the presence of noise, and it follows that we need to\r\n% measure a representative sample in order to arrive at the distribution\r\n% and\/or a reasonable deduction as to the performance of the code. The\r\n% principle here is that we measure multiple executions and provide you\r\n% with the data you need to analyze the behavior. Whether you use classical\r\n% statistical measures, Bayesian approaches, or something else. The data\r\n% is at your command.\r\n\r\n%% Your principles\r\n% So that's a solid overview of the principles and philosophies of the\r\n% new MATLAB performance framework. Do you have any other principles you\r\n% have encountered when measuring the performance of your code?\r\n\r\n\r\n\r\n\r\n\r\n##### SOURCE END ##### dd9eada0109942d889fe4f82d21af546\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/developer\/files\/2016GarbageMeasurement.png\" onError=\"this.style.display ='none';\" \/><\/div><!--introduction--><p>We saw <a href=\"https:\/\/blogs.mathworks.com\/developer\/2016\/04\/11\/performance-ab-testing\/\">last time<\/a> how we can use the performance testing framework to easily see the runtime differences between multiple competing algorithms. This can be done without needing to learn about the philosophies and guiding principles of the framework, indeed you don't even need to know that you are using a performance test framework or that the script you are writing is actually even considered a \"test\".... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/developer\/2016\/05\/06\/performance-principles\/\">read more >><\/a><\/p>","protected":false},"author":90,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13,7],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/508"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/users\/90"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/comments?post=508"}],"version-history":[{"count":6,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/508\/revisions"}],"predecessor-version":[{"id":522,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/posts\/508\/revisions\/522"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/media?parent=508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/categories?post=508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/developer\/wp-json\/wp\/v2\/tags?post=508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}