{"id":1901,"date":"2018-10-05T18:26:56","date_gmt":"2018-10-05T22:26:56","guid":{"rendered":"https:\/\/blogs.mathworks.com\/developer\/?p=1901"},"modified":"2018-10-05T18:34:47","modified_gmt":"2018-10-05T22:34:47","slug":"keepmeasuring","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/developer\/2018\/10\/05\/keepmeasuring\/","title":{"rendered":"Just Keep Swimming"},"content":{"rendered":"

Remember Dory?<\/p>

\"\" <\/p>

Image Credit: Silvio Tanaka [ CC BY 2.0<\/a> ], via Wikimedia Commons<\/a><\/i><\/p>

The model of persistence in the face of difficult circumstances, the hilarious and free spirited fish of the vast ocean expanse, the adopted aunt of our lovable Nemo, that Dory?<\/p>

Well she was ahead of her time. Who knew of the wisdom of her sage advice, just keep swimming.<\/p>

We need a little bit of that sometimes when we are performance testing. Specifically, we need to just keep swimming (so to speak) when we are measuring code that is just too fast. For example, let's take the example from last post<\/a> (CQ's matrix library). The performance tests we wrote here look like this:<\/p>

\r\nclassdef<\/span> tMatrixLibrary < matlab.perftest.TestCase\r\n    \r\n    properties<\/span>(TestParameter)\r\n        TestMatrix = struct('midSize'<\/span>, magic(600),...<\/span>\r\n            'largeSize'<\/span>, magic(1000));\r\n    end<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testSum(testCase, TestMatrix)\r\n            matrix_sum(TestMatrix);\r\n        end<\/span>\r\n        \r\n        function<\/span> testMean(testCase, TestMatrix)\r\n            matrix_mean(TestMatrix);\r\n        end<\/span>\r\n        \r\n        function<\/span> testEig(testCase, TestMatrix)\r\n            \r\n            testCase.assertReturnsTrue(@() size(TestMatrix,1) == size(TestMatrix,2), ...<\/span>\r\n                'Eig only works on square matrix'<\/span>);\r\n            testCase.startMeasuring;\r\n            matrix_eig(TestMatrix);\r\n            testCase.stopMeasuring;\r\n            \r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n<\/pre>
Here you can see that we tested against a \"medium size\" problem and a \"large size\" problem. (Un)conveniently missing, however, is a \"small size\" problem. Why is this? Well, why don't we add one...<\/p>
\r\nclassdef<\/span> tMatrixLibrary_v2 < matlab.perftest.TestCase\r\n    \r\n    properties<\/span>(TestParameter)\r\n        TestMatrix = struct('smallSize'<\/span>, magic(100), 'midSize'<\/span>, magic(600),...<\/span>\r\n            'largeSize'<\/span>, magic(1000));\r\n    end<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testSum(testCase, TestMatrix)\r\n            matrix_sum(TestMatrix);\r\n        end<\/span>\r\n        \r\n        function<\/span> testMean(testCase, TestMatrix)\r\n            matrix_mean(TestMatrix);\r\n        end<\/span>\r\n        \r\n        function<\/span> testEig(testCase, TestMatrix)\r\n            \r\n            testCase.assertReturnsTrue(@() size(TestMatrix,1) == size(TestMatrix,2), ...<\/span>\r\n                'Eig only works on square matrix'<\/span>);\r\n            testCase.startMeasuring;\r\n            matrix_eig(TestMatrix);\r\n            testCase.stopMeasuring;\r\n            \r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n<\/pre>
...along with a quick function to check the validity of the result and we'll find out:<\/p>
\r\nfunction<\/span> checkResults(results)\r\ndisp(newline)\r\ndispFrame\r\nif<\/span> ~all([results.Valid])\r\n    disp('Oh no Dory, some measurements were invalid!'<\/span>)\r\n\r\nelse<\/span>\r\n    disp('Thanks Dory, you''re the best! All our measurements are good.'<\/span>)\r\nend<\/span>\r\ndispFrame\r\n\r\nend<\/span>\r\nfunction<\/span> dispFrame\r\ndisp(':::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::'<\/span>);\r\nend<\/span>\r\n\r\n<\/pre>
results = runperf('tMatrixLibrary_v2'<\/span>);\r\ncheckResults(results)\r\n<\/pre>
Running tMatrixLibrary_v2\r\n........\r\n================================================================================\r\ntMatrixLibrary_v2\/testSum(TestMatrix=smallSize) was filtered.\r\n    Test Diagnostic: The MeasuredTime should not be too close to the precision of the framework.\r\n================================================================================\r\n.. .......... .......... .......... .......\r\n================================================================================\r\ntMatrixLibrary_v2\/testMean(TestMatrix=smallSize) was filtered.\r\n    Test Diagnostic: The MeasuredTime should not be too close to the precision of the framework.\r\n================================================================================\r\n...\r\n.......... .......... .......... .......... .Warning: Target Relative Margin of Error not met after running the MaxSamples\r\nfor tMatrixLibrary_v2\/testMean(TestMatrix=largeSize). \r\n.........\r\n.......... .....\r\nDone tMatrixLibrary_v2\r\n__________\r\n\r\nFailure Summary:\r\n\r\n     Name                                              Failed  Incomplete  Reason(s)\r\n    ===============================================================================================\r\n     tMatrixLibrary_v2\/testSum(TestMatrix=smallSize)               X       Filtered by assumption.\r\n    -----------------------------------------------------------------------------------------------\r\n     tMatrixLibrary_v2\/testMean(TestMatrix=smallSize)              X       Filtered by assumption.\r\n\r\n\r\n:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\r\nOh no Dory, some measurements were invalid!\r\n:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\r\n<\/pre>
Uh oh, that doesn't look ideal. It looks like some of our small sized measurements weren't valid. They weren't valid because the framework recognized that the code execution time was too close to the measurable precision of the framework. The framework was able to determine that it would have been a garbage measurement, so rather than risk providing bad data as a result it has proactively filtered the tests that were too fast and marked the result as invalid. Well, I still want to measure this fast case, so what do we do? Typically, what we see happening is people wrapping their code with a static for<\/tt><\/b> loop, like so:<\/p>
\r\nclassdef<\/span> tMatrixLibrary_v3 < matlab.perftest.TestCase\r\n    \r\n    properties<\/span>(TestParameter)\r\n        TestMatrix = struct('smallSize'<\/span>, magic(100), 'midSize'<\/span>, magic(600),...<\/span>\r\n            'largeSize'<\/span>, magic(1000));\r\n    end<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testSum(testCase, TestMatrix)\r\n            for<\/span> idx = 1:1000\r\n                matrix_sum(TestMatrix);\r\n            end<\/span>\r\n        end<\/span>\r\n        \r\n        function<\/span> testMean(testCase, TestMatrix)\r\n            for<\/span> idx = 1:1000\r\n                matrix_mean(TestMatrix);\r\n            end<\/span>\r\n        end<\/span>\r\n        \r\n        function<\/span> testEig(testCase, TestMatrix)\r\n            \r\n            testCase.assertReturnsTrue(@() size(TestMatrix,1) == size(TestMatrix,2), ...<\/span>\r\n                'Eig only works on square matrix'<\/span>);\r\n            testCase.startMeasuring;\r\n            matrix_eig(TestMatrix);\r\n            testCase.stopMeasuring;\r\n            \r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n<\/pre>
results = runperf('tMatrixLibrary_v3'<\/span>);\r\ncheckResults(results)\r\n<\/pre>
Running tMatrixLibrary_v3\r\n.......... .......... .......... .......... ..........\r\n.......... .......... ..\r\nDone tMatrixLibrary_v3\r\n__________\r\n\r\n\r\n\r\n:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\r\nThanks Dory, you're the best! All our measurements are good.\r\n:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\r\n<\/pre>
Well the results look good at least. Think about this like you were measuring the weight of a feather using a kitchen scale. Measuring one feather will give you a bunk measurement, so instead you gather together 1000 or more feathers, put them in a box, and measure the whole thing to get a good idea of the average weight of a feather.<\/p>
Problem solved? Well not really. There are some big drawbacks to this approach. Let's enumerate:<\/p>
  1. I had to choose a number of iterations to test against. What's more, this choice was arbitrary. How do I know that I am not right on the edge of framework precision? If so I am likely to experience tests that sporadically are too fast to measure! In addition, the framework precision is machine dependent, so different machines will have different precision thresholds. There is no one good number to go off of so it's anyone's guess. Not comforting.<\/li>
  2. #ohmygoodness<\/b><\/i> was this slow! Why was it slow? Because I had to run 1000 iterations for everything, including the larger matrix sizes. These larger sizes don't need to be run in a loop, but in order to maintain an apples\/oranges comparison they need to if the smaller sizes are.<\/li>
  3. This approach falls flat when comparing other algorithms against each other. If one algorithm needs 1000 iterations, but the other only needs 750, we quickly get into comparing apples and oranges and we lose insight into our true performance.<\/li>
  4. Let's say we diligently track our code performance over time, and furthermore we do a bang-up job optimizing our critical code and make vast improvements in the code performance. Well this improvement may require that we \"up\" the iteration count, since 1000 iterations may suddenly become too fast to measure as a result of our code optimizations and we need to now measure 10,000 iterations. Once we do this however, all future measurements are on a different scale than our historical data. Lame.<\/li>
  5. Finally, and perhaps as a root cause of some of these apples\/oranges troubles discussed above, it is simply not the true measurement of the code's execution time.<\/li><\/ol><\/div>
    So takeaway, once your code hits the limitations of how fast we can measure something, we can add a static for<\/tt><\/b> loop and maybe hobble along, but it's really not the best experience.<\/p>
    Enter R2018b<\/b>.<\/i><\/p>
    In R2018b the performance testing framework now has a new keepMeasuring<\/tt><\/b> method on the matlab.perftest.TestCase<\/tt><\/b> class to support faster code measurement workflows. How is this used? Put it in a while loop and let the framework determine the right number of iterations:<\/p>
    \r\nclassdef<\/span> tMatrixLibrary_final < matlab.perftest.TestCase\r\n    \r\n    properties<\/span>(TestParameter)\r\n        TestMatrix = struct('smallSize'<\/span>, magic(100), 'midSize'<\/span>, magic(600),...<\/span>\r\n            'largeSize'<\/span>, magic(1000));\r\n    end<\/span>\r\n    \r\n    methods<\/span>(Test)\r\n        function<\/span> testSum(testCase, TestMatrix)\r\n            while<\/span> testCase.keepMeasuring\r\n                matrix_sum(TestMatrix);\r\n            end<\/span>\r\n        end<\/span>\r\n        \r\n        function<\/span> testMean(testCase, TestMatrix)\r\n            while<\/span> testCase.keepMeasuring\r\n                matrix_mean(TestMatrix);\r\n            end<\/span>\r\n        end<\/span>\r\n        \r\n        function<\/span> testEig(testCase, TestMatrix)\r\n            \r\n            testCase.assertReturnsTrue(@() size(TestMatrix,1) == size(TestMatrix,2), ...<\/span>\r\n                'Eig only works on square matrix'<\/span>);\r\n            while<\/span> testCase.keepMeasuring\r\n                matrix_eig(TestMatrix);\r\n            end<\/span>\r\n            \r\n        end<\/span>\r\n    end<\/span>\r\nend<\/span>\r\n\r\n<\/pre>
    results = runperf('tMatrixLibrary_final'<\/span>);\r\ncheckResults(results)\r\n<\/pre>
    Running tMatrixLibrary_final\r\n.......... .......... .......... .......... ..........\r\n.......... .......... ..\r\nDone tMatrixLibrary_final\r\n__________\r\n\r\n\r\n\r\n:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\r\nThanks Dory, you're the best! All our measurements are good.\r\n:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\r\n<\/pre>
    Ah! Isn't that lovely? I didn't need to hard code any static for<\/tt><\/b> loop values, we were able to accurately measure significantly faster code, and the measured values returned actually are the real times taken rather than the values offset by an arbitrary scaling factor. Also, the code that needs more iterations is the fast code, so I didn't even notice any difference in test time, whereas the slower tests didn't need any more iterations at all for an accurate measurement. So nice. Look at the sample summary and you can see the real time taken:<\/p>
    sampleSummary(results)\r\n<\/pre>
    \r\nans =\r\n\r\n  9×7 table\r\n\r\n                           Name                            SampleSize       Mean       StandardDeviation       Min          Median         Max    \r\n    ___________________________________________________    __________    __________    _________________    __________    __________    __________\r\n\r\n    tMatrixLibrary_final\/testSum(TestMatrix=smallSize)         4         1.4886e-05        1.242e-07        1.4807e-05    1.4833e-05    1.5071e-05\r\n    tMatrixLibrary_final\/testSum(TestMatrix=midSize)           4         0.00060309       1.4084e-05        0.00058393    0.00060658    0.00061526\r\n    tMatrixLibrary_final\/testSum(TestMatrix=largeSize)         4           0.003275       6.2535e-05         0.0031897     0.0032854     0.0033395\r\n    tMatrixLibrary_final\/testMean(TestMatrix=smallSize)        4         1.5534e-05       3.4959e-07        1.5221e-05    1.5535e-05    1.5847e-05\r\n    tMatrixLibrary_final\/testMean(TestMatrix=midSize)          4         0.00059933       9.2749e-06        0.00058845    0.00060012    0.00060865\r\n    tMatrixLibrary_final\/testMean(TestMatrix=largeSize)        4          0.0032668       4.8834e-05         0.0031958     0.0032859     0.0032997\r\n    tMatrixLibrary_final\/testEig(TestMatrix=smallSize)         4           0.003086       6.1447e-05           0.00304     0.0030639     0.0031762\r\n    tMatrixLibrary_final\/testEig(TestMatrix=midSize)           4            0.16333         0.001356           0.16232       0.16287       0.16527\r\n    tMatrixLibrary_final\/testEig(TestMatrix=largeSize)         4            0.39709       0.00096197           0.39613       0.39704       0.39813\r\n\r\n<\/pre>
    It is worth noting that this is not a silver bullet. There still is some framework overhead in the keepMeasuring method that prevents us from measuring some really fast code. Think about it like measuring a group of feathers, but if each feather comes individually wrapped in a small packet, there comes a point where we are measuring the overhead of the packet rather than the actual feather. So, while there is still some code that will be too fast to measure (don't expect a valid measurement of 1+1 please), using the keepMeasuring<\/tt><\/b> method as shown opened up 2 orders of magnitude in allowable precision in our experiments.<\/p>
    Have fun, and like Dory, just keep measuring y'all!<\/p>