{"id":45,"date":"2006-07-12T16:01:45","date_gmt":"2006-07-12T21:01:45","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=45"},"modified":"2006-07-17T09:29:45","modified_gmt":"2006-07-17T14:29:45","slug":"what-are-you-really-measuring","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2006\/07\/12\/what-are-you-really-measuring\/","title":{"rendered":"What Are You Really Measuring?"},"content":{"rendered":"<div xmlns:mwsh=\"https:\/\/www.mathworks.com\/namespace\/mcode\/v1\/syntaxhighlight.dtd\" class=\"content\">\r\n   <introduction>\r\n      <p>I had an interesting encounter with a colleague, Bob, last week.  We were talking about timing some calculations and we realized\r\n         that the written code was actually measuring something different.  Here's a small cautionary tale along with a neat factoid\r\n         about vectors and <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/for.html\"><tt>for<\/tt><\/a> loops in MATLAB.\r\n      <\/p>\r\n   <\/introduction>\r\n   <h3>Contents<\/h3>\r\n   <div>\r\n      <ul>\r\n         <li><a href=\"#1\">The Story<\/a><\/li>\r\n         <li><a href=\"#4\">Prep<\/a><\/li>\r\n         <li><a href=\"#5\">[], measure with tic-toc<\/a><\/li>\r\n         <li><a href=\"#6\">(), measure with tic-toc<\/a><\/li>\r\n         <li><a href=\"#7\">[], measure with cputime<\/a><\/li>\r\n         <li><a href=\"#8\">(), measure with cputime<\/a><\/li>\r\n         <li><a href=\"#9\">Explanation<\/a><\/li>\r\n         <li><a href=\"#12\">Remarks<\/a><\/li>\r\n      <\/ul>\r\n   <\/div>\r\n   <h3>The Story<a name=\"1\"><\/a><\/h3>\r\n   <p>The story Bob and I started off with is less interesting than where we ended up. After looking at some code, I mentioned to\r\n      Bob that using square brackets when there was no real concatenation was sometimes expensive in MATLAB.  You'll see this as\r\n      a message from <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/mlint.html\"><tt>mlint<\/tt><\/a>. We pared the code in question down to this:\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">dbtype <span style=\"color: #A020F0\">forsquare<\/span><\/pre><pre style=\"font-style:oblique\">\r\n1     %% forsquare\r\n2     for j = 1:ntimes\r\n3         for k=[1:theEnd] \r\n4             k; \r\n5         end\r\n6     end\r\n<\/pre><p>and set out to compare it with this code:<\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">dbtype <span style=\"color: #A020F0\">forparen<\/span><\/pre><pre style=\"font-style:oblique\">\r\n1     %% forparen\r\n2     for j = 1:ntimes\r\n3         for k=(1:theEnd) \r\n4             k; \r\n5         end\r\n6     end\r\n<\/pre><p>Notice that the only difference between these M-files is the way format of the expression, the first one using <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/matlab_prog\/matlab-operators-and-special-characters.html\">square brackets<\/a> and the second using parentheses to group the expression.  We'll now set out to time the two M-files.\r\n   <\/p>\r\n   <h3>Prep<a name=\"4\"><\/a><\/h3><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">clc, clear\r\nntimes = 10;\r\nntotal = 1e7;\r\ntheEnd = ntotal\/ntimes;<\/pre><h3>[], measure with tic-toc<a name=\"5\"><\/a><\/h3><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">tic, forsquare, toc<\/pre><pre style=\"font-style:oblique\">Elapsed time is 42.060557 seconds.\r\n<\/pre><h3>(), measure with tic-toc<a name=\"6\"><\/a><\/h3><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">tic, forparen, toc<\/pre><pre style=\"font-style:oblique\">Elapsed time is 0.173959 seconds.\r\n<\/pre><h3>[], measure with cputime<a name=\"7\"><\/a><\/h3><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">t = cputime; forsquare, cputime-t<\/pre><pre style=\"font-style:oblique\">ans =\r\n   41.2794\r\n<\/pre><h3>(), measure with cputime<a name=\"8\"><\/a><\/h3><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">t = cputime; forparen, cputime-t<\/pre><pre style=\"font-style:oblique\">ans =\r\n    0.1702\r\n<\/pre><h3>Explanation<a name=\"9\"><\/a><\/h3>\r\n   <p>I am timing these on my home laptop with MATLAB R2006a.  I have 512 MB of RAM and 1.70 GHz processor.  I am performing the\r\n      timings with both <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/tic.html\"><tt>tic<\/tt> <tt>toc<\/tt><\/a> and <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/cputime.html\"><tt>cputime<\/tt><\/a> <tt>tic<\/tt> - <tt>toc<\/tt> is easier, but I am not being too careful about what else is running on my machine so I am also using <tt>cputime<\/tt>.  As you can see, it doesn't make much difference in this case.  The construct with the <tt>[]<\/tt> is noticeably slower.  Why is that?\r\n   <\/p>\r\n   <p>In <tt>forsquare<\/tt>, the <tt>for<\/tt> loop expression delimited by the square brackets <tt>[]<\/tt> first constructs the full array inside the brackets in preparation for concatenating the array.  In this case, there is nothing\r\n      else being tacked on, so we create the full vector, <tt>ntotal<\/tt> elements in length.  However, in the case of <tt>forparen<\/tt>, the parentheses <tt>()<\/tt> are used solely for grouping and MATLAB recognizes the expression as one it can expand during the execution of the <tt>for<\/tt> loop. In <tt>forsquare<\/tt> we create a large vector and that allocation dominates the time of the operation. In case you don't believe me that expressions\r\n      in <tt>for<\/tt> loops are expanded as they are used, when possible, here's some code to look at:\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">dbtype <span style=\"color: #A020F0\">forinf<\/span><\/pre><pre style=\"font-style:oblique\">\r\n1     %% forinf\r\n2     for j = 1:Inf\r\n3        if j &gt; 17\r\n4            break\r\n5        end\r\n6     end\r\n<\/pre><p>Notice that the end of the <tt>for<\/tt> expression is <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/inf.html\"><tt>Inf<\/tt><\/a> ! There's no way I have enough memory to create this vector, and yet the code runs just fine:\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">tic, forinf, toc<\/pre><pre style=\"font-style:oblique\">Elapsed time is 0.000061 seconds.\r\n<\/pre><h3>Remarks<a name=\"12\"><\/a><\/h3>\r\n   <p>Timing is a delicate matter.  Be sure you know what exactly you are timing when you make comparisons.  Do you have any timing\r\n      war stories to <a href=\"?p=45#respond\">share<\/a>?\r\n   <\/p>\r\n   <p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br>\r\n      Published with MATLAB&reg; 7.2<br><\/p>\r\n<\/div>\r\n<!--\r\n##### SOURCE BEGIN #####\r\n%% What Are You Really Measuring?\r\n% I had an interesting encounter with a colleague, Bob, last week.  We were\r\n% talking about timing some calculations and we realized that the written\r\n% code was actually measuring something different.  Here's a small\r\n% cautionary tale along with a neat factoid about vectors and \r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/for.html |for|>\r\n% loops in MATLAB.\r\n%% The Story\r\n% The story Bob and I started off with is less interesting than where we\r\n% ended up. After looking at some code, I mentioned to Bob that using square brackets when there\r\n% was no real concatenation was sometimes expensive in MATLAB.  You'll \r\n% see this as a message from \r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/mlint.html |mlint|>.\r\n% We pared the code in question down to this:\r\ndbtype forsquare\r\n%%\r\n% and set out to compare it with this code:\r\ndbtype forparen\r\n%%\r\n% Notice that the only difference between these M-files is the way format\r\n% of the expression, the first one using \r\n% <https:\/\/www.mathworks.com\/help\/matlab\/matlab_prog\/matlab-operators-and-special-characters.html square brackets>\r\n% and the second using parentheses to group the expression.  We'll now set\r\n% out to time the two M-files.\r\n\r\n%% Prep\r\nclc, clear\r\nntimes = 10;\r\nntotal = 1e7;\r\ntheEnd = ntotal\/ntimes;\r\n%%  [], measure with tic-toc\r\ntic, forsquare, toc\r\n%%  (), measure with tic-toc\r\ntic, forparen, toc\r\n%%  [], measure with cputime\r\nt = cputime; forsquare, cputime-t\r\n%%  (), measure with cputime\r\nt = cputime; forparen, cputime-t\r\n%% Explanation\r\n% I am timing these on my home laptop with MATLAB R2006a.  I have 512 MB of RAM\r\n% and 1.70 GHz processor.  I am performing the timings with both \r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/tic.html |tic| |toc|>\r\n% and\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/cputime.html |cputime|> \r\n% |tic| - |toc| is easier, but I am not being too careful about what else\r\n% is running on my machine so I am also using |cputime|.  As you can see,\r\n% it doesn't make much difference in this case.  The construct with the\r\n% |[]| is noticeably slower.  Why is that?\r\n%%\r\n% In |forsquare|, the |for| loop expression delimited by the square\r\n% brackets |[]| first constructs the full array inside the brackets in\r\n% preparation for concatenating the array.  In this case, there is nothing\r\n% else being tacked on, so we create the full vector, |ntotal|\r\n% elements in length.  However, in the case of |forparen|, the parentheses\r\n% |()| are used solely for grouping and MATLAB recognizes the expression as\r\n% one it can expand during the execution of the |for| loop.  \r\n% In |forsquare| we create a large vector and that allocation dominates the\r\n% time of the operation.\r\n% In case you\r\n% don't believe me that expressions in |for| loops are expanded as they\r\n% are used, when possible, here's some code to look at:\r\ndbtype forinf\r\n%%\r\n% Notice that the end of the |for| expression is\r\n% <https:\/\/www.mathworks.com\/help\/matlab\/ref\/inf.html |Inf|> !\r\n% There's no way I have enough memory to create this vector, and yet the\r\n% code runs just fine:\r\ntic, forinf, toc\r\n\r\n%% Remarks\r\n% Timing is a delicate matter.  Be sure you know what exactly you are\r\n% timing when you make comparisons.  Do you have any timing war stories to\r\n% <http:?p=45\/Respond share>?\r\n\r\n##### SOURCE END #####\r\n-->","protected":false},"excerpt":{"rendered":"<p>\r\n   \r\n      I had an interesting encounter with a colleague, Bob, last week.  We were talking about timing some calculations and we realized\r\n         that the written code was actually measuring... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2006\/07\/12\/what-are-you-really-measuring\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[16,7],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/45"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=45"}],"version-history":[{"count":0,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/45\/revisions"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=45"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=45"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=45"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}