{"id":208,"date":"2009-12-04T18:54:26","date_gmt":"2009-12-04T18:54:26","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/2009\/12\/04\/comparing-single-threaded-vs-multithreaded-floating-point-calculations\/"},"modified":"2018-01-08T14:25:30","modified_gmt":"2018-01-08T19:25:30","slug":"comparing-single-threaded-vs-multithreaded-floating-point-calculations","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2009\/12\/04\/comparing-single-threaded-vs-multithreaded-floating-point-calculations\/","title":{"rendered":"Comparing Single-threaded vs. Multithreaded Floating Point Calculations"},"content":{"rendered":"<div xmlns:mwsh=\"https:\/\/www.mathworks.com\/namespace\/mcode\/v1\/syntaxhighlight.dtd\" class=\"content\">\r\n   <introduction>\r\n      <p>There continue to be a flurry of queries suggesting that MATLAB has bugs when it comes to certain operations like addition\r\n         with more than values. Sometimes what prompts this is a user noticing that he or she gets different answers when using MATLAB\r\n         on multiple cores or processors and comparing those results to running on a single core. Here is one example exchange in the MATLAB newsgroup.\r\n      <\/p>\r\n      <p>In actuality, you can always get different results doing floating point sums in MATLAB, even sticking to a single thread.\r\n          Most languages do not guarantee an order of execution of instructions for a given line of code, and what you get might be\r\n         compiler dependent (this is true for C). Having multiple threads simply compounds the issue even more since the compiler has\r\n         more options for how it will control the statement execution.\r\n      <\/p>\r\n      <p>In MATLAB, as you add more elements to a variable, you may see different results depending on how many threads are being used.\r\n          The reason is numerical roundoff and is not a bug.  Fortunately, it is easy to simulate having more than one core or thread\r\n         for some simple computations by forcing the computational order using parentheses.  Let's try out a small example.  We'll\r\n         compute the following expression several ways and compare the results.\r\n      <\/p><pre>            1e-16 + 1e-16 + 1e30 - 1e30<\/pre><\/introduction>\r\n   <h3>Contents<\/h3>\r\n   <div>\r\n      <ul>\r\n         <li><a href=\"#1\">Single Thread (Sequential) Computation<\/a><\/li>\r\n         <li><a href=\"#2\">Multithreaded Computation (for Two Threads)<\/a><\/li>\r\n         <li><a href=\"#3\">One More Combination<\/a><\/li>\r\n         <li><a href=\"#5\">Correctness of Answers<\/a><\/li>\r\n      <\/ul>\r\n   <\/div>\r\n   <h3>Single Thread (Sequential) Computation<a name=\"1\"><\/a><\/h3>\r\n   <p>One a single, the sum would get evaluate from left to right. Using parentheses, we can force this to happen.<\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">sequentialSum = ((1e-16 + 1e-16) + 1e30) - 1e30<\/pre><pre style=\"font-style:oblique\">sequentialSum =\r\n     0\r\n<\/pre><h3>Multithreaded Computation (for Two Threads)<a name=\"2\"><\/a><\/h3>\r\n   <p>The most logical way a vector would get split for between two threads is the first half on one and the second on the other.\r\n       We simulate that here.\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">twoThreadSum = (1e-16 + 1e-16) + (1e30 - 1e30)<\/pre><pre style=\"font-style:oblique\">twoThreadSum =\r\n                    2e-016\r\n<\/pre><h3>One More Combination<a name=\"3\"><\/a><\/h3>\r\n   <p>Let's try taking every other element instead.  That would essentially be<\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">alternateSum = (1e-16 + 1e30) + (1e-16 - 1e30)<\/pre><pre style=\"font-style:oblique\">alternateSum =\r\n     0\r\n<\/pre><h3>Correctness of Answers<a name=\"5\"><\/a><\/h3>\r\n   <p>All of the answers are correct!  Welcome to the delights of floating point arithmetic.  Have you encountered a situation similar\r\n      to one of these that puzzled you?  How did you figure out that the issue was really a non-issue?  Do you program defensively\r\n      when it matters (e.g., don't check for equality for floating point answers, but test that the values lie within some limited\r\n      range)?  Let me know <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=208#respond\">here<\/a>.\r\n   <\/p><script language=\"JavaScript\">\r\n<!--\r\n\r\n    function grabCode_e3b3f405696a4a02ac14668bfdcc3c7b() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='e3b3f405696a4a02ac14668bfdcc3c7b ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' e3b3f405696a4a02ac14668bfdcc3c7b';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        author = 'Loren Shure';\r\n        copyright = 'Copyright 2009 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add author and copyright lines at the bottom if specified.\r\n        if ((author.length > 0) || (copyright.length > 0)) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (author.length > 0) {\r\n                d.writeln('% _' + author + '_');\r\n            }\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n      \r\n      d.title = title + ' (MATLAB code)';\r\n      d.close();\r\n      }   \r\n      \r\n-->\r\n<\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_e3b3f405696a4a02ac14668bfdcc3c7b()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n            the MATLAB code \r\n            <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; 7.9<br><\/p>\r\n<\/div>\r\n<!--\r\ne3b3f405696a4a02ac14668bfdcc3c7b ##### SOURCE BEGIN #####\r\n%% Comparing Single-threaded vs. Multithreaded Floating Point Calculations\r\n% There continue to be a flurry of queries suggesting that MATLAB has bugs\r\n% when it comes to certain operations like addition with more than values.\r\n% Sometimes what prompts this is a user noticing that he or she gets\r\n% different answers when using MATLAB on multiple cores or processors and\r\n% comparing those results to running on a single core.  \r\n% <http:\/\/view_thread\/261541#686025 Here>\r\n% is one example exchange in the MATLAB newsgroup.\r\n%\r\n% In actuality, you can always get different results doing floating point \r\n% sums in MATLAB, even sticking to a single thread.  Most languages do not\r\n% guarantee an order of execution of instructions for a given line of code,\r\n% and what you get might be compiler dependent (this is true for C).\r\n% Having multiple threads simply compounds the issue even more since the\r\n% compiler has more options for how it will control the statement\r\n% execution.\r\n%\r\n% In MATLAB, as you add more elements to a variable, you may see different \r\n% results depending on how many threads are being used.  The reason is\r\n% numerical roundoff and is not a bug.  Fortunately, it is easy to simulate\r\n% having more than one core or thread for some simple computations by\r\n% forcing the computational order using parentheses.  Let's try out a small\r\n% example.  We'll compute the following expression several ways and compare\r\n% the results.\r\n%\r\n%              1e-16 + 1e-16 + 1e30 - 1e30\r\n%% Single Thread (Sequential) Computation\r\n% One a single, the sum would get evaluate from left to right.\r\n% Using parentheses, we can force this to happen.\r\nsequentialSum = ((1e-16 + 1e-16) + 1e30) - 1e30\r\n%% Multithreaded Computation (for Two Threads)\r\n% The most logical way a vector would get split for between two threads is\r\n% the first half on one and the second on the other.  We simulate that\r\n% here.\r\ntwoThreadSum = (1e-16 + 1e-16) + (1e30 - 1e30)\r\n%% One More Combination\r\n% Let's try taking every other element instead.  That would essentially be\r\nalternateSum = (1e-16 + 1e30) + (1e-16 - 1e30)\r\n%%\r\n%% Correctness of Answers\r\n% All of the answers are correct!  Welcome to the delights of floating\r\n% point arithmetic.  Have you encountered a situation similar to one of\r\n% these that puzzled you?  How did you figure out that the issue was really\r\n% a non-issue?  Do you program defensively when it matters (e.g., don't\r\n% check for equality for floating point answers, but test that the values\r\n% lie within some limited range)?  Let me know\r\n% <https:\/\/blogs.mathworks.com\/loren\/?p=208#respond here>.\r\n##### SOURCE END ##### e3b3f405696a4a02ac14668bfdcc3c7b\r\n-->","protected":false},"excerpt":{"rendered":"<p>\r\n   \r\n      There continue to be a flurry of queries suggesting that MATLAB has bugs when it comes to certain operations like addition\r\n         with more than values. Sometimes what prompts this is... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2009\/12\/04\/comparing-single-threaded-vs-multithreaded-floating-point-calculations\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[14,26],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/208"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=208"}],"version-history":[{"count":1,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/208\/revisions"}],"predecessor-version":[{"id":2528,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/208\/revisions\/2528"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=208"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=208"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}