{"id":11077,"date":"2024-03-15T14:05:58","date_gmt":"2024-03-15T18:05:58","guid":{"rendered":"https:\/\/blogs.mathworks.com\/cleve\/?p=11077"},"modified":"2024-03-15T14:05:58","modified_gmt":"2024-03-15T18:05:58","slug":"twenty-years-of-parallel-matlab","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/cleve\/2024\/03\/15\/twenty-years-of-parallel-matlab\/","title":{"rendered":"Twenty Years of Parallel MATLAB"},"content":{"rendered":"<div class=\"content\"><!--introduction--><p>I have just returned from the MathWorks company meeting celebrating our 40th Anniversary. In one of the presentations, Jos Martin described how Parallel MATLAB was introduced almost twenty years ago.  Here are a few slides from Jos's talk.<\/p><!--\/introduction--><h3>Contents<\/h3><div><ul><li><a href=\"#eb526631-dc7b-4d6f-b20d-0df539aebec4\">Why There Wasn't Any Parallel MATLAB<\/a><\/li><li><a href=\"#46778056-77d3-4a5c-b5fe-b52d97931018\">Twenty-seven Parallel MATLABs<\/a><\/li><li><a href=\"#3df2759d-d597-4b19-9c05-c9112532cadc\">Distributed Computing Toolbox<\/a><\/li><li><a href=\"#7c5caf72-f3b8-485b-8b12-1c06813855e0\">Supercomputing Conference<\/a><\/li><li><a href=\"#9266c227-5e9f-4e31-9d7d-a8d89fc7b2d1\">Bill Gates<\/a><\/li><li><a href=\"#85862059-8ccf-4c67-b17b-018619edab9f\">Now There is a Parallel MATLAB<\/a><\/li><\/ul><\/div><h4>Why There Wasn't Any Parallel MATLAB<a name=\"eb526631-dc7b-4d6f-b20d-0df539aebec4\"><\/a><\/h4><p>In <i>MATLAB News and Notes<\/i> for spring 1995, I wrote a one-page Cleve's Corner titled \"Why there isn't any parallel MATLAB.\" There were three reasons.<\/p><div><ul><li>Memory model.  MATLAB would generate a matrix on a host machine, split it into roughly equally sized submatrices, and distribute each submatrix to a node.  But it took far longer to distribute the data then it did to do the computation.  Any matrix that would fit into memory on the host was too small to make  effective use of the parallel computer itself.<\/li><\/ul><\/div><div><ul><li>Granularity.  The amount of work involved in a single matrix computation is too little to be effectively parallelized.<\/li><\/ul><\/div><div><ul><li>Business situation.  There are too few potential customers at this time (1995) to undertake fundamental changes in MATLAB's architecture.<\/li><\/ul><\/div><p>This one-page note turned out to be one of my most widely cited publications.<\/p><p>.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/Picture3.png\" alt=\"\"> <\/p><p>.<\/p><h4>Twenty-seven Parallel MATLABs<a name=\"46778056-77d3-4a5c-b5fe-b52d97931018\"><\/a><\/h4><p><a href=\"https:\/\/citeseerx.ist.psu.edu\/document?repid=rep1&amp;type=pdf&amp;doi=0b90279927d97c5e4e2ae48a07f5a44f1abe7bac\">A 2001 survey by Ron Choy<\/a> at MIT found 27 different projects that were developing some way to run MATLAB in parallel. All of them involved a MATLAB-based host program calling a fixed library of parallel functions, written in some other language, on the workers. None of the systems were capable of running arbitrary MATLAB programs in parallel.  None of them were MathWorks products.<\/p><h4>Distributed Computing Toolbox<a name=\"3df2759d-d597-4b19-9c05-c9112532cadc\"><\/a><\/h4><p><a href=\"https:\/\/www.hpcwire.com\/2004\/11\/09\/mathworks-unveils-computing-toolbox-for-matlab-and-simulink\/\">MathWorks introduced<\/a> the MATLAB Distributed Computing Toolbox in November 2004. We improvised this demo setup at our first Supercomputing Conference, SC2004 in Pittsburg,<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/SC2004.png\" alt=\"\"> <\/p><p>.<\/p><h4>Supercomputing Conference<a name=\"7c5caf72-f3b8-485b-8b12-1c06813855e0\"><\/a><\/h4><p>A year later, SC2005 was in Seattle and our booth featured four worker machines on a wire shelving unit purchased at a local Home Depot.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/SC2005.png\" alt=\"\"> <\/p><p>.<\/p><h4>Bill Gates<a name=\"9266c227-5e9f-4e31-9d7d-a8d89fc7b2d1\"><\/a><\/h4><p>Since Seattle was his home town, Bill Gates gave the keynote talk at SC2005.  He announced that Microsoft was going into High Performance Computing and used the MathWorks Distributed Computing Toolbox in his demonstration.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/Gates.png\" alt=\"\"> <\/p><p>.<\/p><h4>Now There is a Parallel MATLAB<a name=\"85862059-8ccf-4c67-b17b-018619edab9f\"><\/a><\/h4><p>So, a little more than ten years after the first Cleve's Corner about parallel computing, a second Cleve's Corner in <i>News and Notes<\/i> was able to reverse the situation.<\/p><p>.<\/p><p><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/Picture2.png\" alt=\"\"> <\/p><script language=\"JavaScript\"> <!-- \r\n    function grabCode_14a73fc939e74d6a9266b6fa2deda9a3() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='14a73fc939e74d6a9266b6fa2deda9a3 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 14a73fc939e74d6a9266b6fa2deda9a3';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        copyright = 'Copyright 2024 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add copyright line at the bottom if specified.\r\n        if (copyright.length > 0) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n\r\n        d.title = title + ' (MATLAB code)';\r\n        d.close();\r\n    }   \r\n     --> <\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_14a73fc939e74d6a9266b6fa2deda9a3()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n      the MATLAB code <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; R2023a<br><\/p><\/div><!--\r\n14a73fc939e74d6a9266b6fa2deda9a3 ##### SOURCE BEGIN #####\r\n%% Twenty Years of Parallel MATLAB\r\n% I have just returned from the MathWorks company meeting\r\n% celebrating our 40th Anniversary. In one of the presentations,\r\n% Jos Martin described how Parallel MATLAB was introduced \r\n% almost twenty years ago.  Here are a few slides\r\n% from Jos's talk.\r\n\r\n%% Why There Wasn't Any Parallel MATLAB\r\n% In _MATLAB News and Notes_ for spring 1995, I wrote a one-page\r\n% Cleve's Corner titled \"Why there isn't any parallel MATLAB.\"\r\n% There were three reasons.\r\n%\r\n% * Memory model.  MATLAB would generate a matrix on a host machine,\r\n% split it into roughly equally sized submatrices, and distribute\r\n% each submatrix to a node.  But it took far longer to distribute\r\n% the data then it did to do the computation.  Any matrix that would\r\n% fit into memory on the host was too small to make  effective use\r\n% of the parallel computer itself.\r\n%\r\n% * Granularity.  The amount of work involved in a single matrix \r\n% computation is too little to be effectively parallelized. \r\n%\r\n% * Business situation.  There are too few potential customers\r\n% at this time (1995) to undertake fundamental changes in \r\n% MATLAB's architecture.\r\n% \r\n% This one-page note turned out to be one of my most widely cited\r\n% publications.\r\n%\r\n% .\r\n% \r\n% <<Picture3.png>>\r\n%\r\n% .\r\n%\r\n\r\n%% Twenty-seven Parallel MATLABs\r\n% <https:\/\/citeseerx.ist.psu.edu\/document?repid=rep1&type=pdf&doi=0b90279927d97c5e4e2ae48a07f5a44f1abe7bac\r\n% A 2001 survey by Ron Choy> at MIT found 27 different projects\r\n% that were developing some way to run MATLAB in parallel.  \r\n% All of them involved a MATLAB-based host program calling a fixed \r\n% library of parallel functions, written in some other language,\r\n% on the workers.\r\n% None of the systems were capable of running arbitrary MATLAB programs\r\n% in parallel.  None of them were MathWorks products.\r\n%\r\n\r\n%%% Distributed Computing Toolbox\r\n% <https:\/\/www.hpcwire.com\/2004\/11\/09\/mathworks-unveils-computing-toolbox-for-matlab-and-simulink\/\r\n% MathWorks introduced>\r\n% the MATLAB Distributed Computing Toolbox in November 2004.\r\n% We improvised this demo setup at our first Supercomputing\r\n% Conference, SC2004 in Pittsburg,\r\n%\r\n% <<SC2004.png>>\r\n%\r\n% .\r\n%\r\n\r\n\r\n%% Supercomputing Conference\r\n% A year later, SC2005 was in Seattle and our booth featured four worker\r\n% machines on a wire shelving unit purchased at a local Home Depot.\r\n%\r\n% <<SC2005.png>>\r\n%\r\n% .\r\n%\r\n\r\n%% Bill Gates\r\n% Since Seattle was his home town, Bill Gates gave the keynote talk at\r\n% SC2005.  He announced that Microsoft was going into High Performance\r\n% Computing and used the MathWorks Distributed Computing Toolbox in\r\n% his demonstration.\r\n%\r\n% <<Gates.png>>\r\n%\r\n% .\r\n%\r\n\r\n\r\n%% Now There is a Parallel MATLAB\r\n% So, a little more than ten years after the first Cleve's Corner\r\n% about parallel computing, a second Cleve's Corner in _News and Notes_\r\n% was able to reverse the situation. \r\n%\r\n% .\r\n%\r\n% <<Picture2.png>>\r\n%\r\n\r\n\r\n##### SOURCE END ##### 14a73fc939e74d6a9266b6fa2deda9a3\r\n-->","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img src=\"https:\/\/blogs.mathworks.com\/cleve\/files\/SC2004.png\" class=\"img-responsive attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"\" decoding=\"async\" loading=\"lazy\" \/><\/div><!--introduction--><p>I have just returned from the MathWorks company meeting celebrating our 40th Anniversary. In one of the presentations, Jos Martin described how Parallel MATLAB was introduced almost twenty years ago.  Here are a few slides from Jos's talk.... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/cleve\/2024\/03\/15\/twenty-years-of-parallel-matlab\/\">read more >><\/a><\/p>","protected":false},"author":78,"featured_media":11065,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[4,19],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/11077"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/users\/78"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/comments?post=11077"}],"version-history":[{"count":2,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/11077\/revisions"}],"predecessor-version":[{"id":11083,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/posts\/11077\/revisions\/11083"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/media\/11065"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/media?parent=11077"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/categories?post=11077"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/cleve\/wp-json\/wp\/v2\/tags?post=11077"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}