{"id":397,"date":"2012-04-20T04:22:43","date_gmt":"2012-04-20T09:22:43","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=397"},"modified":"2012-04-12T04:43:35","modified_gmt":"2012-04-12T09:43:35","slug":"running-scripts-on-a-cluster-using-the-batch-command-in-parallel-computing-toolbox","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2012\/04\/20\/running-scripts-on-a-cluster-using-the-batch-command-in-parallel-computing-toolbox\/","title":{"rendered":"Running Scripts on a Cluster Using the Batch Command in Parallel Computing Toolbox"},"content":{"rendered":"<div xmlns:mwsh=\"https:\/\/www.mathworks.com\/namespace\/mcode\/v1\/syntaxhighlight.dtd\" class=\"content\">\r\n   <introduction>\r\n      <p>I'd like to introduce this week's guest blogger <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/answers\/contributors\/1287414-edric-ellis\">Edric Ellis<\/a>. Edric works for the Parallel Computing development team here at The MathWorks. In this post he will talk about using the\r\n         batch command in Parallel Computing Toolbox.\r\n      <\/p>\r\n      <p>Sometimes, even when you've optimized your MATLAB&reg; code, you find that you have so many simulations to run, or scenarios to\r\n         explore, that running them on your desktop computer just takes too long. <a href=\"https:\/\/www.mathworks.com\/products\/parallel-computing\/\">Parallel Computing Toolbox&#8482;<\/a> together with <a href=\"https:\/\/www.mathworks.com\/products\/distriben\/\">MATLAB Distributed Computing Server&#8482;<\/a> can help you to run your scripts and functions on a cluster of computers. As well as running on a remote cluster, Parallel\r\n         Computing Toolbox lets you use the cores in your desktop machine as a simple local cluster. This article is going to show\r\n         how to use this local cluster - but once you've got things working on the \"local\" cluster, it is simple to scale up to running\r\n         your scripts on a remote cluster.\r\n      <\/p>\r\n      <p>The problem we're going to look at is a Monte-Carlo simulation of a financial instrument - the expected returns from playing\r\n         hands of <a href=\"http:\/\/en.wikipedia.org\/wiki\/Blackjack\">blackjack<\/a> in a casino. We'll approach the problem in three phases:\r\n      <\/p>\r\n      <li>Running the Blackjack Simulation in MATLAB<\/li>\r\n      <li>Submitting Batch Jobs to Run Blackjack on the \"local\" Cluster<\/li>\r\n      <li>Adding <tt>parfor<\/tt> Loops to the Script\r\n      <\/li>\r\n   <\/introduction>\r\n   <h3>Contents<\/h3>\r\n   <div>\r\n      <ul>\r\n         <li><a href=\"#1\">Running the Blackjack Simulation in MATLAB<\/a><\/li>\r\n         <li><a href=\"#2\">Running Blackjack as a Script Directly in MATLAB<\/a><\/li>\r\n         <li><a href=\"#3\">Submitting Batch Jobs to Run Blackjack on the \"local\" Cluster<\/a><\/li>\r\n         <li><a href=\"#4\">Adding parfor Loops to the Script<\/a><\/li>\r\n         <li><a href=\"#5\">Running the parfor script with an open MATLAB pool<\/a><\/li>\r\n         <li><a href=\"#6\">Running a Batch Job Containing a parfor Loop<\/a><\/li>\r\n         <li><a href=\"#7\">Conclusions<\/a><\/li>\r\n      <\/ul>\r\n   <\/div>\r\n   <h3>Running the Blackjack Simulation in MATLAB<a name=\"1\"><\/a><\/h3>\r\n   <p>The calculation of the result of playing a single hand of blackjack was written by <a href=\"https:\/\/www.mathworks.com\/company\/aboutus\/founders\/clevemoler.html\">Cleve Moler<\/a>, and is available on the <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/4404-blackjack\">file exchange<\/a>. During this article, we're going to use a slightly modified version of that program where we've extracted just the calculation\r\n      of a single hand of blackjack so that we can invoke it in a variety of different ways. There's a discussion of the algorithm\r\n      in chapter 9 of <a href=\"https:\/\/www.mathworks.com\/moler\/index_ncm.html\">Cleve's book \"Numerical Computing with MATLAB\"<\/a>.\r\n   <\/p>\r\n   <p>Cleve describes the amount won playing a single hand of blackjack as follows:<\/p>\r\n   <p><i>\"[win] zero for a 'push', win $15 for a blackjack, win or lose $10 on a hand that has not been split or doubled, win or lose\r\n         $20 on hands that have been split or doubled once, and win or lose $30 or $40 on hands that have been doubled after a split.\r\n         The $30 and $40 payoffs occur rarely (and may not be allowed at some casinos).\"<\/i><\/p>\r\n   <p>We can calculate the winnings of playing several individual hands of blackjack simply by invoking <tt>blackjack_kernel<\/tt> a few times.\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">winnings = zeros(1,10);\r\n<span style=\"color: #0000FF\">for<\/span> ii=1:10\r\n    winnings(ii) = blackjack_kernel;\r\n<span style=\"color: #0000FF\">end<\/span>\r\nwinnings<\/pre><pre style=\"font-style:oblique\">\r\nwinnings =\r\n\r\n    15   -10     0     0   -10    10   -10   -10   -10    10\r\n\r\n<\/pre><h3>Running Blackjack as a Script Directly in MATLAB<a name=\"2\"><\/a><\/h3>\r\n   <p>The script <tt>blackjack_script<\/tt> simulates a number of players playing many hands of blackjack, and computes for each player their net profit after those\r\n      hands. Let's run the script and then plot the results as a histogram showing the distribution of profit and loss among the\r\n      players.\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">type <span style=\"color: #A020F0\">blackjack_script<\/span>\r\n\r\nblackjack_script;\r\nhist(results);\r\ntitle(<span style=\"color: #A020F0\">'Profit and loss playing blackjack'<\/span>);\r\nxlabel(<span style=\"color: #A020F0\">'Profit'<\/span>); ylabel(<span style=\"color: #A020F0\">'Count'<\/span>);<\/pre><pre style=\"font-style:oblique\">\r\nnumHands   = 500;\r\nnumPlayers = 50;\r\nresults    = zeros(1, numPlayers);\r\ntic\r\nfor h = 1:numHands\r\n    thisHand = zeros(1, numPlayers);\r\n    for p = 1:numPlayers\r\n        thisHand(p) = blackjack_kernel;\r\n    end\r\n    results = results + thisHand;\r\nend\r\nt = toc;\r\nfprintf('Time to simulate %d players playing %d hands of blackjack: %.3g seconds\\n', ...\r\n        numPlayers, numHands, t);\r\n\r\nTime to simulate 50 players playing 500 hands of blackjack: 8.04 seconds\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/397\/batchblog_01.png\"> <h3>Submitting Batch Jobs to Run Blackjack on the \"local\" Cluster<a name=\"3\"><\/a><\/h3>\r\n   <p>Now that our script works correctly, we can submit the script for execution on a cluster. In this case, we're going to use\r\n      the \"local\" cluster since it's always available, but you might submit the script for execution on a remote cluster if you\r\n      have one available. When you do submit to a remote cluster, you're free to close down your MATLAB session and collect the\r\n      results later. If you're using the local cluster, you can carry on using your MATLAB session while the batch job runs in the\r\n      background; also, you can have several batch jobs running simultaneously.\r\n   <\/p>\r\n   <p>We invoke the <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/batch.html\"><tt>batch<\/tt><\/a> command with the name of the script we wish to run, and here we're also specifying which <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/f5-16141.html\"><tt>'Profile'<\/tt><\/a> we wish to use. Profiles identify connections you have made to different clusters that might be available, the 'local' profile\r\n      is always available. Other cluster types require an MDCS installation. If you only have a single Profile, you don't need to\r\n      specify the Profile argument. The return from the <tt>batch<\/tt> command is a job object. The job object allows us to track the progress of execution on the cluster, and when it has finished,\r\n      we can access the results.\r\n   <\/p>\r\n   <p>As well as using the <tt>batch<\/tt> command directly from the MATLAB command line, you can right-click on a script file in MATLAB's Current Folder browser, and\r\n      select <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/rn\/bs15q67.html#bs3_a0p-1\">\"Run Script as Batch Job\"<\/a>. This uses your default cluster profile.\r\n   <\/p>\r\n   <p>In this example, the first thing that we do with the job is call <tt>wait<\/tt> so that we only proceed when the job has finished executing. Alternatively, you can carry on working in MATLAB and check\r\n      the state of the job by looking at <tt>job.State<\/tt>, loading the results when the state is <tt>'finished'<\/tt>.\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">job = batch(<span style=\"color: #A020F0\">'blackjack_script'<\/span>, <span style=\"color: #A020F0\">'Profile'<\/span>, <span style=\"color: #A020F0\">'local'<\/span>);\r\n\r\n<span style=\"color: #228B22\">% Wait for the job to complete execution.<\/span>\r\nwait(job);\r\n\r\n<span style=\"color: #228B22\">% Load the results of executing the job into the MATLAB workspace<\/span>\r\nload(job);\r\n\r\n<span style=\"color: #228B22\">% And display the diary output from the job<\/span>\r\ndisp(<span style=\"color: #A020F0\">'Diary output from the batch job:'<\/span>)\r\ndiary(job);\r\n\r\n<span style=\"color: #228B22\">% We can plot the results as we did previously.<\/span>\r\nhist(results);\r\ntitle(<span style=\"color: #A020F0\">'Profit and loss playing blackjack'<\/span>);\r\nxlabel(<span style=\"color: #A020F0\">'Profit'<\/span>); ylabel(<span style=\"color: #A020F0\">'Count'<\/span>);<\/pre><pre style=\"font-style:oblique\">Diary output from the batch job:\r\nTime to simulate 50 players playing 500 hands of blackjack: 7.93 seconds\r\n\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/397\/batchblog_02.png\"> <h3>Adding parfor Loops to the Script<a name=\"4\"><\/a><\/h3>\r\n   <p>Running a batch job on the local machine doesn't make things go any quicker, although running a job on a remote cluster might\r\n      give you access to a more powerful machine. Even on the local machine, we can still use <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/f5-16141.html\"><tt>parfor<\/tt><\/a> to gain a speedup. Each iteration of the outer <tt>for<\/tt> loop in <tt>blackjack_script<\/tt> is independent, and the calculation of <tt>results<\/tt> is a \"reduction\" or summary operation that <tt>parfor<\/tt> can understand. So, we can recast the script into a variant using <tt>parfor<\/tt> simply by changing the outer loop:\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">type <span style=\"color: #A020F0\">blackjack_script_parfor<\/span><\/pre><pre style=\"font-style:oblique\">\r\nnumHands   = 500;\r\nnumPlayers = 50;\r\nresults    = zeros(1, numPlayers);\r\ntic\r\nparfor h = 1:numHands\r\n    thisHand = zeros(1, numPlayers);\r\n    for p = 1:numPlayers\r\n        thisHand(p) = blackjack_kernel;\r\n    end\r\n    results = results + thisHand;\r\nend\r\nt = toc;\r\nfprintf('Time to simulate %d players playing %d hands of blackjack: %.3g seconds\\n', ...\r\n        numPlayers, numHands, t);\r\n\r\n\r\n<\/pre><h3>Running the parfor script with an open MATLAB pool<a name=\"5\"><\/a><\/h3>\r\n   <p>We can check that the <tt>parfor<\/tt> loop is working correctly by opening an interactive <tt>matlabpool<\/tt> session. An interactive <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/matlabpool.html\">matlabpool<\/a> session launches workers based on the Profile you specify, and makes them available to your MATLAB session for running the\r\n      body of <tt>parfor<\/tt> loops. Normally, it's a good idea to leave <tt>matlabpool<\/tt> open while you're working, but we're going to close it just after running the script to free up workers so that we can run\r\n      a batch job.\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\"><span style=\"color: #228B22\">% Open matlabpool using the 'local' profile<\/span>\r\nmatlabpool <span style=\"color: #A020F0\">open<\/span> <span style=\"color: #A020F0\">local<\/span>\r\n\r\n<span style=\"color: #228B22\">% Run our script<\/span>\r\nblackjack_script_parfor\r\n\r\n<span style=\"color: #228B22\">% Close the pool again to free up workers for when we run the batch job<\/span>\r\nmatlabpool <span style=\"color: #A020F0\">close<\/span><\/pre><pre style=\"font-style:oblique\">Starting matlabpool using the 'local' profile ... connected to 6 labs.\r\nTime to simulate 50 players playing 500 hands of blackjack: 1.87 seconds\r\nSending a stop signal to all the labs ... stopped.\r\n<\/pre><h3>Running a Batch Job Containing a parfor Loop<a name=\"6\"><\/a><\/h3>\r\n   <p>To run a batch job where a <tt>parfor<\/tt> loop is used, we need to specify an additional argument to the <tt>batch<\/tt> command - <tt>'Matlabpool'<\/tt>. The value of the option specifies how many <b>additional<\/b> workers to use when running the script. The machine I'm using has 6 cores, so I can run 1 worker to control the script execution,\r\n      and 5 additional workers to act as a <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/matlabpool.html\"><tt>matlabpool<\/tt><\/a>. These additional workers operate on the body of the <tt>parfor<\/tt> loop in parallel to calculate the result more quickly.\r\n   <\/p><pre style=\"background: #F9F7F3; padding: 10px; border: 1px solid rgb(200,200,200)\">job = batch(<span style=\"color: #A020F0\">'blackjack_script_parfor'<\/span>, <span style=\"color: #0000FF\">...<\/span><span style=\"color: #228B22\"> % Run the script containing parfor<\/span>\r\n            <span style=\"color: #A020F0\">'Profile'<\/span>, <span style=\"color: #A020F0\">'local'<\/span>, <span style=\"color: #0000FF\">...<\/span><span style=\"color: #228B22\">        % using the 'local' profile<\/span>\r\n            <span style=\"color: #A020F0\">'Matlabpool'<\/span>, 5);              <span style=\"color: #228B22\">% with a Matlabpool of size 5<\/span>\r\n\r\n<span style=\"color: #228B22\">% Wait for the job to complete execution.<\/span>\r\nwait(job);\r\n\r\n<span style=\"color: #228B22\">% Load the results of executing the job into the MATLAB workspace<\/span>\r\nload(job);\r\n\r\n<span style=\"color: #228B22\">% And display the diary output from the job<\/span>\r\ndisp(<span style=\"color: #A020F0\">'Diary output from the batch job using parfor:'<\/span>)\r\ndiary(job);\r\n\r\n<span style=\"color: #228B22\">% We can plot the results as we did previously.<\/span>\r\nhist(results);\r\ntitle(<span style=\"color: #A020F0\">'Profit and loss playing blackjack'<\/span>);\r\nxlabel(<span style=\"color: #A020F0\">'Profit'<\/span>); ylabel(<span style=\"color: #A020F0\">'Count'<\/span>);<\/pre><pre style=\"font-style:oblique\">Diary output from the batch job using parfor:\r\nTime to simulate 50 players playing 500 hands of blackjack: 1.92 seconds\r\n\r\n<\/pre><img decoding=\"async\" vspace=\"5\" hspace=\"5\" src=\"https:\/\/blogs.mathworks.com\/images\/loren\/397\/batchblog_03.png\"> <h3>Conclusions<a name=\"7\"><\/a><\/h3>\r\n   <p>We have seen how using the <tt>batch<\/tt> command, we can send a script to be executed on a cluster. We've used the 'local' cluster here, which is a part of Parallel\r\n      Computing Toolbox, but we could have just as easily used a remote cluster if a MATLAB Distributed Computing Server installation\r\n      was available to us. Using a remote cluster allows even more compute resources to be used during the execution of a job. Some\r\n      <tt>for<\/tt> loops (where each iteration can be executed independently of the others) lend themselves to parallel execution using <tt>parfor<\/tt>, and we saw how we can run a <tt>batch<\/tt> job where our script contained a <tt>parfor<\/tt> loop.\r\n   <\/p>\r\n   <p>There are more details about getting your code running on a cluster in the <a href=\"https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/\">documentation<\/a>. Do you have any other questions about running your code on a cluster? Let us know <a href=\"https:\/\/blogs.mathworks.com\/loren\/?p=397#respond\">here<\/a>.\r\n   <\/p><script language=\"JavaScript\">\r\n<!--\r\n\r\n    function grabCode_4a0364ff842e46d0b3edc19a192153c5() {\r\n        \/\/ Remember the title so we can use it in the new page\r\n        title = document.title;\r\n\r\n        \/\/ Break up these strings so that their presence\r\n        \/\/ in the Javascript doesn't mess up the search for\r\n        \/\/ the MATLAB code.\r\n        t1='4a0364ff842e46d0b3edc19a192153c5 ' + '##### ' + 'SOURCE BEGIN' + ' #####';\r\n        t2='##### ' + 'SOURCE END' + ' #####' + ' 4a0364ff842e46d0b3edc19a192153c5';\r\n    \r\n        b=document.getElementsByTagName('body')[0];\r\n        i1=b.innerHTML.indexOf(t1)+t1.length;\r\n        i2=b.innerHTML.indexOf(t2);\r\n \r\n        code_string = b.innerHTML.substring(i1, i2);\r\n        code_string = code_string.replace(\/REPLACE_WITH_DASH_DASH\/g,'--');\r\n\r\n        \/\/ Use \/x3C\/g instead of the less-than character to avoid errors \r\n        \/\/ in the XML parser.\r\n        \/\/ Use '\\x26#60;' instead of '<' so that the XML parser\r\n        \/\/ doesn't go ahead and substitute the less-than character. \r\n        code_string = code_string.replace(\/\\x3C\/g, '\\x26#60;');\r\n\r\n        author = '';\r\n        copyright = 'Copyright 2012 The MathWorks, Inc.';\r\n\r\n        w = window.open();\r\n        d = w.document;\r\n        d.write('<pre>\\n');\r\n        d.write(code_string);\r\n\r\n        \/\/ Add author and copyright lines at the bottom if specified.\r\n        if ((author.length > 0) || (copyright.length > 0)) {\r\n            d.writeln('');\r\n            d.writeln('%%');\r\n            if (author.length > 0) {\r\n                d.writeln('% _' + author + '_');\r\n            }\r\n            if (copyright.length > 0) {\r\n                d.writeln('% _' + copyright + '_');\r\n            }\r\n        }\r\n\r\n        d.write('<\/pre>\\n');\r\n      \r\n      d.title = title + ' (MATLAB code)';\r\n      d.close();\r\n      }   \r\n      \r\n-->\r\n<\/script><p style=\"text-align: right; font-size: xx-small; font-weight:lighter;   font-style: italic; color: gray\"><br><a href=\"javascript:grabCode_4a0364ff842e46d0b3edc19a192153c5()\"><span style=\"font-size: x-small;        font-style: italic;\">Get \r\n            the MATLAB code \r\n            <noscript>(requires JavaScript)<\/noscript><\/span><\/a><br><br>\r\n      Published with MATLAB&reg; 7.14<br><\/p>\r\n<\/div>\r\n<!--\r\n4a0364ff842e46d0b3edc19a192153c5 ##### SOURCE BEGIN #####\r\n%% Running Scripts on a Cluster Using the Batch Command in Parallel Computing Toolbox\r\n% I'd like to introduce this week's guest blogger\r\n% <https:\/\/www.mathworks.com\/matlabcentral\/answers\/contributors\/1287414-edric-ellis Edric Ellis>. \r\n% Edric works for the Parallel Computing development team here at\r\n% The MathWorks. In this post he will talk about using the batch command in\r\n% Parallel Computing Toolbox.\r\n%\r\n% Sometimes, even when you've optimized your MATLAB(R) code, you find that you have\r\n% so many simulations to run, or scenarios to explore, that running them on your\r\n% desktop computer just takes too long. <https:\/\/www.mathworks.com\/products\/parallel-computing\/ Parallel Computing Toolbox(TM)> together with\r\n% <https:\/\/www.mathworks.com\/products\/distriben\/ MATLAB Distributed Computing Server(TM)> can help you to run your\r\n% scripts and functions on a cluster of computers. As well as running on a\r\n% remote cluster, Parallel Computing Toolbox lets you use the cores in your\r\n% desktop machine as a simple local cluster. This article is going to show how to\r\n% use this local cluster - but once you've got things working on the \"local\"\r\n% cluster, it is simple to scale up to running your scripts on a remote\r\n% cluster.\r\n%\r\n% The problem we're going to look at is a Monte-Carlo simulation of a financial\r\n% instrument - the expected returns from playing hands of <http:\/\/en.wikipedia.org\/wiki\/Blackjack blackjack> in a\r\n% casino. We'll approach the problem in three phases:\r\n%\r\n% # Running the Blackjack Simulation in MATLAB\r\n% # Submitting Batch Jobs to Run Blackjack on the \"local\" Cluster\r\n% # Adding |parfor| Loops to the Script\r\n%\r\n%% Running the Blackjack Simulation in MATLAB\r\n% The calculation of the result of playing a single hand of blackjack was\r\n% written by <https:\/\/www.mathworks.com\/company\/aboutus\/founders\/clevemoler.html Cleve Moler>, and is available on the <https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/4404-blackjack file exchange>.\r\n% During this article, we're going to use a slightly modified version\r\n% of that program where we've extracted just the calculation of a single hand\r\n% of blackjack so that we can invoke it in a variety of different ways. There's\r\n% a discussion of the algorithm in chapter 9 of <https:\/\/www.mathworks.com\/moler\/index_ncm.html Cleve's book \"Numerical Computing with MATLAB\">.\r\n%\r\n% Cleve describes the amount won playing a single hand of blackjack as follows:\r\n%\r\n% _\"[win] zero for a 'push', win $15 for a blackjack, win or lose $10 on a hand\r\n% that has not been split or doubled, win or lose $20 on hands that have been\r\n% split or doubled once, and win or lose $30 or $40 on hands that have been\r\n% doubled after a split. The $30 and $40 payoffs occur rarely (and may not be\r\n% allowed at some casinos).\"_\r\n%\r\n% We can calculate the winnings of playing several individual hands of blackjack\r\n% simply by invoking |blackjack_kernel| a few times.\r\n\r\nwinnings = zeros(1,10);\r\nfor ii=1:10\r\n    winnings(ii) = blackjack_kernel;\r\nend\r\nwinnings\r\n\r\n%% Running Blackjack as a Script Directly in MATLAB\r\n% The script |blackjack_script| simulates a number of players playing many hands\r\n% of blackjack, and computes for each player their net profit after those hands.\r\n% Let's run the script and then plot the results as a histogram showing the\r\n% distribution of profit and loss among the players.\r\n\r\ntype blackjack_script\r\n\r\nblackjack_script;\r\nhist(results);\r\ntitle('Profit and loss playing blackjack');\r\nxlabel('Profit'); ylabel('Count');\r\n\r\n%% Submitting Batch Jobs to Run Blackjack on the \"local\" Cluster\r\n% Now that our script works correctly, we can submit the script for execution on\r\n% a cluster. In this case, we're going to use the \"local\" cluster since it's\r\n% always available, but you might submit the script for execution on a remote\r\n% cluster if you have one available. When you do submit to a remote cluster,\r\n% you're free to close down your MATLAB session and collect the results\r\n% later. If you're using the local cluster, you can carry on using your MATLAB\r\n% session while the batch job runs in the background; also, you can have several\r\n% batch jobs running simultaneously.\r\n%\r\n% We invoke the <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/batch.html |batch|> command with the name of the script we wish to run, and\r\n% here we're also specifying which <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/f5-16141.html |'Profile'|> we wish to use. Profiles identify\r\n% connections you have made to different clusters that might be available, the\r\n% 'local' profile is always available. Other cluster types require an MDCS installation. \r\n% If you only have a single Profile, you\r\n% don't need to specify the Profile argument. The return from the |batch|\r\n% command is a job object. The job object allows us to track the progress of\r\n% execution on the cluster, and when it has finished, we can access the results.\r\n%\r\n% As well as using the |batch| command directly from the MATLAB command line,\r\n% you can right-click on a script file in MATLAB's Current Folder browser, and select\r\n% <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/rn\/bs15q67.html#bs3_a0p-1 \"Run Script as Batch Job\">. This uses your default cluster profile.\r\n%\r\n% In this example, the first thing that we do with the job is call |wait| so\r\n% that we only proceed when the job has finished executing. Alternatively, you\r\n% can carry on working in MATLAB and check the state of the job by looking at\r\n% |job.State|, loading the results when the state is |'finished'|.\r\n\r\njob = batch('blackjack_script', 'Profile', 'local');\r\n\r\n% Wait for the job to complete execution.\r\nwait(job);\r\n\r\n% Load the results of executing the job into the MATLAB workspace\r\nload(job);\r\n\r\n% And display the diary output from the job\r\ndisp('Diary output from the batch job:')\r\ndiary(job);\r\n\r\n% We can plot the results as we did previously.\r\nhist(results);\r\ntitle('Profit and loss playing blackjack');\r\nxlabel('Profit'); ylabel('Count');\r\n\r\n\r\n%% Adding |parfor| Loops to the Script\r\n% Running a batch job on the local machine doesn't make things go any quicker,\r\n% although running a job on a remote cluster might give you access to a more\r\n% powerful machine. Even on the local machine, we can still use <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/f5-16141.html |parfor|> to gain\r\n% a speedup. Each iteration of the outer |for| loop in |blackjack_script| is\r\n% independent, and the calculation of |results| is a \"reduction\" or summary\r\n% operation that |parfor| can understand. So, we can recast the script into a\r\n% variant using |parfor| simply by changing the outer loop:\r\n\r\ntype blackjack_script_parfor\r\n\r\n%% Running the |parfor| script with an open MATLAB pool\r\n% We can check that the |parfor| loop is working correctly by opening an\r\n% interactive |matlabpool| session. An interactive <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/matlabpool.html matlabpool> session launches\r\n% workers based on the Profile you specify, and makes them available to your\r\n% MATLAB session for running the body of |parfor| loops. Normally, it's a good\r\n% idea to leave |matlabpool| open while you're working, but we're going to close\r\n% it just after running the script to free up workers so that we can run a batch\r\n% job.\r\n\r\n% Open matlabpool using the 'local' profile\r\nmatlabpool open local\r\n\r\n% Run our script\r\nblackjack_script_parfor\r\n\r\n% Close the pool again to free up workers for when we run the batch job\r\nmatlabpool close\r\n\r\n%% Running a Batch Job Containing a parfor Loop\r\n% To run a batch job where a |parfor| loop is used, we need to specify an\r\n% additional argument to the |batch| command - |'Matlabpool'|. The value of the\r\n% option specifies how many *additional* workers to use when running the\r\n% script. The machine I'm using has 6 cores, so I can run 1 worker to control the script\r\n% execution, and 5 additional workers to act as a <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/matlabpool.html |matlabpool|>. These additional\r\n% workers operate on the body of the |parfor| loop in parallel to calculate the\r\n% result more quickly.\r\n\r\njob = batch('blackjack_script_parfor', ... % Run the script containing parfor\r\n            'Profile', 'local', ...        % using the 'local' profile\r\n            'Matlabpool', 5);              % with a Matlabpool of size 5\r\n\r\n% Wait for the job to complete execution.\r\nwait(job);\r\n\r\n% Load the results of executing the job into the MATLAB workspace\r\nload(job);\r\n\r\n% And display the diary output from the job\r\ndisp('Diary output from the batch job using parfor:')\r\ndiary(job);\r\n\r\n% We can plot the results as we did previously.\r\nhist(results);\r\ntitle('Profit and loss playing blackjack');\r\nxlabel('Profit'); ylabel('Count');\r\n\r\n%% Conclusions\r\n% We have seen how using the |batch| command, we can send a script to be\r\n% executed on a cluster. We've used the 'local' cluster here, which is a part of\r\n% Parallel Computing Toolbox, but we could have just as easily used a remote\r\n% cluster if a MATLAB Distributed Computing Server installation was available to\r\n% us. Using a remote cluster allows even more compute resources to be used\r\n% during the execution of a job. Some |for| loops (where each iteration can be\r\n% executed independently of the others) lend themselves to parallel execution\r\n% using |parfor|, and we saw how we can run a |batch| job where our script\r\n% contained a |parfor| loop.\r\n%\r\n% There are more details about getting your code running on a cluster in the\r\n% <https:\/\/www.mathworks.com\/help\/releases\/R2012a\/toolbox\/distcomp\/ documentation>. Do you have any other questions about running your code on a\r\n% cluster? Let us know <https:\/\/blogs.mathworks.com\/loren\/?p=397#respond here>.\r\n\r\n##### SOURCE END ##### 4a0364ff842e46d0b3edc19a192153c5\r\n-->","protected":false},"excerpt":{"rendered":"<p>\r\n   \r\n      I'd like to introduce this week's guest blogger Edric Ellis. Edric works for the Parallel Computing development team here at The MathWorks. In this post he will talk about using the\r\n   ... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/loren\/2012\/04\/20\/running-scripts-on-a-cluster-using-the-batch-command-in-parallel-computing-toolbox\/\">read more >><\/a><\/p>","protected":false},"author":39,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[34],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/397"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/comments?post=397"}],"version-history":[{"count":11,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/397\/revisions"}],"predecessor-version":[{"id":427,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/posts\/397\/revisions\/427"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/media?parent=397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/categories?post=397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/loren\/wp-json\/wp\/v2\/tags?post=397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}