{"id":1238,"date":"2023-05-19T08:54:43","date_gmt":"2023-05-19T12:54:43","guid":{"rendered":"https:\/\/blogs.mathworks.com\/matlab\/?p=1238"},"modified":"2023-05-19T10:05:27","modified_gmt":"2023-05-19T14:05:27","slug":"direct-submission-to-hpc-clusters-from-matlab","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/matlab\/2023\/05\/19\/direct-submission-to-hpc-clusters-from-matlab\/","title":{"rendered":"Direct submission to HPC clusters from MATLAB"},"content":{"rendered":"
\r\n

More Compute, More Problems<\/h4>\r\n
When it comes to High Performance Computing resources, I'm a lucky guy. I've got a fairly decent, 8 core desktop along with guest access to a reasonable number of academic HPC clusters around the world. Since I try to be a courteous guest, I make use of a healthy cloud budget when I need to do some heavy computations and use all of the major cloud providers.<\/div>\r\n
In the old days, access to such a diverse array of compute was as much of a curse as it was a blessing. Many systems use different schedulers, for example. Even when two systems have the same scheduler (SLURM<\/a> usually), they have different module files, different authentication procedures and different file system layouts among other things. Making use of these computational riches can be a burden indeed.<\/div>\r\n
Then you have a diverse array of access methods. From the classic ssh\/sftp command line interface to more modern things like OpenOn Demand<\/a>, ThinLinc<\/a> or Jupyter notebooks<\/a>. It's all good stuff but it's a lot of messing around with an array of (admittedly fascinating) technologies when often all you want to do is get your results more quickly with as little fuss as possible.<\/div>\r\n
Wouldn't it be nice if I could interact with all of these HPC and cloud machines without ever leaving MATLAB? No Linux, no schedulers and no shell scripting -- just MATLAB? Well you can, and I'll show you how.<\/div>\r\n
\"\"<\/div>\r\n

Submit to all the things, direct from MATLAB<\/span><\/h4>\r\n
Imagine I have a function:<\/div>\r\n
[result1,result2] = bigComputation(N)<\/span><\/div>\r\n
It's a computation and it's big! The single input argument is an integer that defines just how big and it has two output arrays. Obviously it also runs in parallel because this is a HPC post.<\/div>\r\n
OK, so here's how I might run that function on my various machines using MATLAB. First, we define the problem and create objects representing my compute resources.<\/div>\r\n
\r\n
\r\n
% Define how big a computation I want to do<\/span><\/span><\/div>\r\n<\/div>\r\n
\r\n
N = 1e8;<\/span><\/div>\r\n<\/div>\r\n
\r\n
<\/div>\r\n<\/div>\r\n
\r\n
% Create parallel cluster objects representing my various computational resources<\/span><\/span><\/div>\r\n<\/div>\r\n
\r\n
myMachine = parcluster(\"Processes\"<\/span>); % My local PC<\/span><\/span><\/div>\r\n<\/div>\r\n
\r\n
onPremHPC = parcluster(\"University of Bantshire HPC\"<\/span>); % A SLURM based cluster<\/span><\/span><\/div>\r\n<\/div>\r\n
\r\n
cloudHPC = parcluster(\"MyAWS Cluster\"<\/span>); % A MATLAB Cloud Center Cluster running on AWS<\/span><\/span><\/div>\r\n<\/div>\r\n<\/div>\r\n