{"id":4309,"date":"2026-06-12T09:42:27","date_gmt":"2026-06-12T13:42:27","guid":{"rendered":"https:\/\/blogs.mathworks.com\/matlab\/?p=4309"},"modified":"2026-06-12T09:42:27","modified_gmt":"2026-06-12T13:42:27","slug":"direct-communication-between-multiple-gpus-using-matlabs-spmd-functions","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/matlab\/2026\/06\/12\/direct-communication-between-multiple-gpus-using-matlabs-spmd-functions\/","title":{"rendered":"Direct Communication between Multiple GPUs using MATLAB’s spmd Functions"},"content":{"rendered":"
Today's guest blogger is <\/span>Weinan Chen<\/span><\/a>. Weinan is an Application Engineer on the Parallel Computing team at MathWorks. Additional thanks to Linda Koletsou Soulti, Joss Knight, and the Parallel Computing development team for their help writing up this article<\/span><\/div>
TL;DR: <\/span>spmd<\/span> lets MATLAB users efficiently coordinate and communicate across multiple workers. Behind the scenes, GPUs can exchange data directly with each other, skipping the CPU entirely. Researchers from TU Dresden used this technology to implement a fast multi-GPU FFT.<\/span><\/div>
When it comes to running multi-GPU simulations in MATLAB, a good starting point (depending on the task) is to use simple parallel constructs to distribute work across multiple GPUs. For example, one can parallelize deep learning training with a <\/span>flip of a switch<\/span><\/a> or run data-parallel for-loops using <\/span>parfor across multiple GPUs<\/span><\/a>.<\/span><\/div>
But what if our computational task doesn't resemble either example? For instance, if our multi-GPU application requires frequent exchange of information during the computation, what options do we have?<\/span><\/div>
In this case, <\/span>the spmd family of MATLAB functions<\/span><\/a> is the right choice. For those familiar with the Message Passing Interface (MPI), the syntax of <\/span>spmd<\/span> closely resembles MPI programming. Essentially, this construct gives users the flexibility to tackle advanced distributed-memory tasks (and it also works with <\/span>shared-memory programming models<\/span><\/a> in MATLAB).<\/span><\/div>

Inter-worker communication on the CPUs<\/span><\/h2>
In this post, I\u2019ll focus on the performance aspects of using <\/span>spmd<\/span> for inter-worker communication. Imagine our application needs to concatenate data across workers using an all-reduce-style operation, where . For example:<\/span><\/div>
function <\/span>spmdCatCPU(numWorkers)<\/span><\/span><\/div><\/div>
 <\/div><\/div>
if <\/span>isempty(gcp(<\/span>\"nocreate\"<\/span>))<\/span><\/span><\/div><\/div>
parpool(<\/span>\"Processes\"<\/span>, numWorkers);<\/span><\/span><\/div><\/div>
end<\/span><\/span><\/div><\/div>
 <\/div><\/div>
spmd<\/span><\/span><\/div><\/div>
x = rand(10000, 2000);<\/span><\/span><\/div><\/div>
tic<\/span><\/span><\/div><\/div>
<\/span>for <\/span>i = 1:6<\/span><\/span><\/div><\/div>
z = spmdCat(x);<\/span><\/span><\/div><\/div>
<\/span>end<\/span><\/span><\/div><\/div>
y = toc;<\/span><\/span><\/div><\/div>
end<\/span><\/span><\/div><\/div>
 <\/div><\/div>
y = gather(y);<\/span><\/span><\/div><\/div>
 <\/div><\/div>
% Turn y from cell array to numeric array<\/span><\/span><\/div><\/div>
y = mean(cell2mat(y));<\/span><\/span><\/div><\/div>
 <\/div><\/div>
% Display the mean execution time<\/span><\/span><\/div><\/div>
disp([<\/span>'Mean execution time: '<\/span>, num2str(y), <\/span>' seconds'<\/span>])<\/span><\/span><\/div><\/div>
 <\/div><\/div>
end<\/span><\/span><\/div><\/div><\/div>
numCPUWorkers = 4;<\/span><\/span><\/div><\/div>
spmdCatCPU(numCPUWorkers)<\/span><\/span><\/div>
Mean execution time: 11.168 seconds<\/div><\/div><\/div><\/div><\/div>
In the code snippet above, each MATLAB worker initializes a random matrix <\/span>x<\/span>. We then concatenate all <\/span>x<\/span> values across workers and broadcast the result, <\/span>z<\/span>, back to every worker. Here, the <\/span>spmdCat<\/span><\/a> function performs both the computation (concatenation) and the communication in a single call.<\/span><\/div>
So how long does this take? Running 4 MATLAB workers on <\/span>4 AMD EPYC 7R32 CPUs<\/span>, the script takes <\/span>11.17 seconds<\/span>.<\/span><\/div>
We can use the MATLAB built-in <\/span>mpiprofile<\/span><\/a> function to investigate where that time is spent. In this case, communication <\/span>between some worker pairs takes the majority of the time (around 10 seconds).<\/span><\/div>
\"\"<\/img><\/div>
Figure 1. The communication heat map between CPU workers generated by mpiprofile.<\/span><\/div>

Inter-worker communication on the GPUs<\/span><\/h2>
Now, what if we perform the same operation across multiple GPUs? The only change required is to make <\/span>x<\/span> a <\/span>gpuArray<\/span>, as follows. <\/span><\/div>
function <\/span>spmdCatGPU(numWorkers)<\/span><\/span><\/div><\/div>
 <\/div><\/div>
if <\/span>isempty(gcp(<\/span>\"nocreate\"<\/span>))<\/span><\/span><\/div><\/div>
parpool(<\/span>\"Processes\"<\/span>, numWorkers);<\/span><\/span><\/div><\/div>
end<\/span><\/span><\/div><\/div>
 <\/div><\/div>
spmd<\/span><\/span><\/div><\/div>
x = rand(10000, 2000, <\/span>'gpuArray'<\/span>);<\/span><\/span><\/div><\/div>
tic <\/span><\/span><\/div><\/div>
<\/span>for <\/span>i = 1:6<\/span><\/span><\/div><\/div>
z = spmdCat(x);<\/span><\/span><\/div><\/div>
<\/span>end<\/span><\/span><\/div><\/div>
y = toc;<\/span><\/span><\/div><\/div>
end<\/span><\/span><\/div><\/div>
 <\/div><\/div>
y = gather(y);<\/span><\/span><\/div><\/div>
 <\/div><\/div>
% Turn y from cell array to numeric array<\/span><\/span><\/div><\/div>
y = mean(cell2mat(y));<\/span><\/span><\/div><\/div>
 <\/div><\/div>
% Display the mean execution time<\/span><\/span><\/div><\/div>
disp([<\/span>'Mean execution time: '<\/span>, num2str(y), <\/span>' seconds'<\/span>])<\/span><\/span><\/div><\/div>
 <\/div><\/div>
end<\/span><\/span><\/div><\/div><\/div>
numGPUWorkers = 4;<\/span><\/span><\/div><\/div>
spmdCatGPU(numGPUWorkers)<\/span><\/span><\/div>
Mean execution time: 1.2486 seconds<\/div><\/div><\/div><\/div><\/div>
How long do we think this version will take? Intuitively, we might expect it to be slower than the CPU version: after all, GPUs are often viewed as co-processors, and moving data across multiple GPUs sounds like it would require staging via host memory. In practice, the GPU version is significantly faster. It completes in <\/span>1.24 seconds on 4 NVIDIA A10G GPUs<\/span> over the same PCI-e connection\u2014almost <\/span>9.0\u00d7 faster<\/span> than running the same code on <\/span>4 AMD EPYC 7R32 CPUs<\/span>.<\/span><\/div>
Running this script through <\/span>mpiprofile<\/span> reveals that transferring the same amount of data across multiple GPUs is much faster.<\/span><\/div>
\"\"<\/img><\/div>
Figure <\/span>2<\/span>. The communication heat map between GPU workers generated by mpiprofile.<\/span><\/div>

Why is inter-worker communication on the GPUs faster?<\/span><\/h2>
Now, back to our original question, how is multi-GPU communication faster than its multi-CPU counterpart?<\/span><\/div>
The answer is that our Parallel Computing Toolbox developers have employed a technique called <\/span>NVIDIA CUDA Inter-Process Communication (IPC)<\/span><\/a> <\/span>when implementing <\/span>spmd<\/span> function for GPU communications. CUDA IPC makes it possible to share GPU allocations between processes via portable handles, so that data can be accessed without staging through host memory. <\/span>Multi-GPU <\/span>spmd<\/span> workflows in MATLAB can benefit from these GPU-aware mechanisms, which contribute to the speedup we saw earlier. One limitation is that this applies to <\/span>process pools<\/span> (not thread-based pools).<\/span><\/div>
As a result, when MATLAB runs multi-GPU code, it automatically benefits from these optimizations to accelerate simulations.<\/span><\/div>

Implementing a multi-GPU Fast Fourier Transform<\/span><\/h2>
One example of how this technology has helped MATLAB speed up multi-GPU calculations is discussed in this <\/span>paper<\/span><\/a> (<\/span>code<\/span><\/a>). By using the <\/span>spmd<\/span> family of functions (<\/span>spmdCat<\/span>, <\/span>spmdSend<\/span>, <\/span>spmdReceive<\/span>, etc.) to implement a multi-GPU <\/span>fft<\/span>, the authors achieved a maximum speedup of approximately <\/span>60\u00d7<\/span> when running on <\/span>4 NVIDIA H100 GPUs<\/span>, compared to <\/span>100 CPUs<\/span>. <\/span><\/div>
If you are working with<\/span> advanced <\/span>data-communication patterns in your multi-GPU MATLAB applications, don\u2019t hesitate to give the <\/span>spmd<\/span> functions a try!<\/span><\/div>\r\n<\/div>