People have different ideas about what costs a lot, in terms of memory, in MATLAB. And sometimes people don't know the details. Today I am going to talk about when MATLAB makes copies of data, for both calling functions, and for data stored in variables. MATLAB makes copies of arrays it passes only when the data referred to changes (this is called copy on write or lazy copying).
Contents
Passing Arrays to Functions
The question is, when does MATLAB copy memory when passing arrays to functions. Some users think that because MATLAB behaves as if data are passed by value (as opposed to by reference), that MATLAB always makes copies of the inputs when calling a function. This is not necessarily true. Take a look at this function.
type fred1function y = foo(x,a,b) a(1) = a(1) + 12; y = a * x + b;
In fred1, the first and third inputs, x and b, are not altered inside. MATLAB recognizes this and passes both these variables in without making any extra copies. This can be a big memory savings, for example, if x is a large dataset. However, in fred1, we can see that the second input, a, gets modified inside. MATLAB recognizes when this is happening to a variable and makes a copy of it to work with so that the original variable in the calling workspace is not modified.
Structures and Memory
Each structure member is treated as a separate array in MATLAB. This means that if you modify one member of a structure, the other members, which are unchanged, are not copied. It's time for an illustration here.
Create some rgb image data with 3 planes: red, green, and blue.
im1.r = rand(300,300); im1.g = rand(300,300); im1.b = rand(300,300);
Instead, rearrange the same data so that we have an array of structs each element containing an [r g b]-triplet.
im2(300,300).rgb = [0 0 0]; % preallocate the array for r = 1:300 for c = 1:300 im2(r,c).rgb = [im1.r(r,c) im1.g(r,c) im1.b(r,c)]; end end
Let’s compare im1 and im2.
clear c r % tidy up the workspace whos
Name Size Bytes Class im1 1x1 2160372 struct array im2 300x300 7560064 struct array s 1x1 392 struct array sNew 1x1 392 struct array Grand total is 630043 elements using 9721220 bytes
im1 is a scalar structure with members that hold m x n arrays.
- im1.r = imageRedPlane --- size m x n
- im1.g = imageGreenPlane --- size m x n
- im1.b = imageBluePlane --- size m x n
im1 is size 1 x 1; total # of arrays inside im1: 3
im2 is an m x n structure array with fields containing 3-element vectors.
- im2(i,j).rgb = imageOneRGBPixel --- size 1 x 3
im2 is size m x n; total # of arrays inside im2: m x n
Notes: Every MATLAB array allocates a header with information. This makes im1 more memory-efficient than im2 (more generally, scalar structs containing arrays are more memory-efficient than a struct array). When one field in a structure is changed, and possibly copied, the other fields are left intact.
s.A = rand(3); s.B = magic(3); sNew = s; sNew.A(3) = 14;
Since s and sNew have unaltered copies of B, the B fields share memory, but the A fields do not. See the documentation section titled Using Memory Efficiently for more information.
What's Your Mental Model for MATLAB Memory Management?
Say that three times fast!
Does the description here and/or in the documentation change your model?
Published with MATLAB® 7.2
My mental model is ambiguous, because it’s all folklore from other MATLAB users that has accumulated across many MATLAB releases. I’ve seen someone say on the newsgroup that MATLAB does “copy on write” which is what this post seems to show (and I am glad to hear), but that individual’s claim didn’t come with a @mathworks.com address in the From: field, so I classified it as folklore at best, despite how often he seems to post.
It’s good to get the facts from the source at least; to see how Matlab memory management behaves in the wild. It would be nice to see any other optimizations or behaviors as in the case of function fred1.
Hi!
I often have the situation that a variable is passed to a function, changed inside that function and then used in the calling function instead of the old version, which is no longer used. As an example, I could call function myfun like this:
x = myfun(x); % overwrite x with the new value
Function myfun has the following form with x as in- and output argument:
function x = myfun(x)
% modify x here, for example
x = x.^2;
Does Matlab recognize that in this case there is no need for copying the input array x, even if it is changed inside the function? x could be a large array or a structure with many fields, so this could save quite some effort.
When implementing myfun as a mex-file, it is possible to change variable values without copying the data. However, I have always been following the advice somewhere in the Matlab documentation *not* to do this because this could lead to “unexpected results”. Is this advice still of relevance? I guess there would hardly be unexpected results when changing an element of a matrix, but what if expanding a matrix or adding a field to a structure?
Markus and Ben-
Thanks for your comments.
To answer the last question first, yes, we still recommend that you not overwrite input data in MEX-files. That’s because, since MATLAB semantics is value-based, other users (or you sometime later perhaps) are not generally expecting calling function workspace variables used as inputs to change. And if an error occurs somewhere within the chain of calculations, you may end up with some “corrupted” variables that would not occur otherwise.
For the first question, in cases like this,
MATLAB R2006a and earlier does not take advantage of reusing the memory for the input variable x, nor equivalent behavior for a field of a structure. We are constantly looking for and making performance improvements in MATLAB. The one you mention is on our list.
Great writeup!
I haven’t been clear about structures not necessarily needing continuous memory space. It states that in the documentation. Does that mean each field does not need to be in the same continuous space as the rest of the structure?
For functions like this, where x is very large, I have been using nested functions. Actually I tend to have several large variables and this works out well for me. I’d prefer matlab be smart about the whole thing since nested functions can complicate things a little.
function x = myfun(x)
% modify x here, for example
x = x.^2;
Allen-
Each field of a struct is its own MATLAB array and each one needs its own contiguous memory, but the union of the struct fields/elements does not need to be contiguous.
Thanks for your comments.
–l
Hi,
In which future MatlabRelease will the improved Variable-handling be implemented?
( function x = myfun(x)
% modify x here, for example
x = x.^2;) -> don´t make a local copy of x;
In this context I also miss the possibility to hand a variable over as reference. So no lokal copy of it will be made.
In C++ this is done through a ‘&’ before the variable.
function x = myfun(&x)
Thanks a lot for answering and
Best regards
Hartmut
Harmut-
Thanks for your interest. In general, we don’t pre-announce features for given releases. Sorry.
We have also have the request for references. That is on our idea list for the future.
–Loren
Hi!
As was stated previously, a most useful writeup.
Just a question of clarification:
Arrays are not copied until changed when sent as arguments to functions. Is this also true if you create a (huge) array in a function and return it as output to the caller?
/Lars
Hi Lars-
Yes, this is true. If you do something inside a function to create an output, an extra copy is not made to assign it to the output variable in the caller. The exception is if there’s an unusual output like an indexed variable:
In this case, the large array is copied into the memory in the variable A. Otherwise, there is no extra copy made.
COW is great for the common case - simple semantics, good performance, transparently. How do I deal with the less common case, where I actually want to know whats going on? (what copies are being done)
I have about 200MB of data. While processing it, I suspect lots of memory is being allocated, because Matlab seems to take >600MB of system memory. This might be unavoidable, but I want to know what lines in my program allocate the bulk of the memory, so I can consider changing the implementation.
Is a memory profiler in the todo lists? in addition to being a useful tool, it would also probably improve all of our models of the memory management system… in the mean time, any practical advice would be welcome.
Dear Loren,
many thanks and congratulations for your helpful column.
I write you concerning to the problem
x = myfun(x) .
I understand your recommendation that mex files should not overwrite input data because other users (or the same user sometime later) are not generally expecting inputs to change. Anyway I implemented a mex file called “inplace” (in order to warn me about its behaviour) that does it. The last argument tells the function what to do and the first ones are the actual arguments of the function.
To date I implemented a few functions I need, i.e. some C-like operators and circshift:
inplace(x, y, ‘+=’);
inplace(x, y, ‘.*=’);
inplace(x, ns, ‘circshift’);
…
They do not allocate memory and compared to the matlab equivalent
x = x + y;
x = x .* y;
are (when working with arrays above 300-400 MB) much faster (tenths of second instead of a few minutes) because the pc does not start swapping to the disk.
For the C-like operators, a matlab-way workaround to limit allocation could be to split the operation in suboperations like:
for i = n:n:N
ii = (i-n+1):i;
x(ii) = x(ii) + y(ii);
end;
x((i+1):end) = x((i+1):end) + y((i+1):end);
but it is not as fast and not very elegant.
For this reason and to perform inplace circshift (much harder to split in suboperations), I decided to go for the mex file.
But a problem arise because of the copy-on-write policy (a really smart and efficient solution in most situations):
w = …something…;
x = w;
inplace(x, y, ‘+=’);
The unpredictable result is that w changes together with x after inplace is called.
Obviously the workaround is to change x before passing it to inplace:
w = …something…;
x = w;
x(1) = -x(1);
x(1) = -x(1);
inplace(x, y, ‘+=’);
and it works. But what if sometime later I forget about this trick? Searching the web for a solution I found someone using two apparently undocumented functions: mxIsSharedArray and mxUnshareArray. Then performing
if(mxIsSharedArray(prhs[0]))
mxUnshareArray(prhs[0]);
before using prhs[0] solves the problem. A more conservative approach could be:
if(mxIsSharedArray(prhs[0]))
mexErrMsgTxt(”The array is shared. Cannot proceed.”);
Now my questions…
Why in matlab such a clean and fast solution as “x += y” is not supported?
Concerning to my solution, can you suggest me better ones avoiding breaking the rule to not alter inputs?
If not, is what I am doing safe?
Concerning to the two functions, is there a risk that in a future implementation they will be dropped? Why are these undocumented? Is the use of mxUnshareArray safe or should I switch to the more conservative approach?
Thank you for your help.
Regards,
Luca
Simple arithemtic operations like the ones you mention:
x = x + y;
x = x .* y;
…are currently performed in-place in M-code, but not at the command line (watch the task manager with a big array),. It is performed by the JIT/Accelerator which does not operate at the command line. As Loren said, we are looking at making this work for M-functions, built-in (C) functions and perhaps Mex files.
Sorry I’ve been off-line for a while. Here are some very quick responses.
Daniel- A memory profiler is on the futures list.
Luca- I am not a mex expert and am hoping someone who is can make a helpful comment.
As for +=, there are a lot of things to consider, which is the left-hand side might have duplicate array values: x([3 4 3]) += 17. Do we add 17 to the 3rd entry or 34? We can legitimately choose either and explain it but we want it to be clean and useful. Plus, it will be much cleaner for us to implement after we do some heavy internal refactoring/cleaning up in that part of the code base. Finally, we need to consider how users of the class system can overload it. So it’s not just a no-brainer for us.
Thank you both for the answers.
Stuart…
I only tried from the command line and noticed the big increase in memory usage. But good to know that with M-files it is performed in a smarter way.
Loren…
I agree with you concerning to the x([3 4 3]) += 17 case. To avoid confusion, I would make it behave like x([3 4 3]) = x([3 4 3]) + 17 i.e. to add 17 only once. Actually I think it is performed twice but the last one overrides the first one because in
a([1 3 1 1]) = a([1 3 1 1]) + [1 2 3 4]
a(1) is increased by 4
and
tic; a(ones(1000000,1)) = a(ones(1000000,1)) + 1; toc
takes much more time than a(1) = a(1) + 1.
Anyway if, like Stuart said, the operation is optimized in M-files += (and the others) are not a priority.
Regards,
Luca
Here’s a technique I came up with the other day to prevent copy-on-write occuring in some cases:
function dosomething(x, deleteme)
assignin(’caller’, deleteme, []);
x(1) = 123;
…
end
…
dosomething(big_matrix, ‘big_matrix’);
As long as big_matrix was the only variable using that chunk of data, then copy-on-write does not occur when changing x inside the dosomething function.
A question about partial arrays: in the following lines, will b be copied or will it contain just a pointer into A?
A = rand(big_number,5);
b = A(:,3); % is there a copy performed here?
I presume b=A(3,:) must be copied as this is against the grain, but b=A(:,3) is a continous chunk.
Thanks for your help,
Andras
Hi Andras-
Only complete arrays are shared. So b, in your example, has a copy of the 3rd column of A. b is a contiguous chunk but wouldn’t be if you selected all the columns in a specific row, e.g., A(3,:).
–Loren
Thanks Loren. That is too bad. Does the JIT/Accelerator eliminate this copy when I use A(3,:) in an asignment:
c = c + A(3,:);
or
c=c+A(:,3); % non-continous case
thus making
b=A(3,:);
c=c+b;
less efficient then the above?
Thanks again,
Andras
Andras-
Currently, the indexing case that’s not contiguous does cause an allocation in the JIT/Accelerator; the contiguous case does not.
–Loren
Hi,
maybe this is the place to ask: I frequently run into ‘out of memory messages’ when batch processing large numbers of medical files (images of ~40MB). I pack, I clear, I run calculations only on the necessary parts of the images. By this I can move the ‘out of memory…’ message back from say the 4th image to the 10th. MATLAB memory usage grows continuously, no matter what is visible in the workspace.
Is there any way to handle this? Help would be very much appreciated! I have looked around, but haven’t found a solution.
Thanks!
Georg
Georg-
I recommend you take a look at this collection of slides and examples that Stuart put together on the File Exchange.
–Loren
Hi,
After looking at various memory management info I still have the following question. Hope you can help.
I need to know how the preallocation works when I don’t know what the final size of my matrix will be. I will know the maximum size, but I don’t want my final matrix to be of this maximum size. And I dont really want to operate on the matrix with that maximum size. I have no problem preallocating a matrix, but suppose I really want to grow it with a statement such as A = [A;B].
Example: If I preallocate A = ones(1,1000), does that mean that A has that memory regardless of changes made to the variable, for example, if I now set A = B (where B is 1×10). Has A now lost its 1000×1000 space? Suppose I later set A = [A;B] making A 1×20, is it now growing in non-continuous memory or does it still have access to the 1×1000 area created when A was first created? Another way to phrase my question: does A only lose its 1×1000 preallocated memory when I use the clear function?
Thanks
Dadi
Dadi-
If you preallocate A to 1000×1000 and then say B = 1:10, A = B, A will have only space for 10 elements. If you want to keep it with 1000×1000, you must assign by indexing, e.g.,
A(1:length(B)) = B;
A = [A B]
will find contiguous space large enough for all the data from A to be together and then delete the space for the 2 smaller pieces. The original space is long gone. Again, to preserve it, you must assign using indexing on the left-hand side.
–Loren
Loren
I’m loooking for a way to decide which variables to clear to set memory free. That is in code in functions where the same data may be refered to by variables in different scopes. The problem is that little memory is set free if I clear a variable, the data of which is referenced be another variable.
/ per
Per-
There isn’t a good way. If variables are linked, then deleting one of them will not free the memory associated with the data, as you’ve noted. You’d have to know somehow yourself if you created certain arrays and they could be cleared.
Would you want this for debugging? If so, you might explore format debug. It shows the pointer to the data and if 2 arrays point to the same data, they will show that. Programmatically there is not a good way however.
–Loren
Loren-
Thanks Loren. No, my problem is not debugging. I try to make a cache.
I’m developing an interactive tool for looking at measured data. The total amount of data is orders of magnitude larger than would fit in ram-memory. I want to give the user the feeling all data are available all the time. To that end, I store the measured data it special binary files, use memmapfile, and have a cache.
This works surprisingly well until it’s time to clear data from the cache. The measure data is never changed, but in my code it is refered to from closures (function handles of nested functions), which I have problems keeping track of. Now a clear the oldest data in the cache, which causes major problems. The following happens: 1) time series A is in use, 2) I delete A from the cache, 3) user asks for A for a new plot, 4) a new copy of A is created in the cache, 5) et cetera. Work around: close tool and start with a fresh memory.
I guess there a books on caches, non of which I have opened.
/ per
Loren,
I am trying to improved performance large (2-3 Gb) loading binary multiplexed (int16) data from a file. The problem is that I need to load only some channels, making non-continuous reads. I used fread before with various buffer sizes, and keeping only subset of raws. Now I tried the same data load with memmapfile thinking it would give better performance and it doesn’t. I gain due to caching with second read from the same file, but on the first run from uncached files I get about the same speed. At the end of the day, doesn’t it depend on the page size matlab internally uses for memmaping vs me for fread buffer? Is there a way to get the optimal performance with eaither solutions? E.g. what page sizes do fread and memmapfile use? Also, I am running 64bit Linux and 64bit matlab 7.1 sp3, and max size memmapfile allows me to map is 2^31-1, why is that?
Anton-
MATLAB doesn’t allow the user to control the buffer size now so unfortunately there is nothing specific I can tell you. As for the 64-bit MATLAB, even there for now, the maximum data size is controlled by a 32-bit pointer.
–Loren
I am thinking about storing data in compressed in memory form way and feed Cashe with decompressed data. It would solve number of problems, but I am missing standard Cashe support in MATLAB. Some ideas? Existing custom implementations?
I am trying to read large binary data files (over 2GB).
At present, I am using memory-mapped file IO via a C++ MEX file.
Does the creation of the memory-map view via the win32 “MapViewOfFile” function violate the “allocate memory with mxCalloc & mxMalloc only” rule?
William-
The short answer is “yes,” the use you suggest does violate the rule you mention. You are really best off sticking exactly with our documented APIs and methods for now.
–Loren
Does COW maybe work for local variables (not only for functions args/automatic vars)? Say after v=zeros(100);a=v;b=v;c=v; will all 4 variables share the same matrix? If they do, what happens after v(1,1)=1? Do they all get their own copy, or do maybe a, b, and c still share the same copy?
Thanks,
Ljubomir
Ljubomir-
Copy on write works for variables in general. v,a,b, and c will all point to the same memory at first. Once v has an element changed, a,b, and c will all share memory and v will have its own.
–Loren
Excellent, thanks Loren.
Ljubomir
Hi Loren,
Sorry if this should be obvious from the above, but is it better to use a subset of a large array within a function than to pass only that subset. E.g. is the first option below any better than the second, in terms of memory? If so, is it better only in the contiguous case? Many thanks,
Ged
%% option 1, “pass” (copy-on-write) then subref
x = randn(1e4);
process(x, 1:10)
function [y z] = process(x, sub)
y = mean(x(:, sub)); % contiguous chunk
z = mean(x(sub, :)); % non-contiguous
%% option 2, pass in subset
x = randn(1e4);
process(x(:, 1:10), x(1:10, :))
function [y z] = process(x1, x2)
y = mean(x1);
z = mean(x2);
Ged-
There is not a simple answer. Depends if you are running into memory issues or not. And how large a subset you typically work on. And your computer’s memory and cache. In either case for you, you are creating that new matrix either explicitly or as a temporary variable when you process so I don’t think I see any major difference memory-wise for you. I’d worry about maintainability and readability as well.
–Loren
I’ve been battling the same problem Markus had (x = myfun(x)), and was inspired by Paul’s deleteme idea. Here’s my solution to our problem:
myfun(x, deleteme)
assignin(’caller’, deleteme, []);
x = x.^2;
assignin(’caller’, deleteme, x);
end
Call it with:
myfun(x, ‘x’);
As Paul noted, as long as x is the only variable using that memory, this trick works properly. Otherwise, when the caller’s variable is cleared, Matlab makes a copy for the callee, causing the function to bog down again. Loren, do you know if this is going to get me in trouble somehow, or does this look to you like a good way to achieve pseudo-pass-by-reference?
-Jonathan
Jonathan and Markus-
Have you read this blog entry? It explains the in-place paradigm in MATLAB. You might want to be sure you are putting that to the best use before directly manipulating workspaces from the caller.
–Loren
I’m looking for a way to set memory free in a while loop. At the beginning I was not able to run the first iteration of the loop there were always an “Out of Memory error”. I was thinking that I will be fine if I can run just the first iteration of the loop, planning to clear all variables in the loop before starting the second iteration. After reducing the size of arrays and matrix, I got the first iteration. I used then the function clear var1 var2 … that is fine with my laptop (windows) but not in the lab computer (Unix). It is as if the function clear.m doesn’t free memory after clearing the variables in Unix system. Do any one know how to free memory allocated for the cleared variables in Unix?
Etienne-
The variables are very likely cleared but may not leave enough contiguous space in memory for new arrays to get created the next time through. You might want to look at technical note 1106 for help with memory issues or contact technical support for issues specific to your code.
–Loren