Loren on the Art of MATLAB
April 19th, 2006
High Performance File I/O
Today I'd like to tell you how you might improve the performance of reading and writing files using MATLAB's low-level file i/o functions. The key here is to disable the automated buffer flushing.
Experience at MathWorks
The default behavior for fprintf and fwrite is to flush the file buffer after each call to either of these functions. This can have a noticeable performance impact if you are calling either of these functions many times to write small chunks of data each time.
MATLAB has a way to disable the auto-flushing, which is to fopen the file using mode 'W' instead of 'w' (or 'A' instead of 'a' for append). Since fclose also flushes the file, if you have a function that fopens a file, writes small chunks of data to it many times, and then fcloses it, you might want to consider changing your fopen mode.
We made made this change inside wk1write.m recently. When writing out a 200x200 array of doubles using wk1write, the elapsed time shrank from 6.55 seconds down to 4.34 seconds, a savings of 50%.
This technique won't work well if you are in the situation where you need to both read and write to the same file often throughout processing. For that, you need to flush the buffer after each call.
What sorts of file reading/writing bottlenecks do you encounter? Will this tip help you? Let me know.
2:23 pm |
Posted in Efficiency |
Permalink |
You can follow any responses to this entry through the RSS 2.0 feed.
You can skip to the end and leave a response. Pinging is currently not allowed.
Leave a Reply
|
I haven’t used it in awhile, but last time I did movie2avi seemed to run much slower than it should. I looked through the code and it seemed that the avi file created by this function was being opened and closed for every frame. After reading this post, I went back and looked at addframe.m again and I see this line (308):
fid = fopen(filename,’a',’l');
Could it be that simply changing the ‘a’ to ‘A’ here would speed up the writing of a movie to an avi file?
When using this technique, is the only way to flush the buffer to call fclose?
Or is there some other function call that will flush?
Or does it automatically flush when the buffer reaches its limit? If so, what is the limit (in bytes)?
Loren,
I am writting in connection to a recent answer given by you to on the newsgroup and concerning out of memory problems …
Also with the most recent version of matlab (2006a) there are cases of routines that do not support single precision
In my view this is a major problem for many users of matlab,
and huge memory boards do not help as much as a matlab tuning to make single precison always possible
I have a direct experience with stream3 that cannot be forced to use single precison and uses 3 huge gradient arrays in double
thank you , Gianni
%%%%%
Chris
>>
> MATLAB supports arithmetic and lots of other math on singles for
> MATLAB
> 7. Perhaps upgrading would be beneficial for you.
I would recommend in the future sending information like this to our Technical Support group. They pass information like this along to the development team who then prioritize their work with user requests in mind. The link is:
http://www.mathworks.com/support/contact_us/ts/ebd/enhance_bug_doc_1.html
Pat-
The only way to guarantee the buffer is flushed is to use fclose if you open with A or A. Using a or w flushes the buffer after each write.
–Loren
Wow! That was a good one.
I use a lot of fprintf in link level simulations of communication systems. The data is printed to files frame-by-frame, and the files are used to test Verilog implementation of the actual modem hardware.
I’ve just tried your tip, and savings in simulation time are tremendous.
Thank you!
Eric