The mystery of the failing TIFF append loop10

Posted by Steve Eddins,

For today's post I'd like to introduce guest blogger Ashish Uthama, a MATLAB developer. He's got a story to share with you about a topic, TIFF import and export, that I know interests many blog readers.

I work in the Image and Scientific data team. Our team goal is to make it easy for you to work with various formats like TIFF, NetCDF and HDF5 to name a few. We recently came across an interesting issue with imwrite that Steve suggested would be of interest to readers of this blog. In fact, a comment on this blog was one of the data points in our investigation.

About three months ago, we started hearing about instances where the following piece of code would error out sporadically after completing a few loop iterations successfully:

   imageData = ones([400 400 100]);
tiffFile = 'file.tif';
for ind = 1:100
imwrite( imageData (:,:,ind), tiffFile,'WriteMode','append');
end

The error message was a basic file open error:

   ??? Error using ==> writetif at 100
Couldn't open 'file.tif' for writing.

We didn’t have much luck reproducing this issue initially. Sporadic failures are extremely frustrating for both users and developers. Imagine a complex program, processing hundred’s of images, erroring out sporadically with this message!

The first step was to analyze the failure itself. In each iteration, imwrite tries to open the file in read-write mode, appends an image to the TIF file and closes it. The error message indicates that at one of these iterations, imwrite was unable to open the file. This implies that either the permission changed or the file was locked otherwise for writing at that moment.

The second step was trying to figure out the cause for this lock. We were lucky with one of the first reports where the user associated the failure mode with an open Windows Explorer window pointing to the destination folder. His program did not error out when he closed the Windows Explorer.

We continue to investigate this behavior, but our working assumption now is that the OS locks the file while it gathers meta-data or creates a preview thumbnail when Windows Explorer is open. So far, the OS in the reported cases has been limited to Windows 7.

With the information we have as of this moment, we are not sure yet if a truly robust workaround exists other than closing the Windows Explorer window. A first defensive programming instinct might be to check for a lock before writing within imwrite. However, unless the check and the write operations are atomic, this approach would not be successful. The function (or MATLAB for that matter) would not be able to check efficiently if a file is genuinely unwritable or if it was locked momentarily for writing.

When we find an issue like this and have information which would help our users proceed with their work, our technical support team puts together a page with the details of the workaround. Here is the page our team created for this issue:

https://www.mathworks.com/support/solutions/en/data/1-DQPRHC/?solution=1-DQPRHC

You can also get to it from https://www.mathworks.com/support/ by searching for ‘imwrite loop’.

Get the MATLAB code

Published with MATLAB® 7.11

Note

Mark Hayworth replied on : 1 of 10
One useful tool in situations like this is unlocker: http://download.cnet.com/Unlocker/3000-2248_4-10493998.html After you install it, it lies waiting for a locking violation like that. Then it springs into action and tells you which application has it's hands on your file. Then it gives you the opportunity to pry that app's hands off your file (unlock the file), or you can delete the file, or just do nothing.
Ruediger Alshut replied on : 2 of 10
I figured out that the windows indexed search is the reason for this error. When I just run the code in a folder that is excluded from the windows indexed search I didn't get the error. (Win 7 x64)
Arthur replied on : 3 of 10
Luckily I never had problems like this (WinXP). But would it be possible to solve it with a try... catch? So you try to write the tiff, and if you catch this particular error, you simply try to write it again?
Steve replied on : 4 of 10
Ruediger—Thanks for the tip. Arthur—As far as we know so far, this problem only happens with Windows 7. Also, we didn't mention that a related symptom of the problem is that performance (speed) goes way down. The try-catch idea, even if it could be done robustly, would likely worsen the speed problem.
Mark replied on : 5 of 10
I batch process 1000's of TIFFs at a time (read and write), and occasionally run into this error. I was suspicious that it was my network being flakey, so I put in a try/catch in a while loop so that if it happens, it pauses for 1 minute and then re-tries to write the file. So during one of the try/catch/pause loops, I experimented. What I found is that this problem happens even when I'm running local, and sometimes it was my error (disk full), and sometimes it was due to virus scanning and file indexing. I have this same problem on both Windows 7 and XP, MATLAB 2009a and 2010b.
Steve replied on : 6 of 10
Mark—Thanks for the info.
Pete replied on : 7 of 10
"We were lucky with one of the first reports where the user associated the failure mode with an open Windows Explorer window pointing to the destination folder. His program did not error out when he closed the Windows Explorer." Cool! That was me! Okay, I can't be sure that was me, but that's how I summarized my service request (imwrite errors when an explorer windows is open to the imwrite target folder). I'm kind of excited that an observation I made might save somebody a little frustration. I haven't been coding long, so I've exclusively been on the receiving end of helpful advice until now. @Ruediger Alshut: It's not the indexing. Or at least, it's not solely due to the indexing service. I can reproduce the error on my removable disks (which are definitely not indexed), as well as indexed or unindexed folders on the system disks on my MATLAB boxes. In fact, I first saw the error when writing pages to multipage tiffs on my removeable drives. That's part of the difficulty Ashish mentioned in the post. The sporadic nature makes it hard to diagnose, and if you try something to mitigate the issue, but only test once (or even a few times), you can mislead yourself into thinking you've cornered it. In my experience, the only thing I've found that keeps the error from occurring is to close the Explorer window open to imwrite's target folder, or to click back in the Explorer window to get out of the target folder. The latter obviously helps you get back to your data quickly once it's written to disk.
Steve replied on : 8 of 10