roifill design critique
"Is there anything else you would like to tell us?"
This was the question whose answer led me to revisit an old friend of mine, the roifill function. This is a function that smoothly fills a specified region of interest in a gray-scale image. I implemented it in the early 1990s following a hallway conversation with another MathWorks developer, who said to me something along the lines of "I bet you could fill an image region based on the equation for a soap film surface along a wire rim."
(I don't know when the term inpainting became common, but I certainly had not heard it way back then.)
Here's an example of roifill in action.
I = imread('rice.png'); imshow(I)
If you now call roifill, you can select a polyline around any object you like, and the function will remove the object and smoothly fill in the resulting hole. Here's a sample output (can you find the missing grain of rice?):
Anyway, fast-forward to just two weeks ago. A customer that I know was helping us with a usability test for a planned MATLAB feature. I was sitting in the observation room. After the test tasks were completed, the user experience expert asked a common question: Is there anything else you'd like to tell us about?
It turns out that this customer knows a great deal about the Image Processing Toolbox, and he had some interesting feedback about the roifill function and the mask image.
roifill, you see, takes a "mask" image that specifies the set of pixel values that are to be filled. At least, that's the way I should have designed it. Unfortunately, I didn't, and my design decision was causing a lot of confusion. That's because roifill actually only replaces the interior pixels of the mask.
Let me show you a diagram to explain what I mean. Suppose your fill mask looked like this:
It would be a very reasonable expectation to think that roifill would replace all the pixels corresponding to this mask. And that's what our intrepid customer thought should happen. But it actually replaces only the interior pixels of the mask. That is, it replaces the pixels marked in gray:
The pixels on the edge of the mask are used to establish boundary conditions for the fill equation. These pixels stay the same. Hence the confusion.
As I walked back to my office after the usability test, I tried to figure out why I would have originally designed the function this way. I think there were two reasons. First, I was much less experienced with designing image processing functions then. Second, I'm pretty sure I was only thinking about an interactive workflow. That is, a user interactively draws a region around an object using a mouse, and then roifill processes the result.
In the interactive workflow, it doesn't matter (very much) how the mask border pixels are handled. However, in a noninteractive workflow, I now think it really only makes sense for the mask to represent the set of "bad pixels" that all need to be replaced by the function.
Ah well. Lessons learned. I will suggest that the Image Processing Toolbox development team think about how to address this behavior quirk in a future release.
In the meantime, dear reader ... Is there anything else you would like to tell us?
Let us know by leaving a comment.
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.