The Two Amigos
I'd like to introduce Brett Shoelson as guest blogger for today. Many of you already know Brett from his work on the File Exchange Pick of the Week blog, or his posts on the comp.soft-sys.matlab newsgroup, or from his many presentations given on the road. I met Brett "virtually" many years ago, long before he came to work for MathWorks, through his posts on the newsgroup. Brett's been doing fun, interesting, and even useful things to images using MATLAB for quite a long time.
A few weeks ago Brett posted the Two Amigos Challenge on the Pick of the Week blog. Readers of the blog know that long-time contributor Bob Bemis (Hi, Bob!) has moved on from MathWorks and isn't posting there anymore. Brett challenged his readers to digitally retouch the "Three Amigos" picture on the blog and turn it into a "Two Amigos" picture using only MathWorks products.
I was so impressed with Brett's own solution to the Challenge that I invited him to show it to my readers here.
So here is Brett's solution to the Two Amigos Challenge ...
... in which we remove a subject from a photograph, and digitally retouch it to conceal the modification...
- Get/display image
- Segmentation masks
- Decide on a color plane/colorspace
- The Three Amigos
- Now we can clean that up a bit
- Oops...lost Jiro's hair!
- Reconstruct Mask
- Get rid of the road and clean things up a bit
- Three Amigos in color
- Individual masks
- Three Amigos in pseudocolor
- Again, look at some options for subsequent segmentation spaces
- Finding Bob
- Now Bob's face...
- Reconstructing Bob
- Jiro and Brett...
- Jiro, and Brett
- Now let's shift the rgb images to close the "Bob gap"
- Now we can move Jiro and Brett into the space Bob once occupied
- Two Amigos, in color
- Add a background back in
- Finally, we drop Jiro and Brett into the new background
- One more bit of cleanup:
- Final result
url = 'https://blogs.mathworks.com/images/pick/threeamigos-800w.jpg'; original = imread(url); togglefig('Original') imshow(original);
NOTE: togglefig is a helper function I use often. If a figure with the specified title exists, activate and use it. Otherwise, create it.
My approach is going to start with creating a segmentation mask of Jiro, Bob, and Brett, and then creating individual segmentation masks for each.
Decide on a color plane/colorspace
First, let's explore the information in the R,G, and B planes individually, as well as in the HSV, YCbCr, and LAB color spaces.
Here's a link to exploreRGB.
The Three Amigos
Let's segment the Three Amigos in the LAB colorspace
cform = makecform('srgb2lab'); % A first pass at noise reduction using a median filter, operating plane-by-plane: for ii = 1:3 original(:,:,ii) = medfilt2(original(:,:,ii),[6 6]); end lab = applycform(original,cform); % Now we're going to threshold the "a*" colorspace: a = lab(:,:,2); threeAmigos = im2bw(a,0.5); togglefig('Three Amigos') imshow(threeAmigos)
Now we can clean that up a bit
threeAmigos = imfill(threeAmigos,'holes'); % ...calculate connected components... cc = bwconncomp(threeAmigos); % ...calculate the areas of all blobs in the image... stats = regionprops(cc,'Area'); %...and keep only the largest: A = [stats.Area]; [~,biggest] = max(A); threeAmigos(labelmatrix(cc)~=biggest) = 0; togglefig('Three Amigos') imshow(threeAmigos)
Oops...lost Jiro's hair!
That did a reasonable job of segmenting the Three Amigos. But I seem to have lost Jiro's hair, and he might want it back. Let's go back and find it....
jirosHair = original(:,:,3) <= 25.5; % Get rid of "small" (defined here as <= 1050-pixel) blobs jirosHair = bwareaopen(jirosHair,1050); togglefig('Jiro''s Hair') imshow(jirosHair)
Now we can reconstruct the Three Amigos, in all their hairy (or hairless) glory
% Binary OR: threeAmigos = threeAmigos | jirosHair; togglefig('Three Amigos') imshow(threeAmigos)
Get rid of the road and clean things up a bit
...by morphologically opening, then closing, the image:
threeAmigos = imopen(threeAmigos,strel('disk',19)); threeAmigos = imerode(threeAmigos,strel('disk',1)); togglefig('Three Amigos') imshow(threeAmigos)
Three Amigos in color
Mask the background; here we set to zero all pixels outside of the threeAmigos mask using logical indexing:
amigosColor = original; r = amigosColor(:,:,1); r(~threeAmigos)= 0; g = amigosColor(:,:,2); g(~threeAmigos)= 0; b = amigosColor(:,:,3); b(~threeAmigos)= 0; % Concatenate in the "3rd dimension" to re-constitute an RGB image. amigosColor = cat(3,r,g,b); togglefig('Amigos in Color') imshow(amigosColor)
Now we can concentrate on segmenting individual masks
Three Amigos in pseudocolor
Let's exaggerate the differences to make this easier
amigosColor = decorrstretch(amigosColor); togglefig('Amigos in Pseudocolor') imshow(amigosColor)
Again, look at some options for subsequent segmentation spaces
Let's again work in the LAB colorspace
lab2 = applycform(amigosColor,cform); % This time, we will use the "b* colorspace" b2 = lab2(:,:,3); % Segment Bob: bobBodyHair = ~im2bw(b2,0.47); bobBodyHair = imclose(bobBodyHair,strel('disk',3)); bobBodyHair = bwareaopen(bobBodyHair,50); %Bob is in the middle; let's keep only the two "middle" regions (let's say, %within 30 pixels of the center) cc = bwconncomp(bobBodyHair); stats = regionprops(cc,'Centroid'); C = reshape([stats.Centroid],2,)'; inds = find(C(:,1) > size(bobBodyHair,2)/2 - 30 & C(:,1) < size(bobBodyHair,2)/2 + 30); bobBodyHair = ismember(labelmatrix(cc), inds); togglefig('Finding Bob') imshow(bobBodyHair)
Now Bob's face...
Still working with "b*," but using a different threshold:
bobFace = im2bw(b2,0.56); bobFace = imclearborder(bobFace); cc = bwconncomp(bobFace); stats = regionprops(cc,'Area'); A = [stats.Area]; [~,biggest] = max(A); bobFace(labelmatrix(cc)~=biggest) = 0; togglefig('Finding Bob') imshow(bobFace)
bob = bobBodyHair | bobFace; bob = imfill(bob,'holes'); togglefig('Finding Bob') imshow(bob)
Jiro and Brett...
jiroBrett = threeAmigos & ~bob; jiroBrett = bwareaopen(jiroBrett,300); togglefig('Jiro and Brett') imshow(jiroBrett)
Jiro, and Brett
The connected components matrix is very convenient here:
cc = bwconncomp(jiroBrett); jiro = labelmatrix(cc) == 1; brett = labelmatrix(cc) == 2; togglefig('Jiro') imshow(jiro); title('Jiro') togglefig('Brett') imshow(brett) title('Brett')
Now let's shift the rgb images to close the "Bob gap"
How "wide" is Bob at the bottom of the image? (Plus a bit of padding.)
shiftAmt = floor(nnz(bob(end,:))/2) + 7;
Now we can move Jiro and Brett into the space Bob once occupied
twoAmigos = circshift(jiro,[0 shiftAmt]) | circshift(brett,[0,-shiftAmt]); togglefig('Two Amigos'); imshow(twoAmigos);
Two Amigos, in color
We can now create a color image that excludes Bob
% (We re-read the original because we reduced noise in it above using median % filtering, prior to masking. Now we want the unsmoothed original back.) original = imread(url); rJiro = original(:,:,1); rJiro(~jiro) = 0; rJiro = circshift(rJiro,[0 shiftAmt]); gJiro = original(:,:,2); gJiro(~jiro) = 0; gJiro = circshift(gJiro,[0 shiftAmt]); bJiro = original(:,:,3); bJiro(~jiro) = 0; bJiro = circshift(bJiro,[0 shiftAmt]); rBrett = original(:,:,1); rBrett(~brett) = 0; rBrett = circshift(rBrett,[0 -shiftAmt]); gBrett = original(:,:,2); gBrett(~brett) = 0; gBrett = circshift(gBrett,[0 -shiftAmt]); bBrett = original(:,:,3); bBrett(~brett) = 0; bBrett = circshift(bBrett,[0 -shiftAmt]); r = max(rJiro,rBrett); g = max(gJiro,gBrett); b = max(bJiro,bBrett); rgb = cat(3,r,g,b); togglefig('Two Amigos Color') imshow(rgb)
Add a background back in
Rather than try to "fix" the original background, I'm going to select a new backdrop for the "Two Amigos": ('forest.tif' is a sample image in the Image Processing Toolbox.)
[bg,map] = imread('forest.tif'); bg = ind2rgb(bg,map); % Resize it... bg = imresize(bg,size(r,1)/size(bg,1),'bicubic'); bg = bg(:,1:size(r,2),:); bg = im2uint8(bg); togglefig('(New) Background') imshow(bg)
Finally, we drop Jiro and Brett into the new background
bgR = bg(:,:,1); bgR(twoAmigos) = 0; bgG = bg(:,:,2); bgG(twoAmigos) = 0; bgB = bg(:,:,3); bgB(twoAmigos) = 0; r = max(r,bgR); g = max(g,bgG); b = max(b,bgB); rgb = cat(3,r,g,b); togglefig('Two Amigos Color') imshow(rgb)
One more bit of cleanup:
The edge is a bit rough...
brettJiroEdge = bwperim(twoAmigos); brettJiroEdge = imdilate(brettJiroEdge,strel('disk',9)); togglefig('Two Amigos Outline') imshow(brettJiroEdge)
Soften the edges: we can soften those edges a bit by applying locally an averaging filter where the (dilated) outline is logically true:
H = fspecial('average'); % Roifilt2 is a 2-D filter. We can use it on RGB images, but we have to % operate plane-by-plane. But first, it's good practice to preallocate the % final matrix: FINAL = zeros(size(rgb),'uint8'); for ii = 1:3 FINAL(:,:,ii) = roifilt2(H,rgb(:,:,ii),brettJiroEdge); end FINAL = imcrop(FINAL,[58 90 691 471]); togglefig('Final Result') imshow(FINAL) % et voila!
To leave a comment, please click here to sign in to your MathWorks Account or create a new one.