I'd like to welcome guest blogger and ace MATLAB training content developer Matt Tearle for today's post. Thanks, Matt!

Contents

An Unusual Assignment

What do you do if you don't have an image processing background and your boss asks you to produce a music video for a MATLAB-vs-Simulink rap battle? It's probably not a question you've given much thought to. I admit that I hadn't either. But that's the awkward situation I found myself in a little while back.

A MATLAB-vs-Simulink rap battle -- obviously I had to be a part of this project! There was just one significant problem: I'm not a video producer. Yes, I have a nice digital camera. I have some very basic editing software. And I certainly have some... "creative" ideas. But I don't have any software designed to do serious video editing.

In particular, as soon as the concept was explained to me, I decided that we needed to do a "green screen" video:

[This background may not seem particularly "entertaining", but if I showed the real one we used, lawyers might get involved... And nobody wants that.]

MATLAB to the Rescue

With no purpose-made software to achieve this, and only a few days before it was due (thanks, boss!), I turned to my default standby: MATLAB!

A video is just a sequence of images, so all I needed to do was to write MATLAB code to "green-screen" two images, then read in two videos and apply my algorithm frame-by-frame. A quick hunt around in the documentation, and I had the framework code ready to go:

% Open the input video files
v1 = VideoReader('background.mp4');
v2 = VideoReader('foreground.mp4');
% Determine the number of frames for the final video
nFrames = min(v1.NumberOfFrames,v2.NumberOfFrames);
% Open a video file to write the result to
vOut = VideoWriter('mixedvideo.mp4','MPEG-4');
vOut.FrameRate = 24;
open(vOut);
% Loop over all the frames
for k = 1:nFrames
    % Get the kth frame of both inputs
    x = read(v1,k);
    y = read(v2,k);
    % Mix them together
    z = magicHappensHere(x,y);  % TODO: Write this...
    % Write the result to file
    writeVideo(vOut,z)
end
% And we're done!
close(vOut);

Now all I needed to do was write the magic function that would do the greenscreening...

Real Life is Complicated

Actually, I soon discovered that I needed to do a bit more preprocessing first. I had created the background video on my computer, but filmed the foreground on my digital camera. Of course the dimensions didn't match... No worries! That's what Image Processing Toolbox is for. I added a line of code to define the desired dimensions of the output video. Then imresize took care of the rest:

% Loop over all the frames
for k = 1:nFrames
    % Get the kth frame of both inputs
    x = imresize(read(v1,k),outDims);
    y = imresize(read(v2,k),outDims);
    % Mix them together
    z = magicHappensHere(x,y);
    % Write the result to file
    writeVideo(vOut,z)
end

Wow, this image processing stuff is easy! What's next, then?

Developing a Green-Screen Algorithm

The first thing to try, of course, when developing an algorithm is to see if one already exists. Sadly, searching the Image Processing Toolbox documentation didn't turn up a greenscreen function or anything I could see that would do the job. So it looks like I have to do this from scratch.

If I can identify the green pixels, then I can just replace those pixels in the foreground image with the corresponding pixels in the background image. So all of this really just boils down to: how do I find the green pixels? I know that a true-color image in MATLAB is an m-by-n-by-3 array, where each of the three m-by-n "planes" represent red, green, and blue intensity, respectively:

"Green" would, ideally, be [0, 255, 0] (in uint8). So a simple way to find the green pixels would be to do some kind of thresholding:

isgreen = (y(:,:,1) < 175) & (y(:,:,2) > 200) & (y(:,:,3) < 175);

This didn't work too badly, but neither was it great. One of the problems is the use of magic threshold values -- three of them, no less! Playing around with three magic parameters to try to find the right combination doesn't sound like fun to me. And besides, using absolute values like this might not be the best way to judge "greenness". If you look at the example image above, you can see that there's a fair amount of variation in the green background. (This is mainly because we were using an unevenly lit conference room that happened to have a green wall -- this was clearly a very professional production!)

At this point, I had to ask myself "what does it really mean to be 'green'?". As a human, I can easily distinguish between the various shades of green on the wall, despite the difference in brightness and shading. At first, this lead me into thinking about color spaces and how a color is a point in some 3-D space (RGB, CYM, HSV, etc.). Perhaps I could define "green" as "sufficiently close to [0 1 0] in RGB space". And so followed a few experiments with rounding, distance calculations, even playing with rgb2ind. But to no great success:

yd = round(double(y)/255);
isgreen = (yd(:,:,1)==0) & (yd(:,:,2)==1) & (yd(:,:,3)==0);

As this image shows, the problem with our makeshift green screen was that it wasn't particularly green -- more a yellow-green, with significant variations in brightness.

Just as I was started to get worried that this wasn't going to work at all, I realized that one obvious characteristic of the green pixels is that the green value is significantly greater than the blue and red values. This is, of course, embedded in my simple thresholding code above, but I can do away with at least some of the magic constants by looking at relative values. So how about calculating a "greenness" value from the three color channels:

yd = double(y)/255;
% Greenness = G*(G-R)*(G-B)
greenness = yd(:,:,2).*(yd(:,:,2)-yd(:,:,1)).*(yd(:,:,2)-yd(:,:,3));

Now that looks very promising. If I threshold that, I should get what I'm looking for. But rather than use an absolute threshold value maybe I can use something based on the distribution of the greenness values. In particular, I can exploit one nice feature of the images: they have a lot of green pixels. The histogram of greenness values looks like this:

I tried a simple thresholding based on the average greenness value, ignoring the negative values:

thresh = 0.3*mean(greenness(greenness>0));
isgreen = greenness > thresh;

Success! OK, so there's one magic constant left in there, but it was pretty quick and easy to tune, and seemed to be quite robust to minor changes. I'm calling that a good result. Now that I know where the green pixels are, I just have to replace them with the corresponding pixels from the background.

% Start with the foreground (essentially preallocation)
z = y;
% Find the green pixels
yd = double(y)/255;
% Greenness = G*(G-R)*(G-B)
greenness = yd(:,:,2).*(yd(:,:,2)-yd(:,:,1)).*(yd(:,:,2)-yd(:,:,3));
thresh = 0.3*mean(greenness(greenness>0));
isgreen = greenness > thresh;
% Blend the images
% Loop over the 3 color planes (RGB)
for j = 1:3
    rgb1 = x(:,:,j);  % Extract the jth plane of the background
    rgb2 = y(:,:,j);  % Extract the jth plane of the foreground
    % Replace the green pixels of the foreground with the background
    rgb2(isgreen) = rgb1(isgreen);
    % Put the combined image into the output
    z(:,:,j) = rgb2;
end

I love using logical indexing -- it's probably my favorite construct in the MATLAB language. Here I'm using it to replace the green pixels of the foreground (rgb2(isgreen)) with the corresponding background pixels (rgb1(isgreen)).

Going Further. Because I Can.

At this point I had a video that would have sufficed, but there was a noticeable green outline around my rappers. (It's not necessarily apparent in a single still frame.) It was only Sunday morning by this point, which meant that I still had a few hours left to tinker. Could I find a way to "shave" a pixel or two from the edge of the black silhouette in the above image of isgreen? To do that, I'd first need to find that edge. I don't know much about image processing, but I do know that "edge detection" is a thing that image people do, so it's time for me to search the doc... Sure enough, there's an edge function:

outline = edge(isgreen,'roberts');

Easy! Now if I can thicken that edge and combine it with the original isgreen, I'll have the slightly larger greenscreen mask that I need. Two more Image Processing Toolbox functions help me thicken the edge:

se = strel('disk',1);
outline = imdilate(outline,se);

The edge variable outline is a logical array, so I can easily combine it with isgreen, to get my final result:

isgreen = isgreen | outline;

Put it all together and I have a pretty simple script:

% Open the input video files
v1 = VideoReader('background.mp4');
v2 = VideoReader('foreground.mp4');
% Determine the number of frames for the final video
nFrames = min(v1.NumberOfFrames,v2.NumberOfFrames);
% Set the output dimensions
outDims = [400 640];
% Open a video file to write the result to
vOut = VideoWriter('mixedvideo.mp4','MPEG-4');
vOut.FrameRate = 24;
open(vOut);
% Loop over all the frames
for k = 1:nFrames
    % Get the kth frame of both inputs
    x = imresize(read(v1,k),outDims);
    y = imresize(read(v2,k),outDims);
    % Mix them together
    z = y;  % Preallocate space for the result
    % Find the green pixels in the foreground (y)
    yd = double(y)/255;
    % Greenness = G*(G-R)*(G-B)
    greenness = yd(:,:,2).*(yd(:,:,2)-yd(:,:,1)).*(yd(:,:,2)-yd(:,:,3));
    % Threshold the greenness value
    thresh = 0.3*mean(greenness(greenness>0));
    isgreen = greenness > thresh;
    % Thicken the outline to expand the greenscreen mask a little
    outline = edge(isgreen,'roberts');
    se = strel('disk',1);
    outline = imdilate(outline,se);
    isgreen = isgreen | outline;
    % Blend the images
    % Loop over the 3 color planes (RGB)
    for j = 1:3
        rgb1 = x(:,:,j);  % Extract the jth plane of the background
        rgb2 = y(:,:,j);  % Extract the jth plane of the foreground
        % Replace the green pixels of the foreground with the background
        rgb2(isgreen) = rgb1(isgreen);
        % Put the combined image into the output
        z(:,:,j) = rgb2;
    end
    % Write the result to file
    writeVideo(vOut,z)
end
% And we're done!
close(vOut);

The Moral of the Story

This is not the best greenscreen code ever written (although it is the best greenscreen code I've ever written). But the real point is that it shouldn't even exist -- I had no prior image processing knowledge, I was working with videos of different dimensions, the greenscreen was unevenly lit (and not even particularly green), and I had a weekend to make it happen. The fact that I was able to do this is one of the main reasons I love MATLAB (and, of course, the associated toolboxes): I was able to spend my time on the heart of my algorithm, not all the coding details. Different-sized images? Fixed with one function call (imresize). Need to find edges of a region? One function call (edge). Want to thicken that edge? Two function calls (strel and imdilate). The final script is less than 50 lines.

My time was spent wrestling with the core of the problem: how I was going to figure out what "green" looked like in my images. Because I could spend my time there, I was able to tinker with different ideas. This, to me, is what MATLAB is all about: rapid prototyping -- ideas becoming working code.

Post Script: Fortune Favors the Brave (and/or Crazy)

Having made the video and feeling pretty good about what I managed to pull together in a few hours of MATLAB, I later realized that I got lucky. The way I calculated greenness (G*(G-R)*(G-B)) was intended to give negative results for anything that didn't have more green than red or blue. But I simply overlooked the possibility of getting a high positive value if both G-R and G-B were negative. According to my formula, a light magenta -- e.g. [1 0.6 1] -- would be very green! Luckily for me, my source videos didn't include anything like that. A more robust approach would have been something like this:

yd = double(y)/255;
% Greenness = G*(G-R)*(G-B)
greenness = yd(:,:,2).*max(yd(:,:,2)-yd(:,:,1),0).*max(yd(:,:,2)-yd(:,:,3),0);

Now any pixel with more red or more blue than green will have a greenness of 0. Everything else works the same.

I think there's a nice moral in this postscript, as well. I love using MATLAB to solve a problem -- a real, immediate problem that I need to solve right now. I don't necessarily need the best solution, or the solution to the most general problem. The greenscreen algorithm I hacked together wasn't the best, but it worked for my application. If my image had light magenta in it, I would have discovered this bug and fixed it, but that didn't happen and didn't matter.

If I had approached this assignment like a professional software developer, I'd still be working on it. But as a MATLAB hacker, I was done in a weekend!

Get the MATLAB code

Published with MATLAB® R2014a