Video-based Motion Analysis with MATLAB

We love MATLAB and we also have many other interests, too. Today's guest blogger, Toshi Takeuchi, found an interesting way to combine his passion for MATLAB with one of his interests, Argentine Tango!

Contents

Motivation

I started taking Argentine Tango classes at MIT when we had a record snow fall a few winters ago in Boston. I was originally planning to learn snowboarding (I even bought a used snowboard) but who would want to spend more time in the snow after having shoveled it day after day?

I was drawn to Tango because I can do the figures once I understand the geometry and physics of the moves. I can't do other dances because I am dyslexic and can't follow steps. Perhaps that's why they say Tango tends to attract people with a technical background? Loren says the same goes for glass blowing.

Argentine Tango is an improvisational partner dance that incorporates a rich variety of elegant figures - watch this YouTube video for example. You have to practice those figures and you often watch videos like this - pausing, rolling back, playing back the same segment over and over - to study how those figures are executed. This is a tedious process often interrupted by a slow connection and annoying ads.

Naturally, I started wondering, "can I use MATLAB to watch videos at slower speed?" and of course you can. I am doing this for dancing, but I am sure you can also use it for other activities, such as checking your golf swing or learning how to play an instrument.

Loading Video

In this example, I will use my own video shot on an iPhone rather than professional videos you see on YouTube. I transferred the video from my phone to the current folder. If you want to follow this example, please use your own video. Now how do we load the video into MATLAB?

You can use the VideoReader object to read video files into MATLAB, and store each frame into a structure array. This will take a bit of time, but this is an important step in order to play back those frames in real time later. I show the progress bar while frames are extracted and then show the first frame when completed. If the orientation of your video is not correct, use imrotate to correct the orientation after extracting each frame.

filename = 'video-1468594243.mp4';                          % video to load
wb = waitbar(0,'Loading the file...');                      % show progress bar
v = VideoReader(filename);                                  % create VideoReader object
w = v.Width;                                                % get width
h = v.Height;                                               % get height
d = v.Duration;                                             % get duration
s = struct('cdata',zeros(h,w,3,'uint8'),'colormap',[]);     % struct to hold frames
k = 1;                                                      % initialize counter
while hasFrame(v)                                           % while object has frame
    f = readFrame(v);                                       % read the frame
    s(k).cdata = f; %  = imrotate(f,90);                    % rotate frame as needed
    k = k + 1;                                              % increment counter
    waitbar(v.CurrentTime/d)                                % update progress bar
end
v.CurrentTime = 18;                                         % set current time
f = readFrame(v);                                           % get the frame
close(wb)                                                   % close progress bar
axes                                                        % create axes
imshow(f)                                                   % show the frame
title('Final Tango Frame')                                        % add title

Playing Back Video in 1/2 Time

You can now play back the video at the speed of your choice with imshow. To ensure real-time playback, I first create the image, and subsequently update its underlying data for each frame using its handle, because it is too slow to create a new image for each frame. You can only see the final frame here, so I added an animated GIF from the figure at the end of this post.

fps = v.FrameRate;                                          % get frame rate
startTime = 14;                                             % in seconds
endTime = 17;                                               % in seconds
speed =  1/2;                                               % play back speed
startFrame = floor(startTime * fps);                        % starting frame
endFrame = floor(endTime * fps);                            % ending frame
curAxes = axes;                                             % create axes
hImage = imshow(s(startFrame).cdata,'Parent',curAxes);      % create hande
title('Tango Video')                                        % add title
for k = startFrame + 1:endFrame                             % loop over others
    set(hImage,'CData',s(k).cdata);                         % update underlying data
    pause(1/(fps*speed));                                   % pause for specified speed
end

Creating the GUI

It is cumbersome to run the script over and over each time I change settings like the starting frame or speed. It would be much easier if I built a GUI for it. When it comes to building a GUI in MATLAB, you have two options.

Since this could be useful for other Tango friends of mine, I would like to be able to share it. I will assume they generally don't have MATLAB. I would therefore go with GUIDE in this case. For details of how you create a GUI with GUIDE, please refer to this tutorial video.

Here is the fig file I created in GUIDE. You can open the fig file in GUIDE by typing guide in Command Window and selecting "Open Existing GUI" tab in the Quick Start window. To open the working GUI, you can press the green play button in GUIDE or run the tangoplayer callback function for the GUI in Command Window.

tangoplayer

"Choose File" Button

The callback function file tangoplayer.m was generated from GUIDE. I then added my custom code in it. The code we saw earlier for loading video is reused for the callback function for the "Choose File" button. Here is the relevant section of chooseBtn_Callback function.

dbtype 'tangoplayer.m' 408:431

408   % ***  Begin Reuse Example Code ***
409   wb = waitbar(0,'Loading the file...');                  % show progress bar
410   v = VideoReader(filename);                              % create VideoReader object
411   w = v.Width;                                            % get width 
412   h = v.Height;                                           % get height
413   d = v.Duration;                                         % get duration
414   s = struct('cdata',zeros(h,w,3,'uint8'),'colormap',[]); % struct to hold frames
415   k = 1;                                                  % initialize counter
416   while hasFrame(v)                                       % while object has frame
417       if rotate == 0                                      % if no rotation
418           s(k).cdata = readFrame(v);                      % read the frame
419       else                                                % otherwise
420           f = readFrame(v);                               % get the frame
421           s(k).cdata = imrotate(f,handles.rotate);        % rotate it
422       end
423       k = k + 1;                                          % increment counter
424       waitbar(v.CurrentTime/d)                            % update progress bar
425   end
426   v.CurrentTime = 0;                                      % back to the beginning
427   f = readFrame(v);                                       % get the frame
428   close(wb)                                               % close progress bar
429   axes(handles.viewer)                                    % get axes
430   hImage = imshow(imrotate(f,rotate));                    % show the frame
431   % ***  End Reuse Example Code ***

We want to share the resulting VideoReader object and structure array that contains extracted frames across other callback functions. You can do this by adding those variables to the handles structure and store it in the GUI's application data using guidata(hObject,handles).

dbtype 'tangoplayer.m' 437:443

437   % share the loaded data across the GUI app with handles
438   handles.v = v;                                          % add v to handles
439   handles.s = s;                                          % add s to handles
440   handles.t = 0;                                          % add t to handles
441   handles.k = 0;                                          % add k to handle
442   handles.fin = false;                                    % add fin to handle
443   guidata(hObject,handles)                                % update handles

"Play" Button

To use the stored application data, you just have to access the relevant fields in the handles structure.

dbtype 'tangoplayer.m' 191:193

191   if ismember('v',fieldnames(handles))                    % if VideoReader object exists
192       v = handles.v;                                      % retrieve it
193       s = handles.s;                                      % get frames

You can also retrieve, in the same way, the values for the start time and the speed set by the sliders. I invite you to look at the source code to see how the callback functions for those sliders work.

dbtype 'tangoplayer.m' 201:204

201   fps = v.FrameRate;                                      % get frame rate
202   speed =  str2num(get(handles.speedBox,'String'));       % get speed
203   startTime = handles.t;                                  % get start time
204   startFrame = handles.k;                                 % get start frame

The actual code that plays back the video is similar to the earlier example except that it has additional lines to take care of some details related to the GUI such as updating the current time display, adding a context menu to the image, and checking for the button state, etc. The context menu adds the ability to draw lines, rectangles, and circles on the image as defined in another section of the file.

dbtype 'tangoplayer.m' 214:225

214   % ***  Begin Reuse Example Code ***
215   set(handles.timeBox,'String',sprintf('%.4f',startTime)) % update box text
216   hImage = imshow(s(startFrame).cdata, ...                % create hande and
217       'Parent',handles.viewer);                           % show the first frame
218   ctm = handles.ctm;                                      % get context menu
219   set(hImage,'uicontextmenu',ctm)                         % add context menu
220   
221   for k = startFrame + 1:length(s)                        % for each remaining frame
222       if get(hObject,'Value')                             % if button is down
223           set(hImage,'CData',s(k).cdata);                 % update underlying data
224           pause(1/(fps*speed));                           % pause for specified speed
225       % ***  End Reuse Example Code ***

TangoPlayer in Action

Here is the app in action - the lines were added using the context menu. It shows that I didn't keep my back straight in that particular frame. Ouch - it is painful to watch your own video, but it is more important to know the issues you need to fix.

Creating a Standalone App

Now that the app is working in MATLAB, I would like to share that with other Tango friends of mine who may or may not have MATLAB. If you have the MATLAB Compiler, you will find the Application Compiler under the Apps tab.

Open the Application Compiler and add tangoplayer.m as the main file and the Application Compiler will automatically add tangoplayer.fig and other dependencies in "Files required for your application to run". You can also make other customizations such as adding icons and a splash screen. Then all you need to do is to click "Package" to compile an executable. That's it! Since I am running MATLAB on Windows, this compiles a Windows app.

Once completed, you find your installer in the "for_redistribution" folder and that's what I would give to my friends. The app requires MATLAB Runtime, and the installer automatically downloads it from the web in the installation process.

Summary

Learning Tango takes many hours, but you can create a custom app in MATLAB and share it as a standalone application in a mere tiny fraction of that time!

Have you used MATLAB for your hobbies before? What did you create? Did you know that you can buy MATLAB Home to support your MATLAB addiction? Let us know what you've created here.

Get the MATLAB code

Published with MATLAB® R2016b