Video-based Motion Analysis with MATLAB
We love MATLAB and we also have many other interests, too. Today's guest blogger, Toshi Takeuchi, found an interesting way to combine his passion for MATLAB with one of his interests, Argentine Tango!
Contents
Motivation
I started taking Argentine Tango classes at MIT when we had a record snow fall a few winters ago in Boston. I was originally planning to learn snowboarding (I even bought a used snowboard) but who would want to spend more time in the snow after having shoveled it day after day?
I was drawn to Tango because I can do the figures once I understand the geometry and physics of the moves. I can't do other dances because I am dyslexic and can't follow steps. Perhaps that's why they say Tango tends to attract people with a technical background? Loren says the same goes for glass blowing.
Argentine Tango is an improvisational partner dance that incorporates a rich variety of elegant figures - watch this YouTube video for example. You have to practice those figures and you often watch videos like this - pausing, rolling back, playing back the same segment over and over - to study how those figures are executed. This is a tedious process often interrupted by a slow connection and annoying ads.
Naturally, I started wondering, "can I use MATLAB to watch videos at slower speed?" and of course you can. I am doing this for dancing, but I am sure you can also use it for other activities, such as checking your golf swing or learning how to play an instrument.
Loading Video
In this example, I will use my own video shot on an iPhone rather than professional videos you see on YouTube. I transferred the video from my phone to the current folder. If you want to follow this example, please use your own video. Now how do we load the video into MATLAB?
You can use the VideoReader object to read video files into MATLAB, and store each frame into a structure array. This will take a bit of time, but this is an important step in order to play back those frames in real time later. I show the progress bar while frames are extracted and then show the first frame when completed. If the orientation of your video is not correct, use imrotate to correct the orientation after extracting each frame.
filename = 'video-1468594243.mp4'; % video to load wb = waitbar(0,'Loading the file...'); % show progress bar v = VideoReader(filename); % create VideoReader object w = v.Width; % get width h = v.Height; % get height d = v.Duration; % get duration s = struct('cdata',zeros(h,w,3,'uint8'),'colormap',[]); % struct to hold frames k = 1; % initialize counter while hasFrame(v) % while object has frame f = readFrame(v); % read the frame s(k).cdata = f; % = imrotate(f,90); % rotate frame as needed k = k + 1; % increment counter waitbar(v.CurrentTime/d) % update progress bar end v.CurrentTime = 18; % set current time f = readFrame(v); % get the frame close(wb) % close progress bar axes % create axes imshow(f) % show the frame title('Final Tango Frame') % add title
Playing Back Video in 1/2 Time
You can now play back the video at the speed of your choice with imshow. To ensure real-time playback, I first create the image, and subsequently update its underlying data for each frame using its handle, because it is too slow to create a new image for each frame. You can only see the final frame here, so I added an animated GIF from the figure at the end of this post.
fps = v.FrameRate; % get frame rate startTime = 14; % in seconds endTime = 17; % in seconds speed = 1/2; % play back speed startFrame = floor(startTime * fps); % starting frame endFrame = floor(endTime * fps); % ending frame curAxes = axes; % create axes hImage = imshow(s(startFrame).cdata,'Parent',curAxes); % create hande title('Tango Video') % add title for k = startFrame + 1:endFrame % loop over others set(hImage,'CData',s(k).cdata); % update underlying data pause(1/(fps*speed)); % pause for specified speed end
Creating the GUI
It is cumbersome to run the script over and over each time I change settings like the starting frame or speed. It would be much easier if I built a GUI for it. When it comes to building a GUI in MATLAB, you have two options.
- App Designer - to create an app that runs on MATLAB
- GUIDE - to create a GUI that can be compiled as a standalone app
Since this could be useful for other Tango friends of mine, I would like to be able to share it. I will assume they generally don't have MATLAB. I would therefore go with GUIDE in this case. For details of how you create a GUI with GUIDE, please refer to this tutorial video.
Here is the fig file I created in GUIDE. You can open the fig file in GUIDE by typing guide in Command Window and selecting "Open Existing GUI" tab in the Quick Start window. To open the working GUI, you can press the green play button in GUIDE or run the tangoplayer callback function for the GUI in Command Window.
tangoplayer
"Choose File" Button
The callback function file tangoplayer.m was generated from GUIDE. I then added my custom code in it. The code we saw earlier for loading video is reused for the callback function for the "Choose File" button. Here is the relevant section of chooseBtn_Callback function.
dbtype 'tangoplayer.m' 408:431
408 % *** Begin Reuse Example Code ***
409 wb = waitbar(0,'Loading the file...'); % show progress bar
410 v = VideoReader(filename); % create VideoReader object
411 w = v.Width; % get width
412 h = v.Height; % get height
413 d = v.Duration; % get duration
414 s = struct('cdata',zeros(h,w,3,'uint8'),'colormap',[]); % struct to hold frames
415 k = 1; % initialize counter
416 while hasFrame(v) % while object has frame
417 if rotate == 0 % if no rotation
418 s(k).cdata = readFrame(v); % read the frame
419 else % otherwise
420 f = readFrame(v); % get the frame
421 s(k).cdata = imrotate(f,handles.rotate); % rotate it
422 end
423 k = k + 1; % increment counter
424 waitbar(v.CurrentTime/d) % update progress bar
425 end
426 v.CurrentTime = 0; % back to the beginning
427 f = readFrame(v); % get the frame
428 close(wb) % close progress bar
429 axes(handles.viewer) % get axes
430 hImage = imshow(imrotate(f,rotate)); % show the frame
431 % *** End Reuse Example Code ***
We want to share the resulting VideoReader object and structure array that contains extracted frames across other callback functions. You can do this by adding those variables to the handles structure and store it in the GUI's application data using guidata(hObject,handles).
dbtype 'tangoplayer.m' 437:443
437 % share the loaded data across the GUI app with handles 438 handles.v = v; % add v to handles 439 handles.s = s; % add s to handles 440 handles.t = 0; % add t to handles 441 handles.k = 0; % add k to handle 442 handles.fin = false; % add fin to handle 443 guidata(hObject,handles) % update handles
"Play" Button
To use the stored application data, you just have to access the relevant fields in the handles structure.
dbtype 'tangoplayer.m' 191:193
191 if ismember('v',fieldnames(handles)) % if VideoReader object exists
192 v = handles.v; % retrieve it
193 s = handles.s; % get frames
You can also retrieve, in the same way, the values for the start time and the speed set by the sliders. I invite you to look at the source code to see how the callback functions for those sliders work.
dbtype 'tangoplayer.m' 201:204
201 fps = v.FrameRate; % get frame rate 202 speed = str2num(get(handles.speedBox,'String')); % get speed 203 startTime = handles.t; % get start time 204 startFrame = handles.k; % get start frame
The actual code that plays back the video is similar to the earlier example except that it has additional lines to take care of some details related to the GUI such as updating the current time display, adding a context menu to the image, and checking for the button state, etc. The context menu adds the ability to draw lines, rectangles, and circles on the image as defined in another section of the file.
dbtype 'tangoplayer.m' 214:225
214 % *** Begin Reuse Example Code ***
215 set(handles.timeBox,'String',sprintf('%.4f',startTime)) % update box text
216 hImage = imshow(s(startFrame).cdata, ... % create hande and
217 'Parent',handles.viewer); % show the first frame
218 ctm = handles.ctm; % get context menu
219 set(hImage,'uicontextmenu',ctm) % add context menu
220
221 for k = startFrame + 1:length(s) % for each remaining frame
222 if get(hObject,'Value') % if button is down
223 set(hImage,'CData',s(k).cdata); % update underlying data
224 pause(1/(fps*speed)); % pause for specified speed
225 % *** End Reuse Example Code ***
TangoPlayer in Action
Here is the app in action - the lines were added using the context menu. It shows that I didn't keep my back straight in that particular frame. Ouch - it is painful to watch your own video, but it is more important to know the issues you need to fix.
Creating a Standalone App
Now that the app is working in MATLAB, I would like to share that with other Tango friends of mine who may or may not have MATLAB. If you have the MATLAB Compiler, you will find the Application Compiler under the Apps tab.
Open the Application Compiler and add tangoplayer.m as the main file and the Application Compiler will automatically add tangoplayer.fig and other dependencies in "Files required for your application to run". You can also make other customizations such as adding icons and a splash screen. Then all you need to do is to click "Package" to compile an executable. That's it! Since I am running MATLAB on Windows, this compiles a Windows app.
Once completed, you find your installer in the "for_redistribution" folder and that's what I would give to my friends. The app requires MATLAB Runtime, and the installer automatically downloads it from the web in the installation process.
Summary
Learning Tango takes many hours, but you can create a custom app in MATLAB and share it as a standalone application in a mere tiny fraction of that time!
Have you used MATLAB for your hobbies before? What did you create? Did you know that you can buy MATLAB Home to support your MATLAB addiction? Let us know what you've created here.
