\r\n\r\n

This post is from Brian Douglas<\/a>, YouTube Content Creator for Control Systems and Deep Learning Applications<\/em><\/p>\r\n\r\n

For about a decade, I've wanted to implement this silly idea I had of\r\n measuring the acceleration of a person's hand to count the number of times\r\n they high five throughout the day. \r\n\r\nI wasn't sure how to accomplish this\r\n using the rule-based approaches to algorithm development that I was familiar\r\n with and so the project sat on hold. It was only while I was making the\r\n MATLAB Tech Talk video series on\r\n Deep Learning<\/a> that I realized that Deep Learning was perfect for\r\n solving this problem!<\/p>\r\n\r\n

The topic for the 4th video in the series was transfer learning and it turned out that\r\n was the key concept that I needed for me to quickly get a high five counting\r\n algorithm up and running. In this blog post, I will walk through the details\r\n of the code I wrote and the tools I used to get my high five counter working for that video. Hopefully, you\r\n can use this as a starting point to solve those difficult classification problems that\r\n you've been sitting on for the last 10 years as well.<\/p>\r\n

This post is divided into the following sections:<\/p>\r\n\r\n

An overview of the hardware<\/a><\/li>\r\n

Reading the accelerometer in MATLAB<\/a><\/li>\r\n

Data preprocessing and the scalogram<\/a><\/li>\r\n

Creating the training data<\/a><\/li>\r\n

Transfer learning and GoogLeNet<\/a><\/li>\r\n

Training the network<\/a><\/li>\r\n

Testing the high five counter<\/a><\/li>\r\n<\/u1>\r\n
<\/h6>\r\n
So, let's get to it!<\/p>\r\n\r\n <\/a>
An Overview of the Hardware<\/h2>\r\n\r\n
The hardware set up is pretty simple. I have an accelerometer that is\r\n connected to an \r\n Arduino Uno<\/a> through an I2C bus. The Arduino is then connected to my\r\n computer through USB.<\/p>\r\n\r\n
$\"Schematic\"$ <\/p>\r\n\r\n
To sense acceleration, I'm using the \r\n MPU-9250<\/a>. It's a 9 degree of\r\n freedom inertial measurement unit from TDK InvenSense. Instead of\r\n integrating the chip into my own custom circuit design, I am using a\r\n \r\n breakout board<\/a> that exposes power, ground, and the I2C communication pins.\r\n The only reason I am using this particular chip is because I already had one lying around, but any\r\n accelerometer will work as long as it's small enough to be moved around\r\n quickly by hand.<\/p>\r\n\r\n
You can see that my hardware setup was pretty crudely constructed\r\n with a breadboard and some jumper wires but I think it's kind of\r\n nice that you don't need to set up anything too fancy for this to work.<\/p>\r\n\r\n
$\"Arduino$ <\/p>\r\n\r\n <\/a>
Reading the Accelerometer in MATLAB <\/h2>\r\n\r\n
To read acceleration from the MPU-9250 through the Arduino, I'm using\r\n the \r\n MATLAB Support Package for Arduino Hardware<\/a>. This package allows\r\n you to communicate with an Arduino without having to compile code for\r\n it. Plus, there is a built in\r\n \r\n mpu9250<\/a> function that allows you to read the sensor with a one-line command.<\/p>\r\n\r\n\r\n
It only takes three lines of code to connect to the Arduino, instantiate an\r\n MPU9250 object, and read the accelerometer.<\/p>\r\n\r\n
$\"Reading$ <\/p>\r\n\r\n <\/a>
Data Preprocessing and the Scalogram<\/h2>\r\n\r\n
If you watched the 4th video in the Tech Talk series on deep learning, you'll\r\n know that I opted to convert the 3-axis acceleration data into an image to take\r\n advantage of \r\n GoogLeNet<\/a> - a network trained to recognize images. In particular,\r\n I used a \r\n continuous wavelet transform<\/a> to create a\r\n \r\n scalogram<\/a>. <\/p>\r\n\r\n
A scalogram is a time-frequency representation that is suitable for signals\r\n that exist at multiple scales. That is, signals that are low frequency and\r\n slowly varying, but then are occasionally interrupted with high frequency\r\n transients. It turns out they are useful for visualizing acceleration data\r\n for the occasional high frequency high five within an otherwise slowly moving\r\n hand.<\/p>\r\n\r\n
$\"High$ <\/p>\r\n\r\n
A cleaned-up version of the MATLAB code that I used to make the above plot is in\r\n the following collapsible block.<\/p>\r\n\r\n
\r\n
\r\n
\r\n \r\n Click to Expand Code<\/a>
\r\n <\/h4>\r\n<\/div>\r\n
\r\n
\r\n
close all\r\nclear\r\n\r\n % If your computer is not able to run this real-time, reduce the sample <\/span>\r\n% rate or comment out the scalogram part<\/span>\r\nfs = 50; % Run at 50 Hz<\/span>\r\n\r\na = arduino('COM3', 'Uno', 'Libraries', 'I2C'); % Change to your arduino<\/span>\r\nimu = mpu9250(a);\r\n\r\nbuffer_length_sec = 2; % Seconds of data to store in buffer<\/span>\r\naccel = zeros(floor(buffer_length_sec * fs) + 1, 3); % Init buffer<\/span>\r\n\r\nt = 0:1\/fs:(buffer_length_sec(end)); % Time vector<\/span>\r\n\r\nsubplot(2, 1, 1)\r\nplot_accel = plot(t, accel); % Set up accel plot<\/span>\r\naxis([0, buffer_length_sec, -50, 50]);\r\n\r\nsubplot(2, 1, 2)\r\nplot_scale = image(zeros(224, 224, 3)); % Set up scalogram<\/span>\r\n\r\ntic % Start timer<\/span>\r\nlast_read_time = 0;\r\n\r\ni = 0;\r\n% Run for 20 seconds<\/span>\r\nwhile(toc <= 20)\r\n current_read_time = toc;\r\n if (current_read_time - last_read_time) >= 1\/fs\r\n i = i + 1;\r\n\r\n accel(1:end-1, :) = accel(2:end, :); % Shift values in FIFO buffer<\/span>\r\n accel(end, :) = readAcceleration(imu);\r\n\r\n plot_accel(1).YData = accel(:, 1);\r\n plot_accel(2).YData = accel(:, 2);\r\n plot_accel(3).YData = accel(:, 3);\r\n\r\n % Only run scalogram every 3rd sample to save on compute time<\/span>\r\n if mod(i, 3) == 0\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 1);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i = imresize(cfs_abs\/8, [224 224]);\r\n\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 2);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i(:, :, 2) = imresize(cfs_abs\/8, [224 224]);\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 3);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i(:, :, 3) = imresize(cfs_abs\/8, [224 224]);\r\n\r\n if~(isempty(accel_i(accel_i>1)))\r\n accel_i(accel_i>1) = 1;\r\n end\r\n\r\n plot_scale.CData = accel_i;\r\n end\r\n\r\n last_read_time = current_read_time;\r\n end\r\nend<\/pre>\r\n<\/div><\/div><\/div>\r\n\r\n
Note that this code uses a function called \r\n cwtfilterbank<\/a> to create the\r\nscalogram which is part of the \r\n Wavelet Toolbox<\/a>. If you don't have access\r\nto this toolbox and you don't want to write the code yourself, try giving\r\nanother type of time-frequency visualization a shot. Maybe a\r\n spectrogram<\/a>\r\nwill work or some other algorithm that you come up with. Whatever you choose,\r\nthe idea here is that we're trying to create an image that will make the\r\nunique and identifiable features of a high five pattern obvious. I have shown that a\r\nscalogram works, but other methods may work as well.<\/p>\r\n\r\n <\/a>
Creating the Training Data<\/h2>\r\n\r\n
To train a network to recognize high fives, we need multiple examples of what\r\n a high five looks like and what a high five doesn't look like. Since we will\r\n be starting from a pre-trained network, we won't need as many training examples\r\n as we would if we were training a network from scratch. I don't know exactly\r\n how much training data is needed to fully capture the solution space for all\r\n possible high fives, however, I collected data for 100 high fives\r\n and 100 non-high fives and that seemed to work pretty well. I suspect I could\r\n have gotten away with less for the video I was making, but I think if I was\r\n really creating a product I would have used a lot more examples. You can\r\n play around with the amount of labeled training data and see how it affects the result.<\/p>\r\n\r\n
Collecting 200 images seems like a lot of work, but I wrote a script that\r\n cycles through them one after another and saves the images in the appropriate\r\n folder. I ran the following script twice; once with the 'high_five' label\r\n with the images being saved to the data\/high_five folder<\/em> and once with the\r\n 'no_high_five' label with the images being saved to the data\/no_high_five<\/em> folder.\r\n<\/p>\r\n\r\n
\r\n\r\n
\r\n \r\n Click to Expand Code<\/a>
\r\n <\/h4>\r\n<\/div>\r\n
\r\n
\r\n
% This script collects training data and places it in the specified<\/span>\r\n% label subfolder. 3 seconds of data is collected from the<\/span>\r\n% sensor but only keeps and saves off the last 2 seconds.<\/span>\r\n% This gives the user some buffer time to start the high five.<\/span>\r\n\r\n% The program pauses between images and prompts the user to continue.<\/span>\r\n\r\n% Note, you'll want to move the figure away from the MATLAB window so that<\/span>\r\n% you can see the acceleration after you respond to the wait prompt.<\/span>\r\n\r\nclose all\r\nclear all\r\n\r\n% If your computer is not able to run this real-time, reduce the sample rate<\/span>\r\nfs = 50; % Run at 50 Hz<\/span>\r\n\r\nparentDir = pwd;\r\ndataDir = 'data';\r\n\r\n%% Set the label for the data that you are generating<\/span>\r\n% labels = 'no_high_five';<\/span>\r\nlabels = 'high_five';\r\n\r\na = arduino('COM3', 'Uno', 'Libraries', 'I2C'); % Change to your arduino<\/span>\r\nimu = mpu9250(a);\r\n\r\nbuffer_length_sec = 2; % Seconds of data to store in buffer<\/span>\r\naccel = zeros(floor(buffer_length_sec * fs) + 1, 3); % Init buffer<\/span>\r\n\r\nt = 0:1\/fs:(buffer_length_sec(end)); % Time vector<\/span>\r\n\r\nsubplot(2, 1, 1)\r\nplot_accel = plot(t, accel); % Set up accel plot<\/span>\r\naxis([0 buffer_length_sec -50 50]);\r\n\r\nsubplot(2, 1, 2)\r\nplot_scale = image(zeros(224, 224, 3)); % Set up scalogram<\/span>\r\n\r\nfor j = 1:100 % Collect 100 images<\/span>\r\n\r\n % Prompt user to be ready to record next high five<\/span>\r\n H = input('Hit enter when ready: ');\r\n\r\n tic % Start timer<\/span>\r\n last_read_time = 0;\r\n\r\n i = 0;\r\n % Run for 3 seconds<\/span>\r\n while(toc <= 3)\r\n current_read_time = toc;\r\n if (current_read_time - last_read_time) >= 1\/fs\r\n i = i + 1;\r\n\r\n accel(1:end-1, :) = accel(2:end, :); % Shift values in buffer<\/span>\r\n accel(end, :) = readAcceleration(imu);\r\n\r\n plot_accel(1).YData = accel(:, 1);\r\n plot_accel(2).YData = accel(:, 2);\r\n plot_accel(3).YData = accel(:, 3);\r\n\r\n % Run scalogram every 3rd sample<\/span>\r\n\r\n if mod(i, 3) == 0\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 1);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i = imresize(cfs_abs\/8, [224 224]);\r\n\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 2);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i(:, :, 2) = imresize(cfs_abs\/8, [224 224]);\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 3);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i(:, :, 3) = imresize(cfs_abs\/8, [224 224]);\r\n\r\n if~(isempty(accel_i(accel_i>1)))\r\n accel_i(accel_i>1) = 1;\r\n end\r\n\r\n plot_scale.CData = accel_i;\r\n end\r\n\r\n last_read_time = current_read_time;\r\n end\r\n end\r\n\r\n % Save image to data folder<\/span>\r\n imageRoot = fullfile(parentDir,dataDir);\r\n imgLoc = fullfile(imageRoot,char(labels));\r\n imFileName = strcat(char(labels),'_',num2str(j),'.jpg');\r\n\r\n imwrite(plot_scale.CData, fullfile(imgLoc,imFileName), 'JPEG');\r\nend<\/pre>\r\n<\/div><\/div><\/div>\r\n\r\n
After running the script, I manually went through my training data and removed\r\n images that I thought would corrupt the training. These were images where the\r\n high five wasn't in the middle of the frame or images where I knew that I did\r\n a poor high five motion. In the below gif, I deleted high five image 49 because\r\n it wasn't in the center of the frame.\r\n<\/p>\r\n\r\n
$\"Deleting$ <\/p>\r\n\r\n <\/a>
Transfer Learning and GoogLeNet <\/h2>\r\n\r\n
With all of my training data in their appropriate folders, the next step is\r\n to set up the network. For this part, I was following along with the MATLAB\r\n example \r\n Classify Time Series Using Wavelet Analysis and Deep Learning<\/a>, except,\r\n rather than run everything through a MATLAB script, I found it easier to set up and\r\n train the network using the \r\n Deep Network Designer<\/a> app. <\/p>\r\n\r\n
I started from the pre-trained GoogLeNet to\r\n take advantage of all of the knowledge this network has for recognizing objects\r\n in images. GoogLeNet was trained to recognize things like fish and hotdogs in\r\n images - clearly not what I'm looking for - but this is where transfer learning\r\n is useful. With transfer learning, I can keep much of the existing network in\r\n place and only replace two layers at the end of the network which\r\n combine those generic features into the specific patterns I'm looking for. Then\r\n when I retrain the network, it's pretty much just these two layers that need\r\nto be trained, which is why training is so much faster with transfer learning. <\/p>\r\n\r\n
$\"Replacing$ <\/p>\r\n\r\n
I recommend you follow along with the MATLAB example I used or watch the Tech\r\n Talk if you'd like to know exactly how I replaced the layers and what training\r\n parameters I used, however, this again is a great place for you to experiment\r\n with something different. You can try to start from a different pretrained\r\n network like SqueezeNet, or you can replace more layers in GoogLeNet, or\r\n change the training parameters. There are a lot of options here and I think\r\n deviating from what I did can help you develop some intuition for how all of\r\n these variables affect the result.\r\n<\/p>\r\n\r\n <\/a>
Training the Network<\/h2>\r\n\r\n
With the network ready to go, training in the Deep Network Designer App is\r\n very simple. In the data tab, I imported the training data by selecting the\r\n folder where I saved the set of high five and no high five images. I also set\r\n aside 20 percent of the images to be used for validation during the training\r\n process. <\/p>\r\n\r\n
$\"Replacing$ <\/p>\r\n\r\n
Then in the training tab, I set my training options. Here, I was using the\r\n same options that were used in the MATLAB example that I was following, however, once again I\r\n encourage you to play around with some of these values and see how they impact\r\n the results.<\/p>\r\n\r\n
$\"\"$ <\/p>\r\n\r\n
\r\n Training took just over 4 minutes on my single CPU and reached about 97%\r\n validation accuracy. Not too bad for a couple hours of work!<\/p>\r\n\r\n
$\"\"$ <\/p>\r\n\r\n <\/a>
Testing the High Five Counter <\/h2>\r\n
Now that I have a trained network, I use the function\r\n \r\n classify<\/a> from the\r\n Deep Learning Toolbox to pass in the scalogram at each sample time and\r\n have the network return a label. If the returned label was \"high_five\"\r\n I increment a counter. To keep from counting the same high five several\r\n times as the acceleration data streaked across the whole buffer, I added\r\n a timeout that would not count a new high five unless it had been at\r\n least 2 seconds since the previous high five. <\/p>\r\n\r\n
Below is a cleaned up version of the code that I used to count high fives.<\/p>\r\n\r\n
\r\n
\r\n
\r\n \r\n Click to Expand Code<\/a>
\r\n <\/h4>\r\n<\/div>\r\n
\r\n
\r\n
close all\r\nclear\r\n\r\n%% Update to the name of your trained network<\/span>\r\nload trainedGN\r\ntrainedNetwork = trainedGN;\r\n\r\n% If your computer is not able to run this real-time, reduce the sample<\/span>\r\n% rate or comment out the scalogram part<\/span>\r\nfs = 50; % Run at 50 Hz<\/span>\r\n\r\na = arduino('COM3', 'Uno', 'Libraries', 'I2C'); % Change to your arduino<\/span>\r\nimu = mpu9250(a);\r\n\r\nbuffer_length_sec = 2; % Seconds of data to store in buffer<\/span>\r\naccel = zeros(floor(buffer_length_sec * fs) + 1, 3); % Init buffer<\/span>\r\n\r\nt = 0:1\/fs:(buffer_length_sec(end)); % Time vector<\/span>\r\n\r\n% Set up plots<\/span>\r\nh = figure;\r\nh.Position = [100 100 900 700];\r\np1 = subplot(2, 1, 1);\r\nplot_accel = plot(t, accel);\r\nplot_accel(1).LineWidth = 3;\r\nplot_accel(2).LineWidth = 3;\r\nplot_accel(3).LineWidth = 3;\r\np1.FontSize = 20;\r\np1.Title.String = 'Acceleration';\r\naxis([0 t(end) -50 60]);\r\nxlabel('Seconds');\r\nylabel('Acceleration, mpss');\r\ngrid on;\r\nlabel_string = text(1.3, 45, 'No High Five');\r\nlabel_string.Interpreter = 'none';\r\nlabel_string.FontSize = 25;\r\ncount_string = text(0.1, 45, 'High five counter:');\r\ncount_string.Interpreter = 'none';\r\ncount_string.FontSize = 15;\r\nval_string = text(0.65, 45, '0');\r\nval_string.Interpreter = 'none';\r\nval_string.FontSize = 15;\r\np2 = subplot(2, 1, 2);\r\nscale_accel = image(zeros(224, 224, 3));\r\np2.Title.String = 'Scalogram';\r\np2.FontSize = 20;\r\n\r\ntelapse = 0;\r\nhfcount = 0;\r\n\r\ntic % Start timer<\/span>\r\nlast_read_time = 0;\r\n\r\ni = 0;\r\n% Run high five counter for 20 seconds<\/span>\r\nwhile(toc <= 20)\r\n current_read_time = toc;\r\n if (current_read_time - last_read_time) >= 1\/fs\r\n i = i + 1;\r\n telapse = telapse + 1;\r\n % Read accel<\/span>\r\n accel(1:end-1, :) = accel(2:end, :); % Shift values in FIFO buffer<\/span>\r\n accel(end, :) = readAcceleration(imu);\r\n\r\n plot_accel(1).YData = accel(:, 1);\r\n plot_accel(2).YData = accel(:, 2);\r\n plot_accel(3).YData = accel(:, 3);\r\n\r\n % Only run scalogram every 3rd sample to save on compute time<\/span>\r\n if mod(i, 3) == 0\r\n\r\n % Scalogram<\/span>\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 1);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i = imresize(cfs_abs\/8, [224 224]);\r\n\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 2);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i(:, :, 2) = imresize(cfs_abs\/8, [224 224]);\r\n\r\n fb = cwtfilterbank('SignalLength', length(t), 'SamplingFrequency', fs, ...\r\n 'VoicesPerOctave', 12);\r\n sig = accel(:, 3);\r\n [cfs, ~] = wt(fb, sig);\r\n cfs_abs = abs(cfs);\r\n accel_i(:, :, 3) = imresize(cfs_abs\/8, [224 224]);\r\n\r\n % Saturate pixels at 1<\/span>\r\n if ~(isempty(accel_i(accel_i>1)))\r\n accel_i(accel_i>1) = 1;\r\n end\r\n\r\n scale_accel.CData = im2uint8(accel_i);\r\n\r\n % Classify Scalogram<\/span>\r\n [YPred,probs] = classify(trainedNetwork,scale_accel.CData);\r\n if strcmp(string(YPred), 'high_five')\r\n label_string.BackgroundColor = [1 0 0];\r\n label_string.String = \"High Five!\";\r\n % Only count if 100 samples have past since last high five<\/span>\r\n if telapse > 100\r\n hfcount = hfcount + 1;\r\n val_string.String = string(hfcount);\r\n telapse = 0;\r\n end\r\n else\r\n label_string.BackgroundColor = [1 1 1];\r\n label_string.String = \"No High Five\";\r\n end\r\n end\r\n end\r\nend<\/pre>\r\n<\/div><\/div><\/div>\r\n\r\n
And here it is in action!\r\n<\/p>\r\n\r\n
$\"High$ <\/p>\r\n\r\n\r\n\r\n\r\n