{"id":3505,"date":"2020-01-30T19:57:48","date_gmt":"2020-01-30T19:57:48","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=3505"},"modified":"2021-04-06T15:49:07","modified_gmt":"2021-04-06T19:49:07","slug":"activity-classification-demo","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2020\/01\/30\/activity-classification-demo\/","title":{"rendered":"Activity Classification Demo"},"content":{"rendered":"Every January, our company hosts an annual \u201ckickoff\u201d: an opportunity for sales, marketing and folks in customer facing roles to meet in Boston and learn about all the new and exciting features coming in 2020 for their relevant product areas. Being on the deep learning marketing team, we decided to put words into actions by <strong>providing everyone the opportunity to interact with a neural network<\/strong> (some for the first time!)\r\n\r\n<div id=\"attachment_3507\" style=\"width: 206px\" class=\"wp-caption alignnone\"><img aria-describedby=\"caption-attachment-3507\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-3507 size-medium\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/example-196x300.png\" alt=\"\" width=\"196\" height=\"300\" \/><p id=\"caption-attachment-3507\" class=\"wp-caption-text\">Starting with an example from deep learning doc.<\/p><\/div>\r\n\r\nHere's how we wanted the demo to work:\r\n\r\nEach team competes to correctly classify as many activities as possible in 10 minutes. Each team has a laptop &amp; webcam. They record a small video of a team member performing an activity, and then send the video to a network for classification.\r\n<h6><\/h6>\r\n<h3><strong>Requirements of the demo<\/strong><\/h3>\r\nSometimes, I\u2019ll admit, I focus heavily on the deep learning model, and not about the entire system. This project was an opportunity for me to learn to incorporate an entire application: from front end to deployment.\r\n<h6><\/h6>\r\n<ul>\r\n \t<li>Interact with neural network regardless of technical background<\/li>\r\n \t<li>Many people can run at one time<\/li>\r\n \t<li>Network prediction reasonably fast (which may require GPU support)<\/li>\r\n<\/ul>\r\nHeather (@heathergorr) and I spent roughly 1 week putting this demo together from start to finish. We started with an example in the documentation (link <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/examples\/classify-videos-using-deep-learning.html\">https:\/\/www.mathworks.com\/help\/deeplearning\/examples\/classify-videos-using-deep-learning.html<\/a>) and retrained the network to recognize 30 activities.\r\n<h6><\/h6>\r\n<h3>Creating the model<\/h3>\r\nThe example in doc does a nice job talking about how the network can identify activities in video streams by first using GoogleNet to pull activations from the video files, and then using an LSTM network to classify those activations into their activity classes.\r\n<h6><\/h6>\r\nThe activations code looks likes this:\r\n<h6><\/h6>\r\n<code>for i = 1:numFiles\r\n\r\n  fprintf(\"Reading file %d of %d...\\n\", i, numFiles)\r\n\r\n  video = readVideo(files(i));\r\n  video = centerCrop(video,inputSize);\r\n\r\n  sequences{i,1} = activations(netCNN,video,layerName,'OutputAs','columns');\r\nend<\/code>\r\n<h6><\/h6>\r\nThe LSTM network looks like this:\r\n<h6><\/h6>\r\n<code>layers = [\r\n  sequenceInputLayer(numFeatures,'Name','sequence')\r\n  bilstmLayer(2000,'OutputMode','last','Name','bilstm')\r\n  dropoutLayer(0.5,'Name','drop')\r\n  fullyConnectedLayer(numClasses,'Name','fc')\r\n  softmaxLayer('Name','softmax')\r\n  classificationLayer('Name','classification')];<\/code>\r\n<h6><\/h6>\r\nAnd the training looks like this:\r\n<h6><\/h6>\r\n<code>miniBatchSize = 16;\r\nnumObservations = numel(sequencesTrain);\r\nnumIterationsPerEpoch = floor(numObservations \/ miniBatchSize);\r\n\r\noptions = trainingOptions('adam', ...\r\n  'MiniBatchSize',miniBatchSize, ...\r\n  'InitialLearnRate',1e-4, ...\r\n  'GradientThreshold',2, ...\r\n  'Shuffle','every-epoch', ...\r\n  'ValidationData',{sequencesValidation,labelsValidation}, ...\r\n  'ValidationFrequency',numIterationsPerEpoch, ...\r\n  'Plots','training-progress', ...\r\n  'Verbose',false);\r\n\r\n[netLSTM,info] = trainNetwork(sequencesTrain,labelsTrain,layers,options);<\/code>\r\n\r\nThe accuracy of the original model with 51 activities left a lot to be desired, with an accuracy of 67.8%. Looking at the training data set, you could see why: There is a lot of variation between the performers of the activities, and more data might have helped too.\r\n<h6><\/h6>\r\n<em>Side note: we didn\u2019t see a lot of improvement in accuracy with hyperparameter tuning. It seemed like the documentation example already has the parameters set to deliver the highest accuracy model. <\/em>\r\n<h6><\/h6>\r\nWe then trained on 20 activities, (71%) and finally 8 activities, finally reaching an accuracy of roughly 80%. We chose the 8 activities based on what we thought could be accomplished in the 10 minutes allotted for the game, and allowing the best opportunity for high classification accuracy.\r\n<h6><\/h6>\r\n<div id=\"attachment_3539\" style=\"width: 1034px\" class=\"wp-caption alignnone\"><a href=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/8class_desktop.jpg\"><img aria-describedby=\"caption-attachment-3539\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-3539 size-large\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/8class_desktop-1024x573.jpg\" alt=\"\" width=\"1024\" height=\"573\" \/><\/a><p id=\"caption-attachment-3539\" class=\"wp-caption-text\">Accuracy of the 8 class model. Overfitting?<\/p><\/div>\r\n\r\nThe front end, created in app designer (video overview link: <a href=\"https:\/\/www.mathworks.com\/videos\/app-designer-overview-1510748719083.html\">https:\/\/www.mathworks.com\/videos\/app-designer-overview-1510748719083.html<\/a>), was intentionally simple and allowed users to quickly choose an activity, record a small video, and receive classification results.\r\n<h6><\/h6>\r\n<div id=\"attachment_3513\" style=\"width: 310px\" class=\"wp-caption alignnone\"><img aria-describedby=\"caption-attachment-3513\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-3513 size-medium aligncenter\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/74035809-F32C-4FA6-B3D5-27870520564A-300x225.jpeg\" alt=\"\" width=\"300\" height=\"225\" \/><p id=\"caption-attachment-3513\" class=\"wp-caption-text\">Heather and Johanna demonstrating the app to a room of demo participants!<\/p><\/div>\r\n<h6><\/h6>\r\nHere is the app and network in action:\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"600\" height=\"388\" class=\"alignnone size-full wp-image-3515\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/ezgif.com-video-to-gif-2.gif\" alt=\"\" \/>\r\n<h6><\/h6>\r\n<h3>Deploying the app<\/h3>\r\nNow \u2013 with a room of 300 people: how do you get them access to the files and run the app?\r\n<h6><\/h6>\r\nEnter <strong>MATLAB Online<\/strong>: information on the product is here: <a href=\"https:\/\/www.mathworks.com\/products\/matlab-online.html\">https:\/\/www.mathworks.com\/products\/matlab-online.html <\/a>and you can run MATLAB Online here: <a href=\"https:\/\/matlab.mathworks.com\/\">https:\/\/matlab.mathworks.com\/<\/a>\r\n<h6><\/h6>\r\nIf you have access to MATLAB, you have access to MATLAB Online, which is a convenient way to run MATLAB away from your standard setup. We were pleasantly surprised that the network prediction was quite speedy: under 5 seconds per prediction without needing GPUs which suited our requirements quite well.\r\n<h6><\/h6>\r\nWe considered a few other options as well:\r\n<h6><\/h6>\r\n<ul>\r\n \t<li>A MATLAB Web App (link: <a href=\"https:\/\/www.mathworks.com\/help\/compiler\/webapps\/install-matlab-web-app-server.html\">https:\/\/www.mathworks.com\/help\/compiler\/webapps\/install-matlab-web-app-server.html<\/a>) that could meet the needs of having everyone in your office run a web app from a browser \u2013 regardless of MATLAB access. This is perfect for running a web app internally. Bringing this to a convention center is not the intended use case of the server, and we didn\u2019t want to risk any issues with needing a VPN.<\/li>\r\n \t<li>There\u2019s MATLAB Compiler (<a href=\"https:\/\/www.mathworks.com\/products\/compiler.html\">https:\/\/www.mathworks.com\/products\/compiler.html<\/a>) \u2013you can create an app that can run on everyone\u2019s computer locally (without the need for internet or a MATLAB license on the final computer). For the purpose of the game, we didn\u2019t want people to need to install an app for so short a time slot.<\/li>\r\n<\/ul>\r\nSince we decided on MATLAB Online, we could share a link to the app, which made sharing the app and code much easier. You can also take a look at the code and run the app too! A link to the read-only code is <a href=\"https:\/\/drive.matlab.com\/sharing\/46a00c6b-c2ca-4552-9331-ae78a324f41c\">here<\/a>.\r\n<h6><\/h6>\r\nThe game was a success: we have fun images of people trying the activities. Yet the model was quite finicky (in the first session, everyone was considered \u201csmoking\u201d regardless of the actual activity), but later sessions were able to get 6 out of 8 activities correct in 10 minutes. We also realized when you put your hand near your face, it\u2019s often classified as smoking or brushing hair! This was a great opportunity for people not as familiar with deep learning to have hands on experience interacting with a neural network, which also prompted some discussions among participants on why deep learning isn't always 100% accurate.\r\n<h6><\/h6>\r\n<div id=\"attachment_3511\" style=\"width: 310px\" class=\"wp-caption alignnone\"><img aria-describedby=\"caption-attachment-3511\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-3511 size-medium aligncenter\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/90CEDE2E-0617-447D-B219-1CF0F87D58D2-300x225.jpeg\" alt=\"\" width=\"300\" height=\"225\" \/><p id=\"caption-attachment-3511\" class=\"wp-caption-text\">Demo Day Success!! We had a room full of people acting out all 8 activities.<\/p><\/div>\r\n\r\nIn general, it was nice to think about the entire package. Not only the model, but how that model would be used: focusing on an app for people to interact with, and the way in which people would access the model.\r\n\r\nI encourage you to try out MATLAB Online this week and let me know your thoughts.\r\n<h6><\/h6>\r\nP.S. R2020a is just around the corner this spring, and from our meetings last week, it looks to be shaping up as a great release. Stay tuned!\r\n<h6><\/h6>","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2020\/01\/example-196x300.png\" onError=\"this.style.display ='none';\" \/><\/div><p>Every January, our company hosts an annual \u201ckickoff\u201d: an opportunity for sales, marketing and folks in customer facing roles to meet in Boston and learn about all the new and exciting features coming... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/deep-learning\/2020\/01\/30\/activity-classification-demo\/\">read more >><\/a><\/p>","protected":false},"author":156,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/3505"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/users\/156"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/comments?post=3505"}],"version-history":[{"count":22,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/3505\/revisions"}],"predecessor-version":[{"id":3561,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/3505\/revisions\/3561"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/media?parent=3505"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/categories?post=3505"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/tags?post=3505"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}