{"id":8633,"date":"2017-06-02T10:19:50","date_gmt":"2017-06-02T14:19:50","guid":{"rendered":"https:\/\/blogs.mathworks.com\/pick\/?p=8633"},"modified":"2018-09-14T06:58:09","modified_gmt":"2018-09-14T10:58:09","slug":"deep-learning-tutorial-series","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/pick\/2017\/06\/02\/deep-learning-tutorial-series\/","title":{"rendered":"Deep Learning Tutorial Series"},"content":{"rendered":"<p>Our guest post this week is written by <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/4758135-johanna-pingel\">Johanna<\/a>: her pick of the week is a new <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/62990-deep-learning-tutorial-series\">Deep Learning Tutorial Series<\/a>.\u00a0This post is going to introduce the tutorial, a new video series on deep learning, and a lot of other links to get started with deep learning. Enjoy!<\/p>\n<p><span style=\"font-size: 16px;\">Avi wrote about deep learning in <a href=\"https:\/\/blogs.mathworks.com\/pick\/2017\/03\/03\/deep-learning-in-11-lines-of-matlab-code\/\">11 lines of code<\/a>. \u00a0If you\u2019re just starting out in deep learning, I encourage you to go there first: it\u2019s a great post and a great way to get started. <strong><em>But let\u2019s go a bit further<\/em><\/strong>. What happens if we want to create our own network to identify <em>new objects<\/em> in images?<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 16px;\">The entry on File Exchange is going to provide you with 3 examples:<\/span><\/p>\n<ul>\n<li><span style=\"font-size: 16px;\">Training a neural network from scratch<\/span><\/li>\n<li><span style=\"font-size: 16px;\">Transfer learning<\/span><\/li>\n<li><span style=\"font-size: 16px;\">Using a neural network as a feature extractor<\/span><\/li>\n<\/ul>\n<p><span style=\"font-size: 16px;\"><br \/>\nThese three examples are intended to help you create a custom neural network for your data. Also, they are less than 125 lines of code\u2026 <em>total!<\/em>\u00a0<\/span><br \/>\n<span style=\"font-size: 16px;\"><br \/>\nThe examples were built using images from <a href=\"https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html\">CIFAR-10<\/a>, a publicly available database of relatively small [32 by 32 pixels] images. There is a file included called <em>DownloadCIFAR10.m<\/em> that will download the images and place them into a test and training folder. It will take some time to download and extract these images, but you only need to run this once.<\/span><\/p>\n<hr \/>\n<p style=\"margin: 1% 13%; background-color: #86c5da;\"><span style=\"color: #ffffff; font-size: 20px;\">If you aren\u2019t ready to dive into code just yet, or if you would like an <strong><em>overview of deep learning<\/em><\/strong>, check out our introductory video series on neural networks. This will help answer these questions:<br \/>\n\u2022 What is deep learning?<br \/>\n\u2022 What is a neural network?<br \/>\n\u2022 What\u2019s the difference between machine learning and deep learning?<\/span><\/p>\n<p style=\"text-align: center;\"><strong><span style=\"color: #000080;\"><a style=\"color: #000080; font-size: 20px;\" href=\"https:\/\/www.mathworks.com\/videos\/series\/introduction-to-deep-learning.html\">Deep Learning Introductory Series<\/a><\/span><\/strong><\/p>\n<hr \/>\n<p><span style=\"font-size: 16px;\"><br \/>\nAll the examples require large sets of image data. If you are new to MATLAB and you have lots of data &#8211; image data or other &#8211; please check out <span style=\"color: #000080;\"><a style=\"color: #000080;\" href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/matlab.io.datastore.imagedatastore.html#buyi31e-1\">ImageDatastore <\/a><\/span>which is built on <span style=\"color: #000080;\"><a style=\"color: #000080;\" href=\"https:\/\/www.mathworks.com\/help\/matlab\/import_export\/what-is-a-datastore.html\">Datastore <\/a><\/span>to better manage large amounts of data.<\/span><\/p>\n<hr width=\"50%\/\" \/>\n<p><span style=\"color: #e67e22; font-size: 20px;\"><strong>Training a neural network from scratch<\/strong><\/span><br \/>\n<span style=\"font-size: 16px;\"><a href=\"https:\/\/www.mathworks.com\/videos\/training-a-neural-network-from-scratch-with-matlab-1492008542195.html\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-765 size-full\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/train_from_scratch.png\" width=\"726\" height=\"138\" \/><\/a><br \/>\nWe\u2019ll jump right into creating a neural network from scratch that can identify 4 categories of objects. Sounds intimidating? A neural network is simply a series of layers that each alter the input and pass the output to the next layer. This figure helps visualize the process.<\/span><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"size-full alignnone\" src=\"https:\/\/www.mathworks.com\/content\/mathworks\/www\/en\/discovery\/deep-learning\/jcr:content\/mainParsys\/band_2123350969_copy_1983242569\/mainParsys\/columns_1635259577\/1\/image_2128876021_cop.adapt.full.high.svg\/1506091562588.svg\" width=\"50%\" height=\"636\" \/><br \/>\n<span style=\"font-size: 16px;\"><br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 16px;\">Want to understand CNNs at a deeper level? Check out this short video where we dive into the key aspects of ConvNets.<\/span><\/p>\n<table style=\"border: 2px solid black;\">\n<tbody>\n<tr>\n<td><span style=\"color: #808080; font-family: Arial;\"><a href=\"https:\/\/www.mathworks.com\/videos\/introduction-to-deep-learning-what-are-convolutional-neural-networks--1489512765771.html\"><img decoding=\"async\" class=\"size-medium wp-image-764 alignleft\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/what_are_CNNs_thumbnail-300x169.jpg\" alt=\"\" \/><\/a><\/span><\/td>\n<td><a href=\"https:\/\/www.mathworks.com\/videos\/introduction-to-deep-learning-what-are-convolutional-neural-networks--1489512765771.html\"><img decoding=\"async\" class=\"alignnone size-full wp-image-776\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/explanation_cnns.png\" alt=\"\" \/><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-size: 16px;\">The code will give you a pre-defined neural network to start with, but the concept of creating a network from scratch means you have the freedom to explore the architecture and alter it as you see fit. This is an opportunity to take this example network see how changing the layers affects the accuracy. Running the code in the example without any alterations \u2013 you will see the accuracy is around 75%. <em>Can you find a way to increase the accuracy further?<\/em><\/span><\/p>\n<hr width=\"50%\/\" \/>\n<p><span style=\"color: #e67e22; font-size: 20px;\"><strong>Transfer Learning<\/strong><\/span><\/p>\n<p><span style=\"font-size: 16px;\"><a href=\"https:\/\/www.mathworks.com\/videos\/transfer-learning-with-neural-networks-in-matlab-1492007175092.html\"><img decoding=\"async\" class=\"alignnone size-full wp-image-766\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/transfer_learning.png\" alt=\"\" \/><\/a><br \/>\nTransfer learning is this semi-bizarre notion that you can take someone else\u2019s fully baked network and transfer all the learning it has done to learn a completely new set of objects. If a network has been fed millions of images of dogs and cats &#8211; and can accurately predict those categories &#8211; it may only take a few hundred images to learn a new category like rabbits. The learning can be <em>transferred<\/em> to new categories. Like the video explains, it helps if your new recognition task is similar or has <strong>similar features<\/strong> to the original categories. For example, a network trained on dogs may be well suited to learn other animals relatively quickly.<\/span><br \/>\n<span style=\"font-size: 16px;\"><br \/>\nIt also helps to have an <strong>accurate network<\/strong> to start with, which is why we\u2019ll start our example using AlexNet, a ConvNet that won the 2012 ILSVRC competition. It has been trained on many kinds of animals, and since we want to identify dogs, frogs, cats and deer, we expect this network to be a great starting point.<\/span><\/p>\n<p><span style=\"font-size: 16px;\">Importing AlexNet is a breeze in MATLAB<\/span><\/p>\n<pre><span style=\"font-size: 16px;\">net = alexnet;<\/span><\/pre>\n<p><em><span style=\"font-size: 14px;\">If this is your first time running this code, you may get an error message. Follow the link to download AlexNet. You only need to download AlexNet once.<\/span><\/em><\/p>\n<p><span style=\"font-size: 16px;\">Let\u2019s look at the last few layers of the network. You\u2019ll see AlexNet was trained for 1000 different categories of objects.<\/span><\/p>\n<pre>layers = net.Layers<\/pre>\n<pre>\u2026\r\n\r\n23\u00a0\u00a0 'fc8'\u00a0\u00a0\u00a0\u00a0\u00a0 Fully Connected\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1000 fully connected layer\r\n\r\n24\u00a0\u00a0 'prob'\u00a0\u00a0\u00a0\u00a0 Softmax\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 softmax\r\n\r\n25\u00a0\u00a0 'output'\u00a0\u00a0 Classification Output\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 crossentropyex with 'tench', 'goldfish', and 998 other classes<\/pre>\n<p><span style=\"font-size: 16px;\"><br \/>\nAs a quick aside, if you want to see all 1000 categories that AlexNet is trained on, here\u2019s a fun line of code:<\/span><\/p>\n<pre>class_names = net.Layers(end).ClassNames;<\/pre>\n<p><span style=\"font-size: 16px;\"><br \/>\nBack to the demo. Notice layer 23, where it says 1000 fully connected layer? We\u2019ll need to change that 1000 to 4, since we have four categories of objects.<\/span><\/p>\n<p><strong>Take layers from AlexNet, then add our own<\/strong><\/p>\n<pre>layers = layers(1:end-3);\r\n\r\n layers(end+1) = fullyConnectedLayer(4, 'Name', 'fc8_2 ');\r\n\r\n layers(end+1) = softmaxLayer;\r\n\r\n layers(end+1) = classificationLayer()<\/pre>\n<p><span style=\"font-size: 16px;\"><br \/>\nAfter we alter the network, we can quickly* re-train it<\/span><\/p>\n<pre>convnet = trainNetwork(imds, layers, opts);<\/pre>\n<p><strong>*Quickly is a relative term depends on your hardware and number of training images.<\/strong><\/p>\n<p><span style=\"font-size: 16px;\">At the very end, we have a custom network to recognize our four categories of objects:<\/span><\/p>\n<table style=\"border: 2px solid black;\">\n<tbody>\n<tr>\n<td style=\"border: 2px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-758 aligncenter\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/deer-1.png\" alt=\"\" width=\"221\" height=\"223\" \/><\/td>\n<td style=\"border: 2px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-758 aligncenter\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/frog-1.png\" alt=\"\" width=\"221\" height=\"223\" \/><\/td>\n<\/tr>\n<tr>\n<td style=\"border: 2px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-758 aligncenter\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/dog.png\" alt=\"\" width=\"221\" height=\"223\" \/><\/td>\n<td style=\"border: 2px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-758 aligncenter\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/cat-2.png\" alt=\"\" width=\"221\" height=\"223\" \/><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-size: 16px;\"><br \/>\nHow well does it do? Running this script gave me an accuracy of roughly 80%. Of course, this is a random process, so you can expect slight variations in this number<em>. Can you get a much higher accuracy on these four categories?<\/em> Leave a comment below!<\/span><\/p>\n<hr width=\"50%\/\" \/>\n<p><span style=\"color: #e67e22; font-size: 20px;\"><strong>Feature Extraction<\/strong><\/span><\/p>\n<p><a href=\"https:\/\/www.mathworks.com\/videos\/using-feature-extraction-with-neural-networks-in-matlab-1492009542601.html\"><img decoding=\"async\" class=\"alignnone size-full wp-image-767\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/train_feature.png\" alt=\"\" \/><\/a><br \/>\n<span style=\"font-size: 16px;\"><br \/>\nPerhaps a less common technique of neural networks is using them as feature extractors. What does that mean exactly? Gabriel explains the concepts of features and feature extraction very well in the video above.<br \/>\n<\/span><span style=\"font-size: 16px;\"><br \/>\nThe accuracy of neural networks is due in part to the huge amount of data used to train them. In fact, AlexNet \u2013 the network used as a starting point for the second and third example &#8211; was trained on <em>millions<\/em> of images. You can expect that the lower layers of the network can identify rich features that define the individual objects, so what happens if we simply extract these features?<\/span><\/p>\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-772\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/wf_machine_learning-1.png\" alt=\"\" \/> <img decoding=\"async\" class=\"alignnone size-full wp-image-771\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/wf_deep_learning-1.png\" alt=\"\" \/><\/p>\n<p><span style=\"font-size: 16px;\">Here, we have a typical machine learning and deep learning workflow. Let\u2019s combine these workflows and make a super-classifier*<strong>!<\/strong><\/span><br \/>\n<strong>*Super-classifier isn\u2019t a technical term.<\/strong><\/p>\n<p><div id=\"UsingaCNNasafeatureextractiontechnique\" style=\"width: 910px\" class=\"wp-caption alignnone\"><img aria-describedby=\"caption-UsingaCNNasafeatureextractiontechnique\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-769 size-full\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/wf_feature_extraction.png\" width=\"900\" height=\"285\" border=\"5\" \/><p id=\"caption-UsingaCNNasafeatureextractiontechnique\" class=\"wp-caption-text\">Using a CNN as a feature extraction technique, and then using those features in a machine learning model can be a highly versatile use of deep learning.<\/p><\/div><\/p>\n<p><span style=\"font-size: 16px;\"><br \/>\nNotice how easy MATLAB makes extracting these features from a test set.<\/span><\/p>\n<pre>featureLayer = 'fc7';\r\n\r\ntrainingFeatures = activations(convnet, trainingSet, featureLayer);<\/pre>\n<table style=\"border: 1px solid grey; margin: 1% 25% 1% 1%;\">\n<tbody>\n<tr>\n<td><a href=\"https:\/\/www.mathworks.com\/videos\/introduction-to-deep-learning-machine-learning-vs-deep-learning-1489503513018.html\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-775 size-thumbnail aligncenter\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/machine_learning_thumb-150x131.png\" width=\"150\" height=\"131\" \/><\/a><\/td>\n<td><a href=\"https:\/\/www.mathworks.com\/videos\/introduction-to-deep-learning-machine-learning-vs-deep-learning-1489503513018.html\"><img decoding=\"async\" class=\"alignnone size-full wp-image-777\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/machine_learning_desc-1.png\" alt=\"\" \/><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-size: 16px;\"><br \/>\nThe features can be extracted from any place in the network. We choose the last fully connected layer these will be the most complex and descriptive for our categories, but you could choose the features at an earlier stage. <em>What if you choose features from another layer? <\/em>How does this affect the accuracy? You can even visualize the activations of the network in <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/examples\/visualize-activations-of-a-convolutional-neural-network.html\">this example<\/a>.<\/span><\/p>\n<p><span style=\"font-size: 16px;\">Another benefit of using a feature extraction approach is the ability to choose whichever classifier will best fit the data. We\u2019re using a multi-class svm in the video:<\/span><\/p>\n<pre><span style=\"font-size: 16px;\">fitcecoc <\/span><\/pre>\n<p><span style=\"font-size: 16px;\">We could also choose a multi-<\/span><span style=\"font-size: 16px;\">class naive Bayes model &#8211;\u00a0<em>fitcnb<\/em>\u00a0&#8211; or a k-nearest neighbor classification model &#8211;\u00a0<em>fitcknn<\/em>\u00a0&#8211; or a variety of other classification models.<\/span><\/p>\n<p><span style=\"font-size: 16px;\"><br \/>\nIt\u2019s great to have to many options,<em> but how on earth do you know which classifier to choose?<\/em><\/span><br \/>\n<span style=\"font-size: 16px;\"><br \/>\nCheck out our <a href=\"https:\/\/www.mathworks.com\/help\/stats\/classificationlearner-app.html\">classification learner app<\/a> to find the best model for the data.<\/span><\/p>\n<p><span style=\"font-size: 16px;\">Another advantage to feature extraction using ConvNets is you need much less data to get an accurate result. At the very end of the demo, we see that with minimal data, we get an accuracy of approximately 77%. We intentionally used less data [50 images per category] in this example to show how you can still use a deep learning approach without thousands of sample images. <em>Would increasing the number of training images also increase the accuracy?<\/em><\/span><\/p>\n<p><span style=\"font-size: 16px;\"><br \/>\nTake the code at line number 11<\/span><\/p>\n<pre>[trainingSet, ~] = splitEachLabel(imds, 50, 'randomize');<\/pre>\n<p><span style=\"font-size: 16px;\"><br \/>\nand change the number 50 to a higher number to increase the number of samples per category.<\/span><\/p>\n<p><span style=\"color: #e67e22; font-size: 20px;\"><strong>Summary<\/strong><\/span><br \/>\n<span style=\"font-size: 16px;\"><br \/>\nThese three examples and corresponding videos can help you explore 3 different ways of creating a custom object recognition model using deep learning techniques.<br \/>\n<\/span><span style=\"font-size: 16px;\"><br \/>\nGive it a try and let us know what you think\u00a0<a href=\"https:\/\/blogs.mathworks.com\/pick\/?p=8352#respond\">here<\/a>\u00a0or leave a\u00a0<a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/60659-deep-learning-in-11-lines-of-matlab-code#comments\">comment<\/a>\u00a0for the Deep Learning Team.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/wp-content\/uploads\/2017\/06\/train_from_scratch.png\" onError=\"this.style.display ='none';\" \/><\/div>\n<p>Our guest post this week is written by Johanna: her pick of the week is a new Deep Learning Tutorial Series.\u00a0This post is going to introduce the tutorial, a new video series on deep learning, and a&#8230; <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/pick\/2017\/06\/02\/deep-learning-tutorial-series\/\">read more >><\/a><\/p>\n","protected":false},"author":132,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[16],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts\/8633"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/users\/132"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/comments?post=8633"}],"version-history":[{"count":4,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts\/8633\/revisions"}],"predecessor-version":[{"id":10137,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/posts\/8633\/revisions\/10137"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/media?parent=8633"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/categories?post=8633"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/pick\/wp-json\/wp\/v2\/tags?post=8633"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}