{"id":9022,"date":"2022-02-16T09:08:00","date_gmt":"2022-02-16T14:08:00","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=9022"},"modified":"2022-02-16T09:08:00","modified_gmt":"2022-02-16T14:08:00","slug":"detection-of-ships-on-satellite-images-using-yolo-v2-model","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2022\/02\/16\/detection-of-ships-on-satellite-images-using-yolo-v2-model\/","title":{"rendered":"Detection of ships on satellite images using YOLO v2 model"},"content":{"rendered":"<em>This post is from Primo\u017e Poto\u010dnik, Assistant Professor at the <a href=\"https:\/\/www.uni-lj.si\/eng\/\" target=\"_blank\" rel=\"noopener\">University of Ljubljana<\/a>, <a href=\"https:\/\/www.fs.uni-lj.si\/en\/\" target=\"_blank\" rel=\"noopener\">Faculty of Mechanical Engineering<\/a>, <a href=\"https:\/\/web.fs.uni-lj.si\/lasin\/\" target=\"_blank\" rel=\"noopener\">Laboratory of Synergetics<\/a>. Professor Poto\u010dnik developed an AI application using a YOLO deep network for the detection of ships on satellite images aimed at providing students of the course <a href=\"https:\/\/www.fpp.uni-lj.si\/en\/study\/second-cycle\/maritime-studies\/2019111911272346\/autonomous-vessels\" target=\"_blank\" rel=\"noopener\">Autonomous Vessels<\/a>, a practical example of the usage of Deep Learning in Maritime Engineering.<\/em>\r\n<h6><\/h6>\r\nPrimo\u017e has a very profound knowledge in neural networks and machine learning and has been collaborating in many industrial projects in the field of industrial condition monitoring and forecasting applications for the energy market. Additionally, he is teaching the courses <a href=\"https:\/\/web.fs.uni-lj.si\/lasin\/en\/pedagoski-proces\/nevronske-mreze\/\" target=\"_blank\" rel=\"noopener\">Neural Networks<\/a>, <a href=\"https:\/\/www.fpp.uni-lj.si\/en\/study\/second-cycle\/maritime-studies\/2019111911272346\/autonomous-vessels\" target=\"_blank\" rel=\"noopener\">Autonomous Vessels<\/a>, and Empirical modeling and characterization of processes at the University of Ljubljana. His bibliography is available on <a href=\"http:\/\/splet02.izum.si\/cobiss\/bibliography?code=15107&amp;langbib=eng\" target=\"_blank\" rel=\"noopener\">COBISS<\/a> and <a href=\"https:\/\/www.researchgate.net\/profile\/Primoz-Potocnik-2\" target=\"_blank\" rel=\"noopener\">ResearchGate<\/a>.\r\n<h6><\/h6>\r\n<h1>Introduction<\/h1>\r\n<h6><\/h6>\r\nThe development of maritime autonomous surface ships (MASS) has rapidly emerged in recent years worldwide, with strong support in northern EU countries, the USA, China, and others, and is also supported by the <a href=\"https:\/\/www.imo.org\/en\/MediaCentre\/HotTopics\/Pages\/Autonomous-shipping.aspx\" target=\"_blank\" rel=\"noopener\">International Maritime Organization<\/a>. Currently, various autonomous and remote-controlled ships are being trialed in testing areas in <a href=\"http:\/\/dma.dk\/growth-and-framework-conditions\/maritime-digitalization\/test-activities-in-denmark\" target=\"_blank\" rel=\"noopener\">Denmark<\/a>, <a href=\"https:\/\/www.oneseaecosystem.net\/\" target=\"_blank\" rel=\"noopener\">Finland<\/a>, <a href=\"https:\/\/nfas.autonomous-ship.org\/resources_page\/projects-page\/\" target=\"_blank\" rel=\"noopener\">Norway<\/a>, and many other areas worldwide. As an example, <a href=\"https:\/\/www.yara.com\/news-and-media\/press-kits\/yara-birkeland-press-kit\/\" target=\"_blank\" rel=\"noopener\">Yara Birkeland<\/a> will be one of the world\u2019s first fully electric and autonomous container vessels. Besides many navigations, regulatory, cyber-security and other challenges, sensory networks, and massive data processing for increasing situational awareness of MASS present one of the major challenges in autonomous maritime shipping.\r\n\r\nThis example demonstrates how to extract useful information from satellite images using a simple deep learning-based ship detector, which can be included in a broad range of situational awareness sensors of MASS systems.\r\n<h6><\/h6>\r\n<h1>Data and resources<\/h1>\r\n<h6><\/h6>\r\nSatellite images from the <a href=\"https:\/\/www.kaggle.com\/c\/airbus-ship-detection\/data\" target=\"_blank\" rel=\"noopener\">Airbus Ship Detection Challenge database<\/a> are used and arranged in two folders: <u>train<\/u> and <u>test<\/u>.\r\n<h6><\/h6>\r\n<ul>\r\n \t<li><strong>Train folder<\/strong> includes 100 ship images. These images are used for training the deep learning detector.<\/li>\r\n \t<li><strong>Test folder<\/strong>\u00a0includes additional 50 ship images. These are used after training to evaluate the performance of the developed detector on new, unseen data.<\/li>\r\n<\/ul>\r\nThis example is based on MATLAB example <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ug\/object-detection-using-yolo-v2.html\" target=\"_blank\" rel=\"noopener\">Object Detection Using YOLO v2 Deep Learning<\/a> which applies transfer learning as a solution approach. <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/gs\/get-started-with-transfer-learning.html\" target=\"_blank\" rel=\"noopener\">Transfer learning<\/a> utilizes a pretrained deep network and is a good starting point to learn a new task. Moreover, using transfer learning requires less data and less computational power as the training doesn\u2019t take as much time as training from scratch.\r\n<h6><\/h6>\r\n<h1>Objective<\/h1>\r\n<h6><\/h6>\r\nThe task of this example is to use the dataset of available images (train folder) and construct the ship detector using deep learning and transfer learning. The objective for the developed detector is then to detect and locate the ships on new images (test folder) which the detector hasn\u2019t seen before.\r\n<h6><\/h6>\r\n<h1>Solution approach<\/h1>\r\n<h6><\/h6>\r\nThe presented solution approach comprises of the following steps:\r\n<ol>\r\n \t<li>Using <a href=\"https:\/\/www.mathworks.com\/help\/vision\/ref\/imagelabeler-app.html\" target=\"_blank\" rel=\"noopener\">Image Labeler<\/a> for labeling the ship segments.<\/li>\r\n \t<li>Preprocessing data (creating datastore and resizing images).<\/li>\r\n \t<li>Applying pretrained <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/resnet50.html\" target=\"_blank\" rel=\"noopener\">ResNet50<\/a> network as a feature extractor.<\/li>\r\n \t<li>Creating and training <a href=\"https:\/\/www.mathworks.com\/help\/vision\/ug\/create-yolo-v2-object-detection-network.html\" target=\"_blank\" rel=\"noopener\">YOLO<\/a> object detection network.<\/li>\r\n \t<li>Evaluating the trained detector on test images.<\/li>\r\n<\/ol>\r\n<h1>Solution workflow<\/h1>\r\n<h6><\/h6>\r\n<h2>Labeling ships in satellite images<\/h2>\r\n<h6><\/h6>\r\nUsing the <a href=\"https:\/\/www.mathworks.com\/help\/vision\/ref\/imagelabeler-app.html\" target=\"_blank\" rel=\"noopener\">Image Labeler<\/a> App, ships are manually labeled in all training satellite images, without writing any line of code. The app is very intuitive and self-explanatory (as shown in Figure 1) and enables the end-user to accomplish the labelling task in a very short time.\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"780\" class=\"alignnone size-large wp-image-9031\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Figure1-1024x780.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\nFigure 1: Working with Image Labeler to label the ship segments.\r\n<h6><\/h6>\r\n<h2>Data preprocessing<\/h2>\r\n<h6><\/h6>\r\nTo use the labeled images for deep network training, the following two steps must be accomplished:\r\n<ul>\r\n \t<li>Create a <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/datastore.html?s_tid=srchtitle_datastore_1\" target=\"_blank\" rel=\"noopener\">datastore<\/a> for loading the images and labels. The datastore acts as a repository for data and overcomes the out-of-memory issue which often appears when dealing with large data such as images.<\/li>\r\n \t<li>Resize images (and bounding boxes) into required YOLO dimensions. In this example, a ResNet-50 feature extractor model is used which requires the images to have the dimension of 224 x 224.<\/li>\r\n<\/ul>\r\nWith these steps, the labeled and properly resized images are ready to be used as training inputs for the deep network detector. Figure 2 shows several training images with labeled ships once the preprocessing is completed.\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"257\" class=\"alignnone size-large wp-image-9028\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Cover_orig-1024x257.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\nFigure 2: Training images with labeled ships\r\n<h6><\/h6>\r\n<h2>Applying pretrained ResNet50 deep network as a feature extractor<\/h2>\r\n<h6><\/h6>\r\nA pretrained version of the <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/64626-deep-learning-toolbox-model-for-resnet-50-network\" target=\"_blank\" rel=\"noopener\">ResNet-50<\/a> deep network model can be found in MATLAB Add-Ons. This model has been trained on a subset of the ImageNet database, has 177 layers in total, corresponding to a 50-layer residual network, and can classify images into 1000 object categories.\r\n<h6><\/h6>\r\nIn our example, the pretrained ResNet-50 model is used as a <strong>feature extractor<\/strong>. Therefore, the 'activation_40_relu' layer is selected as the feature extraction layer, and subsequent layers are replaced with the YOLO v2 detection subnetwork. We can do this process programmatically (writing the code in line) or interactively using the <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/deepnetworkdesigner-app.html?searchHighlight=Deep%20Network%20Designer%20&amp;s_tid=srchtitle_Deep%20Network%20Designer%20_1\" target=\"_blank\" rel=\"noopener\">Deep Network Designer App<\/a>. Figure 3 shows the location of the '<strong>activation_40_relu<\/strong>' layer in the ResNet-50 model.\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"1008\" class=\"alignnone size-large wp-image-9037\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Figure3-1024x1008.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\nFigure 3: Location of the 'activation_40_relu' layer in the ResNet-50 model, used as a feature extractor\r\n<h6><\/h6>\r\n<h2>Creating and training YOLO object detection network<\/h2>\r\n<a href=\"https:\/\/www.mathworks.com\/help\/vision\/ug\/create-yolo-v2-object-detection-network.html\" target=\"_blank\" rel=\"noopener\">YOLO object detection<\/a> algorithm employs convolutional neural networks (CNN) to detect objects in real-time. YOLO is an abbreviation for the term \u2018You Only Look Once\u2019 which suggests that the algorithm requires only a single forward propagation through a neural network to detect objects. This algorithm is popular because of its speed and accuracy. It has been used in various applications to detect traffic signals, people, parking meters, and animals.\r\n<h6><\/h6>\r\nVarious parameters and settings need to be specified for the YOLO algorithm:\r\n<pre>inputSize = [224 224 3]; % network input size\r\n\r\nnumClasses = 1; % number of object classes to detect (we have only one class 'Ship')\r\n\r\nnumAnchors = 7; % number of anchor boxes<\/pre>\r\nAnd then, <a href=\"https:\/\/www.mathworks.com\/help\/vision\/ug\/anchor-boxes-for-object-detection.html\" target=\"_blank\" rel=\"noopener\">Anchor Boxes for Object Detection<\/a> are estimated based on the size of objects in the training data:\r\n<pre>[anchorBoxes, meanIoU] = estimateAnchorBoxes(preprocessedData, numAnchors);<\/pre>\r\nYOLO v2 object detection network is created by using the ResNet50 feature extractor:\r\n<pre>lgraph = yolov2Layers(inputSize,numClasses,anchorBoxes,featureExtractionNetwork,featureLayer)<\/pre>\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"1008\" class=\"alignnone size-large wp-image-9040\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Figure4-1024x1008.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\nFigure 4: The resulting YOLO v2 image detector architecture based on the ResNet-50 feature extractor\r\n<h6><\/h6>\r\nFinally, YOLO v2 object detector is trained (training may take an hour or two) using various training options:\r\n<pre>options = trainingOptions('sgdm', 'MiniBatchSize',12, 'InitialLearnRate',1e-3, 'MaxEpochs',120, 'ValidationData',preprocessedData);\r\n\r\n[detector,info] = trainYOLOv2ObjectDetector(preprocessedData,lgraph,options);<\/pre>\r\n<h6><\/h6>\r\n<em>Note: all these steps (designing, loading training data, setting training options, and training) can be done within the Deep Network Designer App. After the training, the optimized model needs to be exported to the workspace. An automatic function can be generated of the entire process in order to avoid the manual design if needed again.<\/em>\r\n<h6><\/h6>\r\n<h1>Results<\/h1>\r\n<h6><\/h6>\r\n<h2>Evaluating YOLO detector on train images<\/h2>\r\n<h6><\/h6>\r\nFigure 5 shows an excellent performance of the trained YOLO detector on train images (only the first 15 images are shown).\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"604\" class=\"alignnone size-large wp-image-9043\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Figure5-1024x604.png\" alt=\"\" \/>\r\nFigure 5: Evaluating YOLO detector on train images\r\n<h6><\/h6>\r\n<h2>Evaluating YOLO detector on test images<\/h2>\r\n<h6><\/h6>\r\nDetection of ships on new images (test data set) shows worse performance, although several ships are correctly detected (Figure 6).\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"604\" class=\"alignnone size-large wp-image-9046\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Figure6-1024x604.png\" alt=\"\" \/>\r\nFigure 6: Evaluating YOLO detector on test images\r\n<h6><\/h6>\r\nConsidering the very good performance on the train data set and the relatively insufficient performance on the test data set, the trained model is most probably overfitted. However, it demonstrates that the small data set can be used to detect some objects.\r\n<h6><\/h6>\r\nTo improve the detection result on new images, the YOLO detector should be trained on a considerably larger dataset of images, and independent validation data set should be used to avoid overfitting. And finally, the training options (hyperparameters) can be further optimized to obtain a higher accuracy on the test data. This can be done using the <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/experimentmanager-app.html?searchHighlight=experiment%20manager%20app&amp;s_tid=srchtitle_experiment%20manager%20app_1\" target=\"_blank\" rel=\"noopener\">Experiment Manager App<\/a> which is a great tool to keep track of all the hyperparameter changes.\r\n<h6><\/h6>\r\n<h1>Key takeaways<\/h1>\r\n<h6><\/h6>\r\nThis example presents the step-by-step development of a deep learning solution for the detection of ships on satellite images. The solution approach is quite general and can be adapted to different domain areas, similarly as this case study was adopted from an earlier MATLAB application of <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ug\/object-detection-using-yolo-v2.html\" target=\"_blank\" rel=\"noopener\">Object Detection Using YOLO v2 Deep Learning<\/a>. A few conclusions and takeaways can be summarized:\r\n<h6><\/h6>\r\n<ul>\r\n \t<li>The trained YOLO network already successfully detects several ships in satellite images, although some ships on test images are not detected.<\/li>\r\n \t<li>Training data set of only 100 ship images is too small to train a ship detector from scratch, but by using the transfer learning approach, the required number of images is enough to quickly obtain first results.<\/li>\r\n \t<li>The example presents the potential workflow toward the solution, and in practice, many more annotated images (in order of hundreds or thousands of images) should be provided to train a detector with good detection accuracy.<\/li>\r\n \t<li>This material can inspire other lecturers to develop further Deep Learning-based examples within the framework of Maritime Engineering.<\/li>\r\n<\/ul>\r\n<h1>MATLAB code<\/h1>\r\n<h6><\/h6>\r\nThe complete MATLAB code with a database of images is available at <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/\" target=\"_blank\" rel=\"noopener\">MATLAB Central File Exchange<\/a> and <a href=\"https:\/\/github.com\/\" target=\"_blank\" rel=\"noopener\">GitHub<\/a>:\r\n<h6><\/h6>\r\n<ul>\r\n \t<li><a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/105830-detection-of-ships-on-satellite-images-using-yolo-v2\" target=\"_blank\" rel=\"noopener\">https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/105830-detection-of-ships-on-satellite-images-using-yolo-v2<\/a><\/li>\r\n \t<li><a href=\"https:\/\/github.com\/ppotoc\/Detection-of-ships-on-satellite-images-using-YOLO-v2-deep-learning\" target=\"_blank\" rel=\"noopener\">https:\/\/github.com\/ppotoc\/Detection-of-ships-on-satellite-images-using-YOLO-v2-deep-learning<\/a><\/li>\r\n<\/ul>","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/02\/Cover.png\" class=\"img-responsive attachment-post-thumbnail size-post-thumbnail wp-post-image\" alt=\"\" decoding=\"async\" loading=\"lazy\" \/><\/div><p>This post is from Primo\u017e Poto\u010dnik, Assistant Professor at the University of Ljubljana, Faculty of Mechanical Engineering, Laboratory of Synergetics. Professor Poto\u010dnik developed an AI application... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/deep-learning\/2022\/02\/16\/detection-of-ships-on-satellite-images-using-yolo-v2-model\/\">read more >><\/a><\/p>","protected":false},"author":180,"featured_media":9025,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/9022"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/users\/180"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/comments?post=9022"}],"version-history":[{"count":32,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/9022\/revisions"}],"predecessor-version":[{"id":9145,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/9022\/revisions\/9145"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/media\/9025"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/media?parent=9022"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/categories?post=9022"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/tags?post=9022"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}