This post is from Primož Potočnik, Assistant Professor at the University of Ljubljana, Faculty of Mechanical Engineering, Laboratory of Synergetics. Professor Potočnik developed an AI application using a YOLO deep network for the detection of ships on satellite images aimed at providing students of the course Autonomous Vessels, a practical example of the usage of Deep Learning in Maritime Engineering.

Primož has a very profound knowledge in neural networks and machine learning and has been collaborating in many industrial projects in the field of industrial condition monitoring and forecasting applications for the energy market. Additionally, he is teaching the courses Neural Networks, Autonomous Vessels, and Empirical modeling and characterization of processes at the University of Ljubljana. His bibliography is available on COBISS and ResearchGate.

Introduction

The development of maritime autonomous surface ships (MASS) has rapidly emerged in recent years worldwide, with strong support in northern EU countries, the USA, China, and others, and is also supported by the International Maritime Organization. Currently, various autonomous and remote-controlled ships are being trialed in testing areas in Denmark, Finland, Norway, and many other areas worldwide. As an example, Yara Birkeland will be one of the world’s first fully electric and autonomous container vessels. Besides many navigations, regulatory, cyber-security and other challenges, sensory networks, and massive data processing for increasing situational awareness of MASS present one of the major challenges in autonomous maritime shipping. This example demonstrates how to extract useful information from satellite images using a simple deep learning-based ship detector, which can be included in a broad range of situational awareness sensors of MASS systems.

Data and resources

Satellite images from the Airbus Ship Detection Challenge database are used and arranged in two folders: train and test.

• Train folder includes 100 ship images. These images are used for training the deep learning detector.
• Test folder includes additional 50 ship images. These are used after training to evaluate the performance of the developed detector on new, unseen data.

This example is based on MATLAB example Object Detection Using YOLO v2 Deep Learning which applies transfer learning as a solution approach. Transfer learning utilizes a pretrained deep network and is a good starting point to learn a new task. Moreover, using transfer learning requires less data and less computational power as the training doesn’t take as much time as training from scratch.

Objective

The task of this example is to use the dataset of available images (train folder) and construct the ship detector using deep learning and transfer learning. The objective for the developed detector is then to detect and locate the ships on new images (test folder) which the detector hasn’t seen before.

Solution approach

The presented solution approach comprises of the following steps:

1. Using Image Labeler for labeling the ship segments.
2. Preprocessing data (creating datastore and resizing images).
3. Applying pretrained ResNet50 network as a feature extractor.
4. Creating and training YOLO object detection network.
5. Evaluating the trained detector on test images.

Solution workflow

Labeling ships in satellite images

Using the Image Labeler App, ships are manually labeled in all training satellite images, without writing any line of code. The app is very intuitive and self-explanatory (as shown in Figure 1) and enables the end-user to accomplish the labelling task in a very short time.

Figure 1: Working with Image Labeler to label the ship segments.

Data preprocessing

To use the labeled images for deep network training, the following two steps must be accomplished:

• Create a datastore for loading the images and labels. The datastore acts as a repository for data and overcomes the out-of-memory issue which often appears when dealing with large data such as images.
• Resize images (and bounding boxes) into required YOLO dimensions. In this example, a ResNet-50 feature extractor model is used which requires the images to have the dimension of 224 x 224.

With these steps, the labeled and properly resized images are ready to be used as training inputs for the deep network detector. Figure 2 shows several training images with labeled ships once the preprocessing is completed.

Figure 2: Training images with labeled ships

Applying pretrained ResNet50 deep network as a feature extractor

A pretrained version of the ResNet-50 deep network model can be found in MATLAB Add-Ons. This model has been trained on a subset of the ImageNet database, has 177 layers in total, corresponding to a 50-layer residual network, and can classify images into 1000 object categories.

In our example, the pretrained ResNet-50 model is used as a feature extractor. Therefore, the 'activation_40_relu' layer is selected as the feature extraction layer, and subsequent layers are replaced with the YOLO v2 detection subnetwork. We can do this process programmatically (writing the code in line) or interactively using the Deep Network Designer App. Figure 3 shows the location of the 'activation_40_relu' layer in the ResNet-50 model.

Figure 3: Location of the 'activation_40_relu' layer in the ResNet-50 model, used as a feature extractor

Creating and training YOLO object detection network

YOLO object detection algorithm employs convolutional neural networks (CNN) to detect objects in real-time. YOLO is an abbreviation for the term ‘You Only Look Once’ which suggests that the algorithm requires only a single forward propagation through a neural network to detect objects. This algorithm is popular because of its speed and accuracy. It has been used in various applications to detect traffic signals, people, parking meters, and animals.

Various parameters and settings need to be specified for the YOLO algorithm:

inputSize = [224 224 3]; % network input size

numClasses = 1; % number of object classes to detect (we have only one class 'Ship')

numAnchors = 7; % number of anchor boxes

And then, Anchor Boxes for Object Detection are estimated based on the size of objects in the training data:

[anchorBoxes, meanIoU] = estimateAnchorBoxes(preprocessedData, numAnchors);

YOLO v2 object detection network is created by using the ResNet50 feature extractor:

lgraph = yolov2Layers(inputSize,numClasses,anchorBoxes,featureExtractionNetwork,featureLayer)

Figure 4: The resulting YOLO v2 image detector architecture based on the ResNet-50 feature extractor

Finally, YOLO v2 object detector is trained (training may take an hour or two) using various training options:

options = trainingOptions('sgdm', 'MiniBatchSize',12, 'InitialLearnRate',1e-3, 'MaxEpochs',120, 'ValidationData',preprocessedData);

[detector,info] = trainYOLOv2ObjectDetector(preprocessedData,lgraph,options);

Note: all these steps (designing, loading training data, setting training options, and training) can be done within the Deep Network Designer App. After the training, the optimized model needs to be exported to the workspace. An automatic function can be generated of the entire process in order to avoid the manual design if needed again.

Results

Evaluating YOLO detector on train images

Figure 5 shows an excellent performance of the trained YOLO detector on train images (only the first 15 images are shown).

Figure 5: Evaluating YOLO detector on train images

Evaluating YOLO detector on test images

Detection of ships on new images (test data set) shows worse performance, although several ships are correctly detected (Figure 6).

Figure 6: Evaluating YOLO detector on test images

Considering the very good performance on the train data set and the relatively insufficient performance on the test data set, the trained model is most probably overfitted. However, it demonstrates that the small data set can be used to detect some objects.

To improve the detection result on new images, the YOLO detector should be trained on a considerably larger dataset of images, and independent validation data set should be used to avoid overfitting. And finally, the training options (hyperparameters) can be further optimized to obtain a higher accuracy on the test data. This can be done using the Experiment Manager App which is a great tool to keep track of all the hyperparameter changes.

Key takeaways

This example presents the step-by-step development of a deep learning solution for the detection of ships on satellite images. The solution approach is quite general and can be adapted to different domain areas, similarly as this case study was adopted from an earlier MATLAB application of Object Detection Using YOLO v2 Deep Learning. A few conclusions and takeaways can be summarized:

• The trained YOLO network already successfully detects several ships in satellite images, although some ships on test images are not detected.
• Training data set of only 100 ship images is too small to train a ship detector from scratch, but by using the transfer learning approach, the required number of images is enough to quickly obtain first results.
• The example presents the potential workflow toward the solution, and in practice, many more annotated images (in order of hundreds or thousands of images) should be provided to train a detector with good detection accuracy.
• This material can inspire other lecturers to develop further Deep Learning-based examples within the framework of Maritime Engineering.

MATLAB code

The complete MATLAB code with a database of images is available at MATLAB Central File Exchange and GitHub:

• https://www.mathworks.com/matlabcentral/fileexchange/105830-detection-of-ships-on-satellite-images-using-yolo-v2
• https://github.com/ppotoc/Detection-of-ships-on-satellite-images-using-YOLO-v2-deep-learning