{"id":9585,"date":"2023-03-27T15:17:47","date_gmt":"2023-03-27T19:17:47","guid":{"rendered":"https:\/\/blogs.mathworks.com\/student-lounge\/?p=9585"},"modified":"2023-03-27T15:17:47","modified_gmt":"2023-03-27T19:17:47","slug":"mitigating-climate-change-through-deep-learning-in-matlab","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/student-lounge\/2023\/03\/27\/mitigating-climate-change-through-deep-learning-in-matlab\/","title":{"rendered":"Mitigating Climate Change through Deep Learning in MATLAB"},"content":{"rendered":"
\n
Joining us today is <\/span>Kaveh Faraji<\/span><\/a> and <\/span>Azin Al Kajbaf<\/span><\/a>, who won the Best Use of MATLAB award for <\/span>The BioMassters<\/span><\/a> competition! Read on to learn more about this duo and how they used deep learning for biomass estimation. Over to you guys…<\/span><\/div>\n
\"Azin.jpg\"\"Kaveh.jpg\"<\/div>\n
Azin got her Ph.D. in Civil Engineering from the University of Maryland in 2022 and is currently a postdoctoral research fellow at Johns Hopkins University and the National Institute of Standards and Technology (NIST). Kaveh is a Ph.D. candidate in Civil Engineering at the University of Maryland. Our area of academic focus involves the application of machine learning, geospatial analysis, and statistical methods in natural hazard assessment. We started to use deep learning a few years ago. We were implementing machine learning for our research at the time using MATLAB and were also interested in deep learning. MathWorks was sponsoring a deep learning competition at the time, which motivated us to learn about MATLAB\u2019s Deep Learning Toolbox and deep learning concepts in general. Since then, we have participated in multiple deep learning competitions. Currently, we are working on research projects involving machine learning and deep learning applications in the assessment of natural hazards. We have been planning to get some hands-on experience in the application of machine learning and deep learning in working with satellite imagery, which could ultimately be helpful in our research too, and this specific competition provided us with the perfect opportunity.<\/div>\n

Inspiration<\/h2>\n
We are a postdoctoral researcher and a Ph.D. student. We implement machine learning and deep learning methods in our research projects, and MATLAB is one of the programming languages that we use. Participating in data science competitions has become our passion since it motivates us to learn more about machine learning and deep learning methods and their real-world applications. The BioMassters<\/a> competition was particularly interesting because we recently started to work with satellite imagery for natural hazard assessment in our research.<\/div>\n

Breaking down the problem<\/h2>\n
The satellite imagery data contains 15 bands (11 bands for Sentinel-2 and 4 bands for Sentinel-1) for 12 months. The objective was to predict the yearly biomass of these images (pixel values of labels). The data has spatial and temporal aspects. Our idea was:<\/div>\n
    \n
  1. Perform pixel-by-pixel classification of images without considering the spatial effects (for this, we used 1-D CNN)<\/li>\n
  2. Then use a 3-D U-Net structure to consider the spatial relationship between pixels.<\/li>\n<\/ol>\n

    How did we implement it?<\/h2>\n
    This solution contains two main steps:<\/div>\n

    Step 1: Pixel-by-pixel regression with a 1-D CNN<\/h3>\n
    In this network, we read each image and used a custom training loop for training the network. We reshaped satellite imagery (with the size of [15 * 12 * 256 * 256]) to a matrix with the shape of [channel_size(C) = 15 batch_size(B) = (256*256) temporal_size(T) = 12]. We predicted labels for each image in training and testing datasets.<\/div>\n
    For example, you can see the code below, which we used to create and train a 1-D CNN that improved the score.<\/div>\n
    \n
    \n
    %% Buidling a 1-D CNN<\/span><\/span><\/div>\n<\/div>\n
    \n
    filterSize = 3;<\/span><\/div>\n<\/div>\n
    \n
    numFilters = 64;<\/span><\/div>\n<\/div>\n
    \n
    numClasses=10;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    layers = [ …<\/span><\/span><\/div>\n<\/div>\n
    \n
    sequenceInputLayer(15, MinLength=12)<\/span><\/div>\n<\/div>\n
    \n
    convolution1dLayer(filterSize,numFilters,Padding=“causal”<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    reluLayer<\/span><\/div>\n<\/div>\n
    \n
    layerNormalizationLayer<\/span><\/div>\n<\/div>\n
    \n
    maxPooling1dLayer(2)<\/span><\/div>\n<\/div>\n
    \n
    convolution1dLayer(filterSize,2*numFilters,Padding=“causal”<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    reluLayer<\/span><\/div>\n<\/div>\n
    \n
    layerNormalizationLayer<\/span><\/div>\n<\/div>\n
    \n
    globalMaxPooling1dLayer<\/span><\/div>\n<\/div>\n
    \n
    fullyConnectedLayer(512)<\/span><\/div>\n<\/div>\n
    \n
    reluLayer<\/span><\/div>\n<\/div>\n
    \n
    fullyConnectedLayer(128)<\/span><\/div>\n<\/div>\n
    \n
    reluLayer<\/span><\/div>\n<\/div>\n
    \n
    fullyConnectedLayer(1)<\/span><\/div>\n<\/div>\n
    \n
    reluLayer];<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % analyzeNetwork(layers);<\/span><\/span><\/div>\n<\/div>\n
    \n
    lgraph = layerGraph(layers);<\/span><\/div>\n<\/div>\n
    \n
    net = dlnetwork(lgraph);<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    %% Using the pre-trained model or training the model<\/span><\/span><\/div>\n<\/div>\n
    \n
    % If you want to train the model from scratch, change “train_network” value<\/span><\/span><\/div>\n<\/div>\n
    \n
    % to true.<\/span><\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    train_network = true;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    if <\/span>train_network<\/span><\/div>\n<\/div>\n
    \n
    miniBatchSize = 1;<\/span><\/div>\n<\/div>\n
    \n
    numEpochs = 10;<\/span><\/div>\n<\/div>\n
    \n
    numObservations = numel(inputTrain.Files);<\/span><\/div>\n<\/div>\n
    \n
    numIterationsPerEpoch = floor(numObservations.\/miniBatchSize);<\/span><\/div>\n<\/div>\n
    \n
    averageGrad = [];<\/span><\/div>\n<\/div>\n
    \n
    averageSqGrad = [];<\/span><\/div>\n<\/div>\n
    \n
    numIterations = numEpochs * numIterationsPerEpoch;<\/span><\/div>\n<\/div>\n
    \n
    monitor = trainingProgressMonitor(Metrics=“Loss”<\/span>,Info=“Epoch”<\/span>,XLabel=“Iteration”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    iteration = 0;<\/span><\/div>\n<\/div>\n
    \n
    epoch = 0;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    while <\/span>epoch < numEpochs && ~monitor.Stop<\/span><\/div>\n<\/div>\n
    \n
    epoch = epoch + 1;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Shuffle data.<\/span><\/span><\/div>\n<\/div>\n
    \n
    shuffle(mbq);<\/span><\/div>\n<\/div>\n
    \n
    reset(mbq_val);<\/span><\/div>\n<\/div>\n
    \n
    while <\/span>hasdata(mbq) && ~monitor.Stop<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    iteration = iteration + 1;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Read mini-batch of data.<\/span><\/span><\/div>\n<\/div>\n
    \n
    [X,T] = next(mbq);<\/span><\/div>\n<\/div>\n
    \n
    % Convert mini-batch of data to a dlarray.<\/span><\/span><\/div>\n<\/div>\n
    \n
    X = dlarray(single(X),“CBT”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    % We read each image with the size of [15 * 12 * 256 * 256] and<\/span><\/span><\/div>\n<\/div>\n
    \n
    % convert it to a [channel_size(C) = 15 batch_size(B) = (256*256) temporal_size(T) = 12]<\/span><\/span><\/div>\n<\/div>\n
    \n
    % We had to use a batch size smaller than 65501.<\/span><\/span><\/div>\n<\/div>\n
    \n
    % We got errors when we used batch size above this value.<\/span><\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    X = X(:,1:65500,:);<\/span><\/div>\n<\/div>\n
    \n
    T = T(:,1:65500,:);<\/span><\/div>\n<\/div>\n
    \n
    % If training on a GPU, then convert data to a gpuArray.<\/span><\/span><\/div>\n<\/div>\n
    \n
    if <\/span>canUseGPU<\/span><\/div>\n<\/div>\n
    \n
    X = gpuArray(X);<\/span><\/div>\n<\/div>\n
    \n
    T= gpuArray(T);<\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Calculate loss and gradients using the helper loss function.<\/span><\/span><\/div>\n<\/div>\n
    \n
    [loss,gradients] = dlfeval(@modelLoss,net,X,T);<\/span><\/div>\n<\/div>\n
    \n
    % Update the network parameters using the Adam optimizer.<\/span><\/span><\/div>\n<\/div>\n
    \n
    [net,averageGrad,averageSqGrad] = adamupdate(net,gradients,averageGrad,averageSqGrad,iteration);<\/span><\/div>\n<\/div>\n
    \n
    % Update the training progress monitor.<\/span><\/span><\/div>\n<\/div>\n
    \n
    recordMetrics(monitor,iteration,Loss=loss);<\/span><\/div>\n<\/div>\n
    \n
    updateInfo(monitor,Epoch=epoch + ” of ” <\/span>+ numEpochs);<\/span><\/div>\n<\/div>\n
    \n
    monitor.Progress = 100 * iteration\/numIterations;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Validation error<\/span><\/span><\/div>\n<\/div>\n
    \n
    ii=0;<\/span><\/div>\n<\/div>\n
    \n
    while <\/span>hasdata(mbq_val)<\/span><\/div>\n<\/div>\n
    \n
    [X_val, T_val]= next(mbq_val);<\/span><\/div>\n<\/div>\n
    \n
    if <\/span>canUseGPU<\/span><\/div>\n<\/div>\n
    \n
    X_val = dlarray(single(X_val),“CBT”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    T_val = gpuArray(T_val);<\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n
    \n
    Y_val = predict(net,X_val);<\/span><\/div>\n<\/div>\n
    \n
    error = mse(Y_val, T_val)^.5;<\/span><\/div>\n<\/div>\n
    \n
    ii=ii+1;<\/span><\/div>\n<\/div>\n
    \n
    rmse_error(ii) = extractdata(gather(error));<\/span><\/div>\n<\/div>\n
    \n
    % disp(rmse_error(ii))<\/span><\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n
    \n
    disp([‘Epoch’<\/span>+ string(epoch)+‘ Validation Error(RMSE): ‘ <\/span>, mean(rmse_error)])<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n
    \n
    save(‘trainedNetwork_conv1d_submit.mat’<\/span>,‘net’<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    else<\/span><\/span><\/div>\n<\/div>\n
    \n
    lgraph = load(‘trainedNetwork_conv1d.mat’<\/span>);% Load pre-trained network<\/span><\/span><\/div>\n<\/div>\n
    \n
    net = lgraph.net;<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n
    \n
    \n
    \n
    <\/div>\n
    <\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

    Step 2: Using a 3-D U-Net Model<\/h3>\n
    Next, we used a 3-D U-Net model and provided it with inputs with the shape of [16*12*256*256]. The 16th channel in the input of the U-Net network is the labels generated in step 1. You can see more details about this model in the GIF below:<\/div>\n
    \"DDWinnersFA22.gif\"<\/div>\n

    Results<\/h2>\n
    In the below figure, you can see a summary of our solution’s framework and the final score.<\/div>\n
    \"Framework.jpg\"<\/div>\n

    Key Takeaways<\/h2>\n
    This competition provided us with the opportunity to employ deep learning in a new area. We figured that for a project of this nature with the purpose of assigning values or labels to satellite imagery pixels, an ensemble model is required to achieve better performance. The ensemble could include a model that performs pixel-by-pixel prediction and a model that considers the effect of surrounding pixels (3D U-net). We also would like to highlight our experience working with MATLAB’s custom network training loop, which was flexible and intuitive. We had experience working with MATLAB toolboxes in our previous projects. We find the MATLAB Deep Learning Toolbox user-friendly, and most of the time we can find a solution to our problems using documentation and MATLAB Answers.<\/div>\n
    We want to thank MathWorks for sponsoring this competition. It provided a great opportunity to get hands-on experience working with satellite imagery with spatial and temporal aspects and learn more about MATLAB’s Deep Learning Toolbox capabilities.<\/div>\n<\/div>\n