Deep Learning

Understanding and using deep learning networks

Create a Simple DAG Network 4

Posted by Steve Eddins,

Creating a Simple DAG Network

Today I want to show the basic tools needed to build your own DAG (directed acyclic graph) network for deep learning. I'm going to build this network and train it on our digits dataset.

As the first step, I'll create the main branch, which follows the left path shown above. The layerGraph function will save a bunch of connection steps; it creates a graph network from a simple array of layers, connecting the layers together in order.

layers = [
    imageInputLayer([28 28 1],'Name','input')

    convolution2dLayer(5,16,'Padding','same','Name','conv_1')
    batchNormalizationLayer('Name','BN_1')
    reluLayer('Name','relu_1')

    convolution2dLayer(3,32,'Padding','same','Stride',2,'Name','conv_2')
    batchNormalizationLayer('Name','BN_2')
    reluLayer('Name','relu_2')
    convolution2dLayer(3,32,'Padding','same','Name','conv_3')
    batchNormalizationLayer('Name','BN_3')
    reluLayer('Name','relu_3')

    additionLayer(2,'Name','add')

    averagePooling2dLayer(2,'Stride',2,'Name','avpool')
    fullyConnectedLayer(10,'Name','fc')
    softmaxLayer('Name','softmax')
    classificationLayer('Name','classOutput')];

Connect the layers using layerGraph.

lgraph = layerGraph(layers)
lgraph = 

  LayerGraph with properties:

         Layers: [15×1 nnet.cnn.layer.Layer]
    Connections: [14×2 table]

The plot function knows how to visualize a DAG network.

plot(lgraph)
axis off

Next, I need to create the single layer that will be on the other branch. It is a 1-by-1 convolutional layer. Its parameters (number of filters and stride) are set to match the activation size of the 'relu_3' layer. Once the layer is created, addLayers adds it to lgraph.

skip_conv_layer = convolution2dLayer(1,32,'Stride',2,'Name','skipConv');
lgraph = addLayers(lgraph,skip_conv_layer);
plot(lgraph)
axis off

You can see the new layer there, but it's looking a little lonely off to the side. I need to connect it to the other layers using connectLayers.

lgraph = connectLayers(lgraph,'relu_1','skipConv');
lgraph = connectLayers(lgraph,'skipConv','add/in2');
plot(lgraph);
axis off

Now that I've constructed the network, here are the steps for training it using our digits dataset.

[trainImages,trainLabels] = digitTrain4DArrayData;
[valImages,valLabels] = digitTest4DArrayData;

Specify training options and train the network.

options = trainingOptions('sgdm',...
    'MaxEpochs',6,...
    'Shuffle','every-epoch',...
    'ValidationData',{valImages,valLabels},...
    'ValidationFrequency',20,...
    'VerboseFrequency',20);
net = trainNetwork(trainImages,trainLabels,lgraph,options)
Training on single GPU.
Initializing image normalization.
|=======================================================================================================================|
|     Epoch    |   Iteration  | Time Elapsed |  Mini-batch  |  Validation  |  Mini-batch  |  Validation  | Base Learning|
|              |              |  (seconds)   |     Loss     |     Loss     |   Accuracy   |   Accuracy   |     Rate     |
|=======================================================================================================================|
|            1 |            1 |         0.03 |       2.3317 |       2.2729 |       11.72% |       14.38% |       0.0100 |
|            1 |           20 |         0.81 |       0.9775 |       0.9119 |       71.88% |       72.24% |       0.0100 |
|            2 |           40 |         1.61 |       0.3783 |       0.4352 |       92.19% |       88.86% |       0.0100 |
|            2 |           60 |         2.38 |       0.3752 |       0.3111 |       89.06% |       92.06% |       0.0100 |
|            3 |           80 |         3.21 |       0.1935 |       0.1945 |       96.88% |       96.18% |       0.0100 |
|            3 |          100 |         4.04 |       0.1777 |       0.1454 |       97.66% |       97.32% |       0.0100 |
|            4 |          120 |         4.89 |       0.0662 |       0.0956 |      100.00% |       98.50% |       0.0100 |
|            4 |          140 |         5.77 |       0.0764 |       0.0694 |       99.22% |       99.30% |       0.0100 |
|            5 |          160 |         6.58 |       0.0466 |       0.0540 |      100.00% |       99.52% |       0.0100 |
|            5 |          180 |         7.36 |       0.0459 |       0.0459 |       99.22% |       99.60% |       0.0100 |
|            6 |          200 |         8.12 |       0.0276 |       0.0390 |      100.00% |       99.56% |       0.0100 |
|            6 |          220 |         9.01 |       0.0242 |       0.0354 |      100.00% |       99.62% |       0.0100 |
|            6 |          234 |         9.84 |       0.0160 |              |      100.00% |              |       0.0100 |
|=======================================================================================================================|

net = 

  DAGNetwork with properties:

         Layers: [16×1 nnet.cnn.layer.Layer]
    Connections: [16×2 table]

The output of trainNetwork is a DAGNetwork object. The Connections property records how the individual layers are connected to each other.

net.Connections
ans =

  16×2 table

      Source       Destination 
    __________    _____________

    'input'       'conv_1'     
    'conv_1'      'BN_1'       
    'BN_1'        'relu_1'     
    'relu_1'      'conv_2'     
    'relu_1'      'skipConv'   
    'conv_2'      'BN_2'       
    'BN_2'        'relu_2'     
    'relu_2'      'conv_3'     
    'conv_3'      'BN_3'       
    'BN_3'        'relu_3'     
    'relu_3'      'add/in1'    
    'add'         'avpool'     
    'avpool'      'fc'         
    'fc'          'softmax'    
    'softmax'     'classOutput'
    'skipConv'    'add/in2'    

Classify the validation images and calculate the accuracy.

predictedLabels = classify(net,valImages);
accuracy = mean(predictedLabels == valLabels)
accuracy =

    0.9974

Those are the basics. Create layers using the various layer functions and join them up using layerGraph and connectLayers. Then you can train and use the network in the same way you would train and use other networks.

Reference: Create and Train a DAG Network for Deep Learning


Get the MATLAB code

Published with MATLAB® R2017b

4 CommentsOldest to Newest

Daniel Morris replied on : 2 of 4

This is a nice example, although rather than adding layers to a LayerGraph (which is tedious), I would like to define separate LayerGraph networks and then combine them / connect them together into a single LayerGraph. Is this possible?

Paresh Kamble replied on : 3 of 4

Dear Sir,
How can I create a CNN with two outputs using DAG? One performing classification and another regression. Latest Matlab release 2018a pretends to allows the same but with no decent explanation. The link https://in.mathworks.com/solutions/deep-learning/features.html#data_preparation_and_labeling has a diagram depicting the possibility. Kindly provide a small tutorial on the use of DAG in Deep Learning with multiple input and outputs. Thank you!

Add A Comment

Your email address will not be published. Required fields are marked *

Preview: hide