{"id":148,"date":"2018-02-14T14:43:21","date_gmt":"2018-02-14T14:43:21","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=148"},"modified":"2021-04-06T15:52:24","modified_gmt":"2021-04-06T19:52:24","slug":"create-a-simple-dag-network","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2018\/02\/14\/create-a-simple-dag-network\/","title":{"rendered":"Create a Simple DAG Network"},"content":{"rendered":"

Creating a Simple DAG Network<\/h3>

Today I want to show the basic tools needed to build your own DAG (directed acyclic graph) network for deep learning. I'm going to build this network and train it on our digits dataset.<\/p>

\"\" <\/p>

As the first step, I'll create the main branch, which follows the left path shown above. The layerGraph<\/tt> function will save a bunch of connection steps; it creates a graph network from a simple array of layers, connecting the layers together in order.<\/p>

layers = [\r\n    imageInputLayer([28 28 1],'Name'<\/span>,'input'<\/span>)\r\n\r\n    convolution2dLayer(5,16,'Padding'<\/span>,'same'<\/span>,'Name'<\/span>,'conv_1'<\/span>)\r\n    batchNormalizationLayer('Name'<\/span>,'BN_1'<\/span>)\r\n    reluLayer('Name'<\/span>,'relu_1'<\/span>)\r\n\r\n    convolution2dLayer(3,32,'Padding'<\/span>,'same'<\/span>,'Stride'<\/span>,2,'Name'<\/span>,'conv_2'<\/span>)\r\n    batchNormalizationLayer('Name'<\/span>,'BN_2'<\/span>)\r\n    reluLayer('Name'<\/span>,'relu_2'<\/span>)\r\n    convolution2dLayer(3,32,'Padding'<\/span>,'same'<\/span>,'Name'<\/span>,'conv_3'<\/span>)\r\n    batchNormalizationLayer('Name'<\/span>,'BN_3'<\/span>)\r\n    reluLayer('Name'<\/span>,'relu_3'<\/span>)\r\n\r\n    additionLayer(2,'Name'<\/span>,'add'<\/span>)\r\n\r\n    averagePooling2dLayer(2,'Stride'<\/span>,2,'Name'<\/span>,'avpool'<\/span>)\r\n    fullyConnectedLayer(10,'Name'<\/span>,'fc'<\/span>)\r\n    softmaxLayer('Name'<\/span>,'softmax'<\/span>)\r\n    classificationLayer('Name'<\/span>,'classOutput'<\/span>)];\r\n<\/pre>
Connect the layers using layerGraph<\/tt>.<\/p>
lgraph = layerGraph(layers)\r\n<\/pre>
\r\nlgraph = \r\n\r\n  LayerGraph with properties:\r\n\r\n         Layers: [15×1 nnet.cnn.layer.Layer]\r\n    Connections: [14×2 table]\r\n\r\n<\/pre>
The plot<\/tt> function knows how to visualize a DAG network.<\/p>
plot(lgraph)\r\naxis off<\/span>\r\n<\/pre>\"\" 
Next, I need to create the single layer that will be on the other branch. It is a 1-by-1 convolutional layer. Its parameters (number of filters and stride) are set to match the activation size of the 'relu_3'<\/tt> layer. Once the layer is created, addLayers<\/tt> adds it to lgraph<\/tt>.<\/p>
skip_conv_layer = convolution2dLayer(1,32,'Stride'<\/span>,2,'Name'<\/span>,'skipConv'<\/span>);\r\nlgraph = addLayers(lgraph,skip_conv_layer);\r\nplot(lgraph)\r\naxis off<\/span>\r\n<\/pre>\"\" 
You can see the new layer there, but it's looking a little lonely off to the side. I need to connect it to the other layers using connectLayers<\/tt>.<\/p>
lgraph = connectLayers(lgraph,'relu_1'<\/span>,'skipConv'<\/span>);\r\nlgraph = connectLayers(lgraph,'skipConv'<\/span>,'add\/in2'<\/span>);\r\nplot(lgraph);\r\naxis off<\/span>\r\n<\/pre>\"\" 
Now that I've constructed the network, here are the steps for training it using our digits dataset.<\/p>
[trainImages,trainLabels] = digitTrain4DArrayData;\r\n[valImages,valLabels] = digitTest4DArrayData;\r\n<\/pre>
Specify training options and train the network.<\/p>
options = trainingOptions('sgdm'<\/span>,...<\/span>\r\n    'MaxEpochs'<\/span>,6,...<\/span>\r\n    'Shuffle'<\/span>,'every-epoch'<\/span>,...<\/span>\r\n    'ValidationData'<\/span>,{valImages,valLabels},...<\/span>\r\n    'ValidationFrequency'<\/span>,20,...<\/span>\r\n    'VerboseFrequency'<\/span>,20);\r\nnet = trainNetwork(trainImages,trainLabels,lgraph,options)\r\n<\/pre>
Training on single GPU.\r\nInitializing image normalization.\r\n|=======================================================================================================================|\r\n|     Epoch    |   Iteration  | Time Elapsed |  Mini-batch  |  Validation  |  Mini-batch  |  Validation  | Base Learning|\r\n|              |              |  (seconds)   |     Loss     |     Loss     |   Accuracy   |   Accuracy   |     Rate     |\r\n|=======================================================================================================================|\r\n|            1 |            1 |         0.03 |       2.3317 |       2.2729 |       11.72% |       14.38% |       0.0100 |\r\n|            1 |           20 |         0.81 |       0.9775 |       0.9119 |       71.88% |       72.24% |       0.0100 |\r\n|            2 |           40 |         1.61 |       0.3783 |       0.4352 |       92.19% |       88.86% |       0.0100 |\r\n|            2 |           60 |         2.38 |       0.3752 |       0.3111 |       89.06% |       92.06% |       0.0100 |\r\n|            3 |           80 |         3.21 |       0.1935 |       0.1945 |       96.88% |       96.18% |       0.0100 |\r\n|            3 |          100 |         4.04 |       0.1777 |       0.1454 |       97.66% |       97.32% |       0.0100 |\r\n|            4 |          120 |         4.89 |       0.0662 |       0.0956 |      100.00% |       98.50% |       0.0100 |\r\n|            4 |          140 |         5.77 |       0.0764 |       0.0694 |       99.22% |       99.30% |       0.0100 |\r\n|            5 |          160 |         6.58 |       0.0466 |       0.0540 |      100.00% |       99.52% |       0.0100 |\r\n|            5 |          180 |         7.36 |       0.0459 |       0.0459 |       99.22% |       99.60% |       0.0100 |\r\n|            6 |          200 |         8.12 |       0.0276 |       0.0390 |      100.00% |       99.56% |       0.0100 |\r\n|            6 |          220 |         9.01 |       0.0242 |       0.0354 |      100.00% |       99.62% |       0.0100 |\r\n|            6 |          234 |         9.84 |       0.0160 |              |      100.00% |              |       0.0100 |\r\n|=======================================================================================================================|\r\n\r\nnet = \r\n\r\n  DAGNetwork with properties:\r\n\r\n         Layers: [16×1 nnet.cnn.layer.Layer]\r\n    Connections: [16×2 table]\r\n\r\n<\/pre>
The output of trainNetwork<\/tt> is a DAGNetwork<\/tt> object. The Connections<\/tt> property records how the individual layers are connected to each other.<\/p>
net.Connections\r\n<\/pre>
\r\nans =\r\n\r\n  16×2 table\r\n\r\n      Source       Destination \r\n    __________    _____________\r\n\r\n    'input'       'conv_1'     \r\n    'conv_1'      'BN_1'       \r\n    'BN_1'        'relu_1'     \r\n    'relu_1'      'conv_2'     \r\n    'relu_1'      'skipConv'   \r\n    'conv_2'      'BN_2'       \r\n    'BN_2'        'relu_2'     \r\n    'relu_2'      'conv_3'     \r\n    'conv_3'      'BN_3'       \r\n    'BN_3'        'relu_3'     \r\n    'relu_3'      'add\/in1'    \r\n    'add'         'avpool'     \r\n    'avpool'      'fc'         \r\n    'fc'          'softmax'    \r\n    'softmax'     'classOutput'\r\n    'skipConv'    'add\/in2'    \r\n\r\n<\/pre>
Classify the validation images and calculate the accuracy.<\/p>
predictedLabels = classify(net,valImages);\r\naccuracy = mean(predictedLabels == valLabels)\r\n<\/pre>
\r\naccuracy =\r\n\r\n    0.9974\r\n\r\n<\/pre>
Those are the basics. Create layers using the various layer functions and join them up using layerGraph<\/tt> and connectLayers<\/tt>. Then you can train and use the network in the same way you would train and use other networks.<\/p>
Reference: Create and Train a DAG Network for Deep Learning<\/a><\/p>