{"id":182,"date":"2018-03-19T17:41:27","date_gmt":"2018-03-19T17:41:27","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=182"},"modified":"2021-04-06T15:52:19","modified_gmt":"2021-04-06T19:52:19","slug":"creating-a-dag-network-from-dag-parts","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2018\/03\/19\/creating-a-dag-network-from-dag-parts\/","title":{"rendered":"Creating a DAG Network from DAG Parts"},"content":{"rendered":"

In my 14-Feb-2018 blog post<\/a> about creating a simple DAG network, reader Daniel Morris wanted to know if there's a less tedious way, compared to adding layers one at a time, to combine two (or more) DAGs into a network. I asked the development team about this. I learned that, although it is not yet as easy as they would like to make it, one can write a couple of utility functions to make the task easier.<\/p>

The first utility function, createLgraphUsingConnections<\/tt>, is used by a new doc example, Transfer Learning Using GoogLeNet<\/a>, that was added to R2018a. The function is included at the bottom of this blog post.ame)<\/em><\/p>

The second utility function, subgraphConnections<\/tt>, is something I wrote for this post. It handles multiple-input and multiple-output situations when merging two DAGs. This function is also at the bottom of this post.<\/p>

I will illustrate these function by using them to recreate a portion of the structure of GoogLeNet.<\/p>

net = googlenet;\r\nplot(net)\r\naxis off<\/span>\r\nxlim([0 5])\r\nylim([45 65])\r\n<\/pre>\"\" 
Suppose we make six different sub-DAGs: the input branch, the four branches of inception layer 3a, and the 3a output layer. Here's one way to make these sub-DAGs.<\/p>
subdag1 = layerGraph([ ...<\/span>\r\n    imageInputLayer([224 224 3],'Name'<\/span>,'data'<\/span>)\r\n    convolution2dLayer([7 7],64,'NumChannels'<\/span>,3,'Stride'<\/span>,[2 2],'Padding'<\/span>,[3 3 3 3],'Name'<\/span>,'conv1-7x7_s2'<\/span>)\r\n    reluLayer('Name'<\/span>,'conv1-relu_7x7'<\/span>)\r\n    maxPooling2dLayer([3 3],'Stride'<\/span>,[2 2],'Padding'<\/span>,[0 1 0 1],'Name'<\/span>,'pool1-3x3_s2'<\/span>)\r\n    crossChannelNormalizationLayer(5,'K'<\/span>,1,'Name'<\/span>,'pool1-normal'<\/span>)\r\n    convolution2dLayer([1 1],64,'NumChannels'<\/span>,64,'Name'<\/span>,'conv2-3x3_reduce'<\/span>)\r\n    reluLayer('Name'<\/span>,'conv2-relu_3x3_reduce'<\/span>)\r\n    convolution2dLayer([3 3],192,'NumChannels'<\/span>,64,'Padding'<\/span>,[1 1 1 1],'Name'<\/span>,'conv2-3x3'<\/span>)\r\n    reluLayer('Name'<\/span>,'conv2-relu_3x3'<\/span>)\r\n    crossChannelNormalizationLayer(5,'K'<\/span>,1,'Name'<\/span>,'conv2-norm2'<\/span>)\r\n    maxPooling2dLayer([3 3],'Stride'<\/span>,[2 2],'Name'<\/span>,'pool2-3x3_s2'<\/span>)\r\n    ]);\r\n\r\nsubdag2 = layerGraph([ ...<\/span>\r\n    convolution2dLayer([1 1],64,'NumChannels'<\/span>,192,'Name'<\/span>,'inception_3a-1x1'<\/span>)\r\n    reluLayer('Name'<\/span>,'inception_3a-relu_1x1'<\/span>)]);\r\n\r\nsubdag3 = layerGraph([ ...<\/span>\r\n    convolution2dLayer([1 1],96,'NumChannels'<\/span>,192,'Name'<\/span>,'inception_3a-3x3_reduce'<\/span>)\r\n    reluLayer('Name'<\/span>,'inception_3a-relu_3x3_reduce'<\/span>)\r\n    convolution2dLayer([3 3],128,'NumChannels'<\/span>,96,'Padding'<\/span>,[1 1 1 1],'Name'<\/span>,'inception_3a-3x3'<\/span>)\r\n    reluLayer('Name'<\/span>,'inception_3a-relu_3x3'<\/span>)]);\r\n\r\nsubdag4 = layerGraph([ ...<\/span>\r\n    convolution2dLayer([1 1],16,'NumChannels'<\/span>,192,'Name'<\/span>,'inception_3a-5x5_reduce'<\/span>)\r\n    reluLayer('Name'<\/span>,'inception_3a-relu_5x5_reduce'<\/span>)\r\n    convolution2dLayer([5 5],32,'NumChannels'<\/span>,16,'Padding'<\/span>,[2 2 2 2],'Name'<\/span>,'inception_3a-5x5'<\/span>)\r\n    reluLayer('Name'<\/span>,'inception_3a-relu_5x5'<\/span>)]);\r\n\r\nsubdag5 = layerGraph([ ...<\/span>\r\n    maxPooling2dLayer([3 3],'Padding'<\/span>,[1 1 1 1],'Name'<\/span>,'inception_3a-pool'<\/span>)\r\n    convolution2dLayer([1 1],32,'NumChannels'<\/span>,192,'Name'<\/span>,'inception_3a-pool_proj'<\/span>)\r\n    reluLayer('Name'<\/span>,'inception_3a-relu_pool_proj'<\/span>)]);\r\n\r\nsubdag6 = layerGraph(depthConcatenationLayer(4,'Name'<\/span>,'inception_3a-output'<\/span>));\r\n<\/pre>
Let's combine all of the layers.<\/p>
all_layers = [ ...<\/span>\r\n    subdag1.Layers\r\n    subdag2.Layers\r\n    subdag3.Layers\r\n    subdag4.Layers\r\n    subdag5.Layers\r\n    subdag6.Layers ]\r\n<\/pre>
\r\nall_layers = \r\n\r\n  25x1 Layer array with layers:\r\n\r\n     1   'data'                           Image Input                   224x224x3 images with 'zerocenter' normalization\r\n     2   'conv1-7x7_s2'                   Convolution                   64 7x7x3 convolutions with stride [2  2] and padding [3  3  3  3]\r\n     3   'conv1-relu_7x7'                 ReLU                          ReLU\r\n     4   'pool1-3x3_s2'                   Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  1  0  1]\r\n     5   'pool1-normal'                   Cross Channel Normalization   cross channel normalization with 5 channels per element\r\n     6   'conv2-3x3_reduce'               Convolution                   64 1x1x64 convolutions with stride [1  1] and padding [0  0  0  0]\r\n     7   'conv2-relu_3x3_reduce'          ReLU                          ReLU\r\n     8   'conv2-3x3'                      Convolution                   192 3x3x64 convolutions with stride [1  1] and padding [1  1  1  1]\r\n     9   'conv2-relu_3x3'                 ReLU                          ReLU\r\n    10   'conv2-norm2'                    Cross Channel Normalization   cross channel normalization with 5 channels per element\r\n    11   'pool2-3x3_s2'                   Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  0  0  0]\r\n    12   'inception_3a-1x1'               Convolution                   64 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]\r\n    13   'inception_3a-relu_1x1'          ReLU                          ReLU\r\n    14   'inception_3a-3x3_reduce'        Convolution                   96 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]\r\n    15   'inception_3a-relu_3x3_reduce'   ReLU                          ReLU\r\n    16   'inception_3a-3x3'               Convolution                   128 3x3x96 convolutions with stride [1  1] and padding [1  1  1  1]\r\n    17   'inception_3a-relu_3x3'          ReLU                          ReLU\r\n    18   'inception_3a-5x5_reduce'        Convolution                   16 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]\r\n    19   'inception_3a-relu_5x5_reduce'   ReLU                          ReLU\r\n    20   'inception_3a-5x5'               Convolution                   32 5x5x16 convolutions with stride [1  1] and padding [2  2  2  2]\r\n    21   'inception_3a-relu_5x5'          ReLU                          ReLU\r\n    22   'inception_3a-pool'              Max Pooling                   3x3 max pooling with stride [1  1] and padding [1  1  1  1]\r\n    23   'inception_3a-pool_proj'         Convolution                   32 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]\r\n    24   'inception_3a-relu_pool_proj'    ReLU                          ReLU\r\n    25   'inception_3a-output'            Depth concatenation           Depth concatenation of 4 inputs\r\n<\/pre>
Most of the connections in the final DAG already exist in the subDAGs. Let's grab those and combine them into one list of connections.<\/p>
all_connections = [ ...<\/span>\r\n    subdag1.Connections\r\n    subdag2.Connections\r\n    subdag3.Connections\r\n    subdag4.Connections\r\n    subdag5.Connections\r\n    subdag6.Connections]\r\n<\/pre>
\r\nall_connections =\r\n\r\n  19×2 table\r\n\r\n                Source                         Destination          \r\n    ______________________________    ______________________________\r\n\r\n    'data'                            'conv1-7x7_s2'                \r\n    'conv1-7x7_s2'                    'conv1-relu_7x7'              \r\n    'conv1-relu_7x7'                  'pool1-3x3_s2'                \r\n    'pool1-3x3_s2'                    'pool1-normal'                \r\n    'pool1-normal'                    'conv2-3x3_reduce'            \r\n    'conv2-3x3_reduce'                'conv2-relu_3x3_reduce'       \r\n    'conv2-relu_3x3_reduce'           'conv2-3x3'                   \r\n    'conv2-3x3'                       'conv2-relu_3x3'              \r\n    'conv2-relu_3x3'                  'conv2-norm2'                 \r\n    'conv2-norm2'                     'pool2-3x3_s2'                \r\n    'inception_3a-1x1'                'inception_3a-relu_1x1'       \r\n    'inception_3a-3x3_reduce'         'inception_3a-relu_3x3_reduce'\r\n    'inception_3a-relu_3x3_reduce'    'inception_3a-3x3'            \r\n    'inception_3a-3x3'                'inception_3a-relu_3x3'       \r\n    'inception_3a-5x5_reduce'         'inception_3a-relu_5x5_reduce'\r\n    'inception_3a-relu_5x5_reduce'    'inception_3a-5x5'            \r\n    'inception_3a-5x5'                'inception_3a-relu_5x5'       \r\n    'inception_3a-pool'               'inception_3a-pool_proj'      \r\n    'inception_3a-pool_proj'          'inception_3a-relu_pool_proj' \r\n\r\n<\/pre>
Now we have to get the connections that join the various subDAGs together. This is where subgraphConnections<\/tt> comes in handy.<\/p>
First, get all the desired connections between the last layer of subDAG 1 and the first layer of subDAGs 2-5.<\/p>
new_connections = subgraphConnections(subdag1.Layers(end),...<\/span>\r\n    [subdag2.Layers(1) subdag3.Layers(1) subdag4.Layers(1) subdag5.Layers(1)])\r\nall_connections = [all_connections ; new_connections];\r\n<\/pre>
\r\nnew_connections =\r\n\r\n  4×2 table\r\n\r\n        Source               Destination       \r\n    ______________    _________________________\r\n\r\n    'pool2-3x3_s2'    'inception_3a-1x1'       \r\n    'pool2-3x3_s2'    'inception_3a-3x3_reduce'\r\n    'pool2-3x3_s2'    'inception_3a-5x5_reduce'\r\n    'pool2-3x3_s2'    'inception_3a-pool'      \r\n\r\n<\/pre>
Now get the desired connections between the last layer of subDAGs 2-5 and the first (and only) layer of subDAG 6.<\/p>
src_layers = [subdag2.Layers(end) subdag3.Layers(end) subdag4.Layers(end) ...<\/span>\r\n    subdag5.Layers(end)];\r\nnew_connections = subgraphConnections(src_layers,subdag6.Layers(1))\r\n<\/pre>
\r\nnew_connections =\r\n\r\n  4×2 table\r\n\r\n               Source                       Destination       \r\n    _____________________________    _________________________\r\n\r\n    'inception_3a-relu_1x1'          'inception_3a-output\/in1'\r\n    'inception_3a-relu_3x3'          'inception_3a-output\/in2'\r\n    'inception_3a-relu_5x5'          'inception_3a-output\/in3'\r\n    'inception_3a-relu_pool_proj'    'inception_3a-output\/in4'\r\n\r\n<\/pre>
Note that the utility function subgraphConnections<\/tt> automatically adds the input specifiers for a multiple-input layer.<\/p>
new_connections.Destination\r\nall_connections = [all_connections ; new_connections];\r\n<\/pre>
\r\nans =\r\n\r\n  4×1 cell array\r\n\r\n    {'inception_3a-output\/in1'}\r\n    {'inception_3a-output\/in2'}\r\n    {'inception_3a-output\/in3'}\r\n    {'inception_3a-output\/in4'}\r\n\r\n<\/pre>
Finally, use createLgraphUsingConnections<\/tt> to create the merged DAG.<\/p>
lgraph = createLgraphUsingConnections(all_layers,all_connections);\r\nplot(lgraph)\r\naxis off<\/span><\/pre>\r\n\r\n
\r\n\"\" \r\n<\/p>\r\n\r\n
The code below includes the utility functions used above.<\/i><\/p>
function<\/span> lgraph = createLgraphUsingConnections(layers,connections)\r\n% lgraph = createLgraphUsingConnections(layers,connections) creates a layer<\/span>\r\n% graph with the layers in the layer array |layers| connected by the<\/span>\r\n% connections in |connections|.<\/span>\r\n\r\nlgraph = layerGraph();\r\nfor<\/span> i = 1:numel(layers)\r\n    lgraph = addLayers(lgraph,layers(i));\r\nend<\/span>\r\n\r\nfor<\/span> c = 1:size(connections,1)\r\n    lgraph = connectLayers(lgraph,connections.Source{c},connections.Destination{c});\r\nend<\/span>\r\n\r\nend<\/span>\r\n\r\nfunction<\/span> c = subgraphConnections(src,dest)\r\nif<\/span> numel(src) > 1\r\n    dest_name_root = [dest.Name '\/in'<\/span>];\r\n    tmp = cell(numel(src),2);\r\n    for<\/span> k = 1:numel(src)\r\n        tmp{k,1} = src(k).Name;\r\n        tmp{k,2} = [dest_name_root num2str(k)];\r\n    end<\/span>\r\nelseif<\/span> numel(dest) > 1\r\n    tmp = cell(numel(dest),2);\r\n    for<\/span> k = 1:numel(dest)\r\n        tmp{k,1} = src.Name;\r\n        tmp{k,2} = dest(k).Name;\r\n    end<\/span>\r\nend<\/span>\r\nc = table(tmp(:,1),tmp(:,2),'VariableNames'<\/span>,{'Source'<\/span>,'Destination'<\/span>});\r\nend<\/span>\r\n<\/pre>