Deep Learning

Understanding and using deep learning networks

This is machine translation

Translated by Microsoft
Mouseover text to see original. Click the button below to return to the Original version of the page.

Creating a DAG Network from DAG Parts 4

Posted by Steve Eddins,

In my 14-Feb-2018 blog post about creating a simple DAG network, reader Daniel Morris wanted to know if there's a less tedious way, compared to adding layers one at a time, to combine two (or more) DAGs into a network. I asked the development team about this. I learned that, although it is not yet as easy as they would like to make it, one can write a couple of utility functions to make the task easier.

The first utility function, createLgraphUsingConnections, is used by a new doc example, Transfer Learning Using GoogLeNet, that was added to R2018a. The function is included at the bottom of this blog post.ame)

The second utility function, subgraphConnections, is something I wrote for this post. It handles multiple-input and multiple-output situations when merging two DAGs. This function is also at the bottom of this post.

I will illustrate these function by using them to recreate a portion of the structure of GoogLeNet.

net = googlenet;
axis off
xlim([0 5])
ylim([45 65])

Suppose we make six different sub-DAGs: the input branch, the four branches of inception layer 3a, and the 3a output layer. Here's one way to make these sub-DAGs.

subdag1 = layerGraph([ ...
    imageInputLayer([224 224 3],'Name','data')
    convolution2dLayer([7 7],64,'NumChannels',3,'Stride',[2 2],'Padding',[3 3 3 3],'Name','conv1-7x7_s2')
    maxPooling2dLayer([3 3],'Stride',[2 2],'Padding',[0 1 0 1],'Name','pool1-3x3_s2')
    convolution2dLayer([1 1],64,'NumChannels',64,'Name','conv2-3x3_reduce')
    convolution2dLayer([3 3],192,'NumChannels',64,'Padding',[1 1 1 1],'Name','conv2-3x3')
    maxPooling2dLayer([3 3],'Stride',[2 2],'Name','pool2-3x3_s2')

subdag2 = layerGraph([ ...
    convolution2dLayer([1 1],64,'NumChannels',192,'Name','inception_3a-1x1')

subdag3 = layerGraph([ ...
    convolution2dLayer([1 1],96,'NumChannels',192,'Name','inception_3a-3x3_reduce')
    convolution2dLayer([3 3],128,'NumChannels',96,'Padding',[1 1 1 1],'Name','inception_3a-3x3')

subdag4 = layerGraph([ ...
    convolution2dLayer([1 1],16,'NumChannels',192,'Name','inception_3a-5x5_reduce')
    convolution2dLayer([5 5],32,'NumChannels',16,'Padding',[2 2 2 2],'Name','inception_3a-5x5')

subdag5 = layerGraph([ ...
    maxPooling2dLayer([3 3],'Padding',[1 1 1 1],'Name','inception_3a-pool')
    convolution2dLayer([1 1],32,'NumChannels',192,'Name','inception_3a-pool_proj')

subdag6 = layerGraph(depthConcatenationLayer(4,'Name','inception_3a-output'));

Let's combine all of the layers.

all_layers = [ ...
    subdag6.Layers ]
all_layers = 

  25x1 Layer array with layers:

     1   'data'                           Image Input                   224x224x3 images with 'zerocenter' normalization
     2   'conv1-7x7_s2'                   Convolution                   64 7x7x3 convolutions with stride [2  2] and padding [3  3  3  3]
     3   'conv1-relu_7x7'                 ReLU                          ReLU
     4   'pool1-3x3_s2'                   Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  1  0  1]
     5   'pool1-normal'                   Cross Channel Normalization   cross channel normalization with 5 channels per element
     6   'conv2-3x3_reduce'               Convolution                   64 1x1x64 convolutions with stride [1  1] and padding [0  0  0  0]
     7   'conv2-relu_3x3_reduce'          ReLU                          ReLU
     8   'conv2-3x3'                      Convolution                   192 3x3x64 convolutions with stride [1  1] and padding [1  1  1  1]
     9   'conv2-relu_3x3'                 ReLU                          ReLU
    10   'conv2-norm2'                    Cross Channel Normalization   cross channel normalization with 5 channels per element
    11   'pool2-3x3_s2'                   Max Pooling                   3x3 max pooling with stride [2  2] and padding [0  0  0  0]
    12   'inception_3a-1x1'               Convolution                   64 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]
    13   'inception_3a-relu_1x1'          ReLU                          ReLU
    14   'inception_3a-3x3_reduce'        Convolution                   96 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]
    15   'inception_3a-relu_3x3_reduce'   ReLU                          ReLU
    16   'inception_3a-3x3'               Convolution                   128 3x3x96 convolutions with stride [1  1] and padding [1  1  1  1]
    17   'inception_3a-relu_3x3'          ReLU                          ReLU
    18   'inception_3a-5x5_reduce'        Convolution                   16 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]
    19   'inception_3a-relu_5x5_reduce'   ReLU                          ReLU
    20   'inception_3a-5x5'               Convolution                   32 5x5x16 convolutions with stride [1  1] and padding [2  2  2  2]
    21   'inception_3a-relu_5x5'          ReLU                          ReLU
    22   'inception_3a-pool'              Max Pooling                   3x3 max pooling with stride [1  1] and padding [1  1  1  1]
    23   'inception_3a-pool_proj'         Convolution                   32 1x1x192 convolutions with stride [1  1] and padding [0  0  0  0]
    24   'inception_3a-relu_pool_proj'    ReLU                          ReLU
    25   'inception_3a-output'            Depth concatenation           Depth concatenation of 4 inputs

Most of the connections in the final DAG already exist in the subDAGs. Let's grab those and combine them into one list of connections.

all_connections = [ ...
all_connections =

  19×2 table

                Source                         Destination          
    ______________________________    ______________________________

    'data'                            'conv1-7x7_s2'                
    'conv1-7x7_s2'                    'conv1-relu_7x7'              
    'conv1-relu_7x7'                  'pool1-3x3_s2'                
    'pool1-3x3_s2'                    'pool1-normal'                
    'pool1-normal'                    'conv2-3x3_reduce'            
    'conv2-3x3_reduce'                'conv2-relu_3x3_reduce'       
    'conv2-relu_3x3_reduce'           'conv2-3x3'                   
    'conv2-3x3'                       'conv2-relu_3x3'              
    'conv2-relu_3x3'                  'conv2-norm2'                 
    'conv2-norm2'                     'pool2-3x3_s2'                
    'inception_3a-1x1'                'inception_3a-relu_1x1'       
    'inception_3a-3x3_reduce'         'inception_3a-relu_3x3_reduce'
    'inception_3a-relu_3x3_reduce'    'inception_3a-3x3'            
    'inception_3a-3x3'                'inception_3a-relu_3x3'       
    'inception_3a-5x5_reduce'         'inception_3a-relu_5x5_reduce'
    'inception_3a-relu_5x5_reduce'    'inception_3a-5x5'            
    'inception_3a-5x5'                'inception_3a-relu_5x5'       
    'inception_3a-pool'               'inception_3a-pool_proj'      
    'inception_3a-pool_proj'          'inception_3a-relu_pool_proj' 

Now we have to get the connections that join the various subDAGs together. This is where subgraphConnections comes in handy.

First, get all the desired connections between the last layer of subDAG 1 and the first layer of subDAGs 2-5.

new_connections = subgraphConnections(subdag1.Layers(end),...
    [subdag2.Layers(1) subdag3.Layers(1) subdag4.Layers(1) subdag5.Layers(1)])
all_connections = [all_connections ; new_connections];
new_connections =

  4×2 table

        Source               Destination       
    ______________    _________________________

    'pool2-3x3_s2'    'inception_3a-1x1'       
    'pool2-3x3_s2'    'inception_3a-3x3_reduce'
    'pool2-3x3_s2'    'inception_3a-5x5_reduce'
    'pool2-3x3_s2'    'inception_3a-pool'      

Now get the desired connections between the last layer of subDAGs 2-5 and the first (and only) layer of subDAG 6.

src_layers = [subdag2.Layers(end) subdag3.Layers(end) subdag4.Layers(end) ...
new_connections = subgraphConnections(src_layers,subdag6.Layers(1))
new_connections =

  4×2 table

               Source                       Destination       
    _____________________________    _________________________

    'inception_3a-relu_1x1'          'inception_3a-output/in1'
    'inception_3a-relu_3x3'          'inception_3a-output/in2'
    'inception_3a-relu_5x5'          'inception_3a-output/in3'
    'inception_3a-relu_pool_proj'    'inception_3a-output/in4'

Note that the utility function subgraphConnections automatically adds the input specifiers for a multiple-input layer.

all_connections = [all_connections ; new_connections];
ans =

  4×1 cell array


Finally, use createLgraphUsingConnections to create the merged DAG.

lgraph = createLgraphUsingConnections(all_layers,all_connections);
axis off

The code below includes the utility functions used above.

function lgraph = createLgraphUsingConnections(layers,connections)
% lgraph = createLgraphUsingConnections(layers,connections) creates a layer
% graph with the layers in the layer array |layers| connected by the
% connections in |connections|.

lgraph = layerGraph();
for i = 1:numel(layers)
    lgraph = addLayers(lgraph,layers(i));

for c = 1:size(connections,1)
    lgraph = connectLayers(lgraph,connections.Source{c},connections.Destination{c});


function c = subgraphConnections(src,dest)
if numel(src) > 1
    dest_name_root = [dest.Name '/in'];
    tmp = cell(numel(src),2);
    for k = 1:numel(src)
        tmp{k,1} = src(k).Name;
        tmp{k,2} = [dest_name_root num2str(k)];
elseif numel(dest) > 1
    tmp = cell(numel(dest),2);
    for k = 1:numel(dest)
        tmp{k,1} = src.Name;
        tmp{k,2} = dest(k).Name;
c = table(tmp(:,1),tmp(:,2),'VariableNames',{'Source','Destination'});

Get the MATLAB code

Published with MATLAB® R2018a


Comments are closed.

4 CommentsOldest to Newest

Anders Brunström replied on : 1 of 4
Was there a post including *multiple inputs* as well? If I try to add another input-layer to a graph it gives an error ``Error using nnet.internal.cnn.util.validateLayersForLayerGraph>iAssertOnlyOneInputAndOutputLayer (line 72) Unable to add input layer to layer graph because the graph already contains one input layer. Layer graphs can contain only one input layer.`` Is this still not supported for 2018a or am I missing something? Could there be a workaround?
Stu W replied on : 3 of 4
Is there any way to implement DAGs with LSTM, or sequence input data at all, or might there be in future? Appreciate this is different topic but seemed like a good place to ask. Thanks
Stu W replied on : 4 of 4
I'm attempting to build two LSTMs. Based on the same dataset, one performs classification the other regression. I believe the classifier output would assist in the regression. I could probably also use same LSTM layer weights for each net. I'd therefore like to take its fc layer (prior to softmax) and use those weights in a DAG placed in the regression net. Could I define a new deep layer with segregated fc layers perhaps to emulate a DAG? Any suggestions welcome and well received. Nb both regression net and classifier have sequence outputs the same length