{"id":88,"date":"2018-01-05T07:00:54","date_gmt":"2018-01-05T07:00:54","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=88"},"modified":"2021-04-06T15:52:32","modified_gmt":"2021-04-06T19:52:32","slug":"defining-your-own-network-layer","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2018\/01\/05\/defining-your-own-network-layer\/","title":{"rendered":"Defining Your Own Network Layer"},"content":{"rendered":"
\r\n

Note: Post updated 27-Sep-2018 to correct a typo in the implementation of the backward function.<\/i><\/p>\r\n

One of the new Neural Network Toolbox features of R2017b is the ability to define your own network layer. Today I'll show you how to make an exponential linear unit<\/i> (ELU) layer.<\/p>

Joe<\/a> helped me with today's post. Joe is one of the few developers who have been around MathWorks longer than I have. In fact, he's one of the people who interviewed me when I applied for a job here. I've had the pleasure of working closely with Joe for the past several years on many aspects of MATLAB design. He really loves tinkering with deep learning networks.<\/p>

Joe came across the paper \"Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs),\"<\/a> by Clevert, Unterthiner, and Hichreiter, and he wanted to make an ELU layer using R2017b.<\/p>

$f(x) = \\left\\{\\begin{array}{ll} x & x > 0\\\\ \\alpha(e^x - 1) & x \\leq 0 \\end{array} \\right.$<\/p>

Let's compare the ELU shape with a couple of other commonly used activation functions.<\/p>

alpha1 = 1;\r\nelu_fcn = @(x) x.*(x > 0) + alpha1*(exp(x) - 1).*(x <= 0);\r\n\r\nalpha2 = 0.1;\r\nleaky_relu_fcn = @(x) alpha2*x.*(x <= 0) + x.*(x > 0);\r\n\r\nrelu_fcn = @(x) x.*(x > 0);\r\n\r\nfplot(elu_fcn,[-10 3],'LineWidth'<\/span>,2)\r\nhold on<\/span>\r\nfplot(leaky_relu_fcn,[-10 3],'LineWidth'<\/span>,2)\r\nfplot(relu_fcn,[-10 3],'LineWidth'<\/span>,2)\r\nhold off<\/span>\r\nax = gca;\r\nax.XAxisLocation = 'origin'<\/span>;\r\nax.YAxisLocation = 'origin'<\/span>;\r\nbox off<\/span>\r\nlegend({'ELU'<\/span>,'Leaky ReLU'<\/span>,'ReLU'<\/span>},'Location'<\/span>,'northwest'<\/span>)\r\n<\/pre>\"\" 
Joe wanted to make a ELU layer with one learned alpha value per channel. He followed the procedure outlined in Define a Layer with Learnable Parameters<\/a> to make an ELU layer that works with the Neural Network Toolbox.<\/p>
Below is the template for a layer with learnable parameters. We'll explore how to fill in this template to make an ELU layer.<\/p>
\r\nclassdef<\/span> myLayer < nnet.layer.Layer\r\n\r\n    properties<\/span>\r\n        % (Optional) Layer properties<\/span>\r\n\r\n        % Layer properties go here<\/span>\r\n    end<\/span>\r\n\r\n    properties<\/span> (Learnable)\r\n        % (Optional) Layer learnable parameters<\/span>\r\n\r\n        % Layer learnable parameters go here<\/span>\r\n    end<\/span>\r\n    \r\n    methods<\/span>\r\n        function<\/span> layer = myLayer()\r\n            % (Optional) Create a myLayer<\/span>\r\n            % This function must have the same name as the layer<\/span>\r\n\r\n            % Layer constructor function goes here<\/span>\r\n        end<\/span>\r\n        \r\n        function<\/span> Z = predict(layer, X)\r\n            % Forward input data through the layer at prediction time and<\/span>\r\n            % output the result<\/span>\r\n            %<\/span>\r\n            % Inputs:<\/span>\r\n            %         layer    -    Layer to forward propagate through<\/span>\r\n            %         X        -    Input data<\/span>\r\n            % Output:<\/span>\r\n            %         Z        -    Output of layer forward function<\/span>\r\n            \r\n            % Layer forward function for prediction goes here<\/span>\r\n        end<\/span>\r\n\r\n        function<\/span> [Z, memory] = forward(layer, X)\r\n            % (Optional) Forward input data through the layer at training<\/span>\r\n            % time and output the result and a memory value<\/span>\r\n            %<\/span>\r\n            % Inputs:<\/span>\r\n            %         layer  - Layer to forward propagate through<\/span>\r\n            %         X      - Input data<\/span>\r\n            % Output:<\/span>\r\n            %         Z      - Output of layer forward function<\/span>\r\n            %         memory - Memory value which can be used for<\/span>\r\n            %                  backward propagation<\/span>\r\n\r\n            % Layer forward function for training goes here<\/span>\r\n        end<\/span>\r\n\r\n        function<\/span> [dLdX, dLdW1, ...<\/span>, dLdWn] = backward(layer, X, Z, dLdZ, memory)<\/span>\r\n            % Backward propagate the derivative of the loss function through <\/span>\r\n            % the layer<\/span>\r\n            %<\/span>\r\n            % Inputs:<\/span>\r\n            %         layer             - Layer to backward propagate through<\/span>\r\n            %         X                 - Input data<\/span>\r\n            %         Z                 - Output of layer forward function            <\/span>\r\n            %         dLdZ              - Gradient propagated from the deeper layer<\/span>\r\n            %         memory            - Memory value which can be used in<\/span>\r\n            %                             backward propagation<\/span>\r\n            % Output:<\/span>\r\n            %         dLdX              - Derivative of the loss with respect to the<\/span>\r\n            %                             input data<\/span>\r\n            %         dLdW1, ..., dLdWn - Derivatives of the loss with respect to each<\/span>\r\n            %                             learnable parameter<\/span>\r\n            \r\n            % Layer backward function goes here<\/span>\r\n        end\r\n    end\r\nend\r\n\r\n<\/pre>
For our ELU layer with a learnable alpha parameter, here's one way to write the constructor and the Learnable<\/tt> property block.<\/p>
\r\nclassdef<\/span> eluLayer < nnet.layer.Layer\r\n\r\n    properties<\/span> (Learnable)\r\n        alpha\r\n    end<\/span>\r\n    \r\n    methods<\/span>\r\n        function<\/span> layer = eluLayer(num_channels,name)\r\n            layer.Type = 'Exponential Linear Unit'<\/span>;\r\n            \r\n            % Assign layer name if it is passed in.<\/span>\r\n            if<\/span> nargin > 1\r\n                layer.Name = name;\r\n            end<\/span>\r\n            \r\n            % Give the layer a meaningful description.<\/span>\r\n            layer.Description = \"Exponential linear unit with \"<\/span> + ...<\/span>\r\n                num_channels + \" channels\"<\/span>;\r\n            \r\n            % Initialize the learnable alpha parameter.<\/span>\r\n            layer.alpha = rand(1,1,num_channels);\r\n        end<\/span>\r\n\r\n<\/pre>
The predict<\/tt> function is where we implement the activation function. Remember its mathematical form:<\/p>
$f(x) = \\left\\{\\begin{array}{ll}    x & x > 0\\\\    \\alpha(e^x - 1) & x \\leq 0 \\end{array} \\right.$<\/p>
Note: The expression (exp(min(X,0)) - 1)<\/tt> in the predict function is written that way to avoid computing the exponential of large positive numbers, which could result in infinities and NaNs popping up.<\/p>
\r\n        function<\/span> Z = predict(layer,X)\r\n            % Forward input data through the layer at prediction time and<\/span>\r\n            % output the result<\/span>\r\n            %<\/span>\r\n            % Inputs:<\/span>\r\n            %         layer    -    Layer to forward propagate through<\/span>\r\n            %         X        -    Input data<\/span>\r\n            % Output:<\/span>\r\n            %         Z        -    Output of layer forward function<\/span>\r\n            \r\n            % Expressing the computation in vectorized form allows it to<\/span>\r\n            % execute directly on the GPU.<\/span>\r\n            Z = (X .* (X > 0)) + ...<\/span>\r\n                (layer.alpha.*(exp(min(X,0)) - 1) .* (X <= 0));\r\n        end<\/span>\r\n\r\n<\/pre>
\r\n        function<\/span> [dLdX, dLdAlpha] = backward(layer, X, Z, dLdZ, ~)\r\n            % Backward propagate the derivative of the loss function through <\/span>\r\n            % the layer<\/span>\r\n            %<\/span>\r\n            % Inputs:<\/span>\r\n            %         layer             - Layer to backward propagate through<\/span>\r\n            %         X                 - Input data<\/span>\r\n            %         Z                 - Output of layer forward function            <\/span>\r\n            %         dLdZ              - Gradient propagated from the deeper layer<\/span>\r\n            %         memory            - Memory value which can be used in<\/span>\r\n            %                             backward propagation [unused]<\/span>\r\n            % Output:<\/span>\r\n            %         dLdX              - Derivative of the loss with<\/span>\r\n            %                             respect to the input data<\/span>\r\n            %         dLdAlpha          - Derivatives of the loss with<\/span>\r\n            %                             respect to alpha<\/span>\r\n            \r\n            % Original expression:<\/span>\r\n            % dLdX = (dLdZ .* (X > 0)) + ...<\/span>\r\n            %     (dLdZ .* (layer + Z) .* (X <= 0));<\/span>\r\n            %<\/span>\r\n            % Optimized expression:<\/span>\r\n            dLdX = dLdZ .* ((X > 0) + ...<\/span>\r\n                ((layer.alpha + Z) .* (X <= 0)));            \r\n            \r\n            dLdAlpha = (exp(min(X,0)) - 1) .* dLdZ;\r\n            % Sum over the image rows and columns.<\/span>\r\n            dLdAlpha = sum(sum(dLdAlpha,1),2);\r\n            % Sum over all the observations in the mini-batch.<\/span>\r\n            dLdAlpha = sum(dLdAlpha,4);\r\n        end<\/span>\r\n\r\n<\/pre>
That's all we need for our layer. We don't need to implement the forward<\/tt> function because our layer doesn't have memory and doesn't need to do anything special for training.<\/p>
Load in the sample digits training set, and show one of the images from it.<\/p>
[XTrain, YTrain] = digitTrain4DArrayData;\r\nimshow(XTrain(:,:,:,1010),'InitialMagnification'<\/span>,'fit'<\/span>)\r\nYTrain(1010)\r\n<\/pre>
\r\nans = \r\n\r\n  categorical\r\n\r\n     2 \r\n\r\n<\/pre>\"\" 
Make a network that uses our new ELU layer.<\/p>
layers = [ ...<\/span>\r\n    imageInputLayer([28 28 1])\r\n    convolution2dLayer(5,20)\r\n    batchNormalizationLayer\r\n    eluLayer(20)\r\n    fullyConnectedLayer(10)\r\n    softmaxLayer\r\n    classificationLayer];\r\n<\/pre>
Train the network.<\/p>
options = trainingOptions('sgdm'<\/span>);\r\nnet = trainNetwork(XTrain,YTrain,layers,options);\r\n<\/pre>
Training on single GPU.\r\nInitializing image normalization.\r\n|=========================================================================================|\r\n|     Epoch    |   Iteration  | Time Elapsed |  Mini-batch  |  Mini-batch  | Base Learning|\r\n|              |              |  (seconds)   |     Loss     |   Accuracy   |     Rate     |\r\n|=========================================================================================|\r\n|            1 |            1 |         0.03 |       2.5173 |        5.47% |       0.0100 |\r\n|            2 |           50 |         0.63 |       0.4548 |       85.16% |       0.0100 |\r\n|            3 |          100 |         1.20 |       0.1550 |       96.88% |       0.0100 |\r\n|            4 |          150 |         1.78 |       0.0951 |       99.22% |       0.0100 |\r\n|            6 |          200 |         2.37 |       0.0499 |       99.22% |       0.0100 |\r\n|            7 |          250 |         2.96 |       0.0356 |      100.00% |       0.0100 |\r\n|            8 |          300 |         3.55 |       0.0270 |      100.00% |       0.0100 |\r\n|            9 |          350 |         4.13 |       0.0168 |      100.00% |       0.0100 |\r\n|           11 |          400 |         4.74 |       0.0145 |      100.00% |       0.0100 |\r\n|           12 |          450 |         5.32 |       0.0118 |      100.00% |       0.0100 |\r\n|           13 |          500 |         5.89 |       0.0119 |      100.00% |       0.0100 |\r\n|           15 |          550 |         6.45 |       0.0074 |      100.00% |       0.0100 |\r\n|           16 |          600 |         7.03 |       0.0079 |      100.00% |       0.0100 |\r\n|           17 |          650 |         7.60 |       0.0086 |      100.00% |       0.0100 |\r\n|           18 |          700 |         8.18 |       0.0065 |      100.00% |       0.0100 |\r\n|           20 |          750 |         8.76 |       0.0066 |      100.00% |       0.0100 |\r\n|           21 |          800 |         9.34 |       0.0052 |      100.00% |       0.0100 |\r\n|           22 |          850 |         9.92 |       0.0054 |      100.00% |       0.0100 |\r\n|           24 |          900 |        10.51 |       0.0051 |      100.00% |       0.0100 |\r\n|           25 |          950 |        11.12 |       0.0044 |      100.00% |       0.0100 |\r\n|           26 |         1000 |        11.73 |       0.0049 |      100.00% |       0.0100 |\r\n|           27 |         1050 |        12.31 |       0.0040 |      100.00% |       0.0100 |\r\n|           29 |         1100 |        12.93 |       0.0041 |      100.00% |       0.0100 |\r\n|           30 |         1150 |        13.56 |       0.0040 |      100.00% |       0.0100 |\r\n|           30 |         1170 |        13.80 |       0.0043 |      100.00% |       0.0100 |\r\n|=========================================================================================|\r\n<\/pre>
Check the accuracy of the network on our test set.<\/p>
[XTest, YTest] = digitTest4DArrayData;\r\nYPred = classify(net, XTest);\r\naccuracy = sum(YTest==YPred)\/numel(YTest)\r\n<\/pre>
\r\naccuracy =\r\n\r\n    0.9872\r\n\r\n<\/pre>
Look at one of the images in the test set and see how it was classified by the network.<\/p>
k = 1500;\r\nimshow(XTest(:,:,:,k),'InitialMagnification'<\/span>,'fit'<\/span>)\r\nYPred(k)\r\n<\/pre>
\r\nans = \r\n\r\n  categorical\r\n\r\n     2 \r\n\r\n<\/pre>\"\" 
Now you've seen how to define your own layer, include it in a network, and train it up.<\/p>
The backward<\/tt> function implements the derivatives of the loss function, which are needed for training. The Define a Layer with Learnable Parameters<\/a> documentation page explains how to derive the needed quantities.<\/p>