{"id":1052,"date":"2019-01-31T15:26:21","date_gmt":"2019-01-31T15:26:21","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=1052"},"modified":"2021-04-06T15:51:10","modified_gmt":"2021-04-06T19:51:10","slug":"deep-learning-visualizations-cam-visualization","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2019\/01\/31\/deep-learning-visualizations-cam-visualization\/","title":{"rendered":"Deep Learning Visualizations: CAM Visualization"},"content":{"rendered":"I\u2019m hoping by now you\u2019ve heard that MATLAB has great visualizations, which can be helpful in deep learning to help uncover what\u2019s going on inside your neural network.<\/span>\r\n
<\/h6>\r\nLast post, we discussed visualizations of features learned by a neural network. Today, I\u2019d like to write about another visualization you can do in MATLAB for deep learning, that you won\u2019t find by reading the documentation*.<\/span>\r\n
<\/h6>\r\n
<\/h6>\r\n

CAM Visualizations<\/h3>\r\nThis is to help answer the question: \u201cHow did my network decide which category an image falls under?\u201d With class activation mapping, or CAM, you can uncover which region of an image mostly strongly influenced the network prediction. I was surprised at how easy this code was to understand: just a few lines of code that provides insight into a network. The end result will look something like this:<\/span>\r\n\r\n\"\"\r\n\r\nThis can show what areas of an image accounted for the network's prediction.<\/span>\r\n
<\/h6>\r\nTo make it more fun, let\u2019s do this \u201clive\u201d using a webcam. Of course, you can always feed an image into these lines instead of a webcam.<\/span>\r\n
<\/h6>\r\nFirst, read in a pretrained network for image classification. SqueezeNet, GoogLeNet, ResNet-18 are good choices, since they\u2019re relatively fast.<\/span>\r\n
netName = 'squeezenet';\r\nnet = eval(netName);<\/pre>\r\n
<\/h6>\r\nThen we get our webcam running.<\/span>\r\n
cam = webcam;\r\npreview(cam);\r\n<\/pre>\r\n
\"\"
Here\u2019s me running this example.<\/p><\/div>\r\n
<\/h6>\r\n
<\/h6>\r\n*And to be clear, this is all documented functionality - we're not going off the grid here! <\/em>\r\n
<\/h6>\r\n
<\/h6>\r\nGrab the input size, the class names, and the layer name from which we want to extract the activations. We want the ReLU layer that follows the last convolutional layer.<\/span>\r\n\r\nI considered adding a helper function to get the layer name, but it\u2019ll take you 2 seconds to grab the name from this table:<\/span>\r\n
<\/h6>\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Network Name<\/u><\/strong><\/td>\r\nActivation Layer Name<\/u><\/strong><\/td>\r\n<\/tr>\r\n
googlenet<\/td>\r\n'inception_5b-output'<\/td>\r\n<\/tr>\r\n
squeezenet<\/td>\r\n'relu_conv10'<\/td>\r\n<\/tr>\r\n
resnet18<\/td>\r\n'res5b'<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n
<\/h6>\r\n
<\/h6>\r\nNow let\u2019s run this on an image.<\/span>\r\n
im = imread('peppers.png');\r\nimResized = imresize(im,[inputSize(1),NaN]);\r\nimageActivations = activations(net,imResized,layerName);\r\n<\/pre>\r\n
<\/h6>\r\nThe class activation map for a specific class is the activation map of the ReLU layer, weighted by how much each activation contributes to the final score of the class.<\/span>\r\n
<\/h6>\r\nThe weights are from the final fully connected layer of the network for that class. SqueezeNet doesn\u2019t have a final fully connected layer, so the output of the ReLU layer is already the class activation map.<\/span>\r\n
<\/h6>\r\n
scores = squeeze(mean(imageActivations,[1 2]));\r\n[~,classIds] = maxk(scores,3);\r\nclassActivationMap = imageActivations(:,:,classIds(1));\r\n<\/pre>\r\n
<\/h6>\r\nIf you\u2019re using another network (not squeezenet) it looks like this:<\/span>\r\n
scores = squeeze(mean(imageActivations,[1 2]));\r\n[~,classIds] = maxk(scores,3);\r\n\r\nif netName ~= 'squeezenet'\r\n    \r\n    fcWeights = net.Layers(end-2).Weights    ;\r\n    fcBias = net.Layers(end-2).Bias;\r\n    scores = fcWeights*scores + fcBias;\r\n    \r\n    weightVector = shiftdim(fcWeights(classIds(1),:),-1);\r\n    classActivationMap = sum(imageActivations.*weightVector,3);\r\nend\r\n<\/pre>\r\n
<\/h6>\r\nCalculate the top class labels and the final normalized class scores.<\/span>\r\n
<\/h6>\r\n
scores = exp(scores)\/sum(exp(scores));\r\nmaxScores = scores(classIds);\r\nlabels = classes(classIds);\r\n<\/pre>\r\nAnd visualize the results. <\/span>\r\n
subplot(1,2,1);\r\nimshow(im);\r\nsubplot(1,2,2);\r\nCAMshow(im,classActivationMap);\r\ntitle(string(labels) + \", \" + string(maxScores));\r\n\r\ndrawnow;\r\n<\/pre>\r\n
<\/h6>\r\nThe activations for the top prediction are visualized. The top three predictions and confidence are displayed in the title of the plot.<\/span>\r\n
<\/h6>\r\n\"\"\r\n
<\/h6>\r\nNow you can switch from a still image to a webcam, with these lines of code:<\/span>\r\n
<\/h6>\r\n
h = figure('Units','normalized','Position',[.05 .05 .9 .8]);\r\nwhile ishandle(h)\r\n% im = imread('peppers.png'); <-- remove this line\r\nim = snapshot(cam);<\/pre>\r\n
<\/h6>\r\nput an \u2018end\u2019 to the loop right after drawnow; you\u2019re good to run this in a loop now. If I lost you with any of these steps, a link to the full file is at the bottom of this page.<\/em><\/span>\r\n
<\/h6>\r\nIt\u2019s also interesting to note that you can do this for any class of the network. Take a look at this image below. I have a coffee cup that is being accurately predicted as a coffee mug. You can see those class activations. But why is it also being highly classified as an iPod? <\/span>\r\n
<\/h6>\r\n\"\"\r\n
<\/h6>\r\n
<\/h6>\r\nYou can switch the class activations to visualize the second <\/em>most likely class, and visualize what features are triggering that prediction.<\/span>\r\n
<\/h6>\r\n
<\/h6>\r\n\"\"\r\n
<\/h6>\r\n
<\/h6>\r\nFinally, you\u2019ll need two helper functions to make this example run:<\/span>\r\n
function CAMshow(im,CAM)\r\n  imSize = size(im);\r\n  CAM = imresize(CAM,imSize(1:2));\r\n  CAM = normalizeImage(CAM);\r\n  CAM(CAM < .2) = 0;\r\n  cmap = jet(255).*linspace(0,1,255)';\r\n  %'\r\n  CAM = ind2rgb(uint8(CAM*255),cmap)*255;\r\n\r\n  combinedImage = double(rgb2gray(im))\/2 + CAM;\r\n  combinedImage = normalizeImage(combinedImage)*255;\r\n  imshow(uint8(combinedImage));\r\nend\r\n\r\nfunction N= normalizeImage(I)\r\n  minimum = min(I(:));\r\n  maximum = max(I(:));\r\n  N = (I-minimum)\/(maximum-minimum);\r\nend\r\n<\/pre>\r\nGrab the entire code with the blue \"Get the MATLAB Code\" link on the right. Happy visualization!<\/span>\r\n\r\nLeave a comment below, or follow me on Twitter!<\/span> 
Follow @jo_pings<\/a>