{"id":1280,"date":"2019-02-21T20:40:33","date_gmt":"2019-02-21T20:40:33","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=1280"},"modified":"2019-02-26T13:28:40","modified_gmt":"2019-02-26T13:28:40","slug":"image-to-image-regression","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2019\/02\/21\/image-to-image-regression\/","title":{"rendered":"Image-to-Image Regression"},"content":{"rendered":"Today I'd like to talk about the basic concepts of setting up a network to train on an image-to-image regression problem.\r\nThis demo came about for two reasons:\r\n
    \r\n \t
  1. There are quite a few questions on MATLAB answers<\/a> about image\u2013to\u2013image deep learning problems.<\/li>\r\n \t
  2. I\u2019m planning a future in-depth post with an image processing\/deep learning expert, where we\u2019ll be getting into the weeds on regression, and it would be good to understand the basics to keep up with him.<\/li>\r\n<\/ol>\r\nSo, let\u2019s dive into the concept of image-to-image deep learning problems in MATLAB.\r\n\"\"\r\n
    <\/h6>\r\nTypically, deep learning problems can be divided into classification or regression problems. Classification is the problem that most people are familiar with, and we write about often.\r\n
    <\/h6>\r\n\"\"\r\n
    <\/h6>\r\n

    Given an image, predict which category an object belongs to.<\/p>\r\n\r\n

    <\/h6>\r\nIn regression problems, there are no longer discrete categories. The output could be a non-discrete value: for example, given an image, output the rotation value<\/a>.\r\n
    <\/h6>\r\nAlong the same lines, given an image, predict a new image!\r\n
    <\/h6>\r\n
    <\/h6>\r\n\"\"\r\n

    <\/h3>\r\n

    <\/h6>\r\n
    <\/h6>\r\n
    <\/h6>\r\nTo learn more about the concept<\/strong>, of image-to-image deep learning we can start with a simple example in documentation:\r\nhttps:\/\/www.mathworks.com\/help\/deeplearning\/examples\/remove-noise-from-color-image-using-pretrained-neural-network.html<\/a>\r\n
    <\/h6>\r\nThis is a great introduction to the topic that's explained well in the example. Plus, if you\u2019re trying to denoise an image, this example solves the problem, so you're done!\r\n
    <\/h6>\r\nHowever, the goal of this post is understand how to create our custom deep learning algorithm from scratch<\/strong>. The hardest part is getting the data set up. Everything else should be reasonably<\/em> straightforward.\r\n
    <\/h6>\r\n
    <\/h6>\r\n

    All About Datastores<\/h2>\r\n

    <\/h6>\r\nDatastores deserve a post of their own, but let me just say, if you can appreciate and master datastores, you can conquer the world.\r\n
    <\/h6>\r\nAt a high level, datastores make sense: They are an efficient way of bringing in data for deep learning (and other) applications. You don\u2019t have to deal with memory management, and deep learning functions know how to handle the datastores as an input to the function. This is all good.\r\n\u201cHow do I get datastores to work for image-to-image deep learning training data?\u201d<\/em> Great question!!\r\n
    <\/h6>\r\n

    randomPatchExtractionDatastore<\/h3>\r\nI'm going to recommend using this handy function called Random Patch Extraction Datastore<\/a>, which is what I use in the example below.\r\n
    <\/h6>\r\nWe\u2019re not exactly short with our naming convention here, but you get a great idea of what you\u2019re getting with this function!\r\n
    <\/h6>\r\nExtracting random patches of your images is a great way to cultivate more input images, especially if you're low on data. The algorithm needs enough data samples to train accurately, so we can cut the images into smaller pieces and deliver more examples for the network to learn.\r\n
    <\/h6>\r\nThis function will take an input datastore, a corresponding output datastore, and a patch size.\r\n
    <\/h6>\r\n

    The code:<\/h2>\r\n

    <\/h6>\r\nOur problem is going to be image deblurring. And we're going to set up this up from scratch.\r\n
    <\/h6>\r\nI have a perfect final image:\r\n\"\"\r\n
    <\/h6>\r\nI blur the image:\r\n\"\"\r\n
    <\/h6>\r\nand I put all of my data into individual folders.\r\n
    blurredDir = createTrainingSet(trainImages);\r\nblurredImages = imageDatastore(blurredDir,'FileExtensions','.mat','ReadFcn',@matRead);\r\nimagesDir = '.';\r\ntrainImagesDir = fullfile(imagesDir,'iaprtc12','images','02');\r\nexts = {'.jpg','.bmp','.png'};\r\ntrainImages = imageDatastore(trainImagesDir,'FileExtensions',exts);<\/pre>\r\n
    <\/h6>\r\nThe blurred image is my input, the perfect\/original image is my output. This felt backwards, but I reminded myself: I want the network to see a blurry image and output the clean image as a final result.\r\n
    <\/h6>\r\nVisualize the input and output images\r\n
    <\/h6>\r\n
    im_orig = trainImages.readimage(ii);\r\nim_blurred = blurredImages.readimage(ii);\r\n\r\nimshow(im_orig);\r\ntitle('Clean Image - Final Result');\r\nfigure; imshow(im_blurred);\r\ntitle('Blurred Image - Input');<\/pre>\r\n
    <\/h6>\r\n\"\"\r\n\r\n\"\"\r\n
    <\/h6>\r\nSet up data augmentation for even more variety of training images.\r\n
    augmenter = imageDataAugmenter( ...\r\n    'RandRotation',@()randi([0,1],1)*90, ...\r\n    'RandXReflection',true);<\/pre>\r\nThis will rotate the input images a random amount, and allow for reflection on the X axis.\r\n
    <\/h6>\r\nThen our random patch extraction datastore is used to compile the input and output images in a way the trainNetwork command will understand.\r\n
    miniBatchSize = 64;\r\npatchSize = [40 40];\r\npatchds = randomPatchExtractionDatastore(blurredImages,trainImages,patchSize, ....\r\n'PatchesPerImage',64, ...\r\n'DataAugmentation',augmenter);\r\npatchds.MiniBatchSize = miniBatchSize;<\/pre>\r\n
    <\/h6>\r\n
    Network layers<\/h2>\r\nTo set up an image-to-image regression network, let's start with a set of layers almost <\/em> right for our example.\r\n
    <\/h6>\r\nComputer Vision Toolbox has the function unetLayers<\/a> that allows you to set up the layers of a semantic segmentation network (U-Net) quickly.\r\n
    <\/h6>\r\n
    lgraph = unetLayers([40 40 3] , 3,'encoderDepth',3);<\/pre>\r\nWe have to alter this slightly to fit our network by adding an L2 loss layer.\r\nRemove the last 2 layers, replace them with a regression layer.\r\n
    lgraph = lgraph.removeLayers('Softmax-Layer');\r\nlgraph = lgraph.removeLayers('Segmentation-Layer');\r\nlgraph = lgraph.addLayers(regressionLayer('name','regressionLayer'));\r\nlgraph = lgraph.connectLayers('Final-ConvolutionLayer','regressionLayer');<\/pre>\r\n
    <\/h6>\r\ndeepNetworkDesigner app will also remove and connect new layers for you as shown below.\r\n\r\n\"\"\r\n
    <\/h6>\r\nSet the training parameters\r\n
    maxEpochs = 100;\r\nepochIntervals = 1;\r\ninitLearningRate = 0.1;\r\nlearningRateFactor = 0.1;\r\nl2reg = 0.0001;\r\noptions = trainingOptions('sgdm', ...\r\n    'Momentum',0.9, ...\r\n    'InitialLearnRate',initLearningRate, ...\r\n    'LearnRateSchedule','piecewise', ...\r\n    'LearnRateDropPeriod',10, ...\r\n    'LearnRateDropFactor',learningRateFactor, ...\r\n    'L2Regularization',l2reg, ...\r\n    'MaxEpochs',maxEpochs ,...\r\n    'MiniBatchSize',miniBatchSize, ...\r\n    'GradientThresholdMethod','l2norm', ...\r\n    'Plots','training-progress', ...\r\n    'GradientThreshold',0.01);<\/pre>\r\n
    <\/h6>\r\nand train\r\n
       modelDateTime = datestr(now,'dd-mmm-yyyy-HH-MM-SS');\r\n    net = trainNetwork(patchds,lgraph,options);\r\n    save(['trainedNet-' modelDateTime '-Epoch-' num2str(maxEpochs*epochIntervals) ...\r\n            'ScaleFactors-' num2str(234) '.mat'],'net','options');<\/pre>\r\n
    <\/h6>\r\n(...8 hours later...)\r\n
    <\/h6>\r\nI came back this morning and\u2026 I have a fully trained network!\r\n
    <\/h6>\r\nNow the quality may not be the best for deblurring images, because my main intention was to show the setup of the training images and the network. But I have a network that really tries.<\/em>\r\n
    <\/h6>\r\nShow the original image and the blurred image.\r\n
    testImage = testImages.readimage(randi(400));\r\n\r\nLEN = 21;\r\nTHETA = 11;\r\nPSF = fspecial('motion', LEN, THETA);\r\n\r\nblurredImage = imfilter(testImage, PSF, 'conv', 'circular');\r\ntitle('Blurry Image');\r\n\r\nfigure; imshow(testImage);\r\ntitle('Original Image');<\/pre>\r\n
    <\/h6>\r\n\"\"\r\n\r\n\"\"\r\n
    <\/h6>\r\n... and create a 'deblurred' image from the network:\r\n
    <\/h6>\r\n
    Ideblurred = activations(net,blurredImage,'regressionoutput');\r\nfigure; imshow(Ideblurred)\r\nIapprox = rescale(Ideblurred);\r\nIapprox = im2uint8(Iapprox);\r\nimshow(Iapprox)\r\ntitle('Denoised Image')<\/pre>\r\n\"\"\r\n\r\n
    <\/h6>\r\nUpdate: I know it says denoised, rather than deblurred, I coppied the code from another example and forgot to switch the title.<\/em>\r\n
    <\/h6>\r\nKeep in mind, the quality of the network was not the point, though now I\u2019m very curious to keep working and improving this network. That\u2019s all today! I hope you found this useful \u2013 I had a great time playing in MATLAB, and I hope you do too.\r\n\r\n
    <\/h6>
    <\/h6>\r\nUPDATE: I changed a few training parameters and ran the network again. If you're planning on running this code, I would highly suggest training with these parameters: \r\n
    options = trainingOptions('adam','InitialLearnRate',1e-4,'MiniBatchSize',64,...\r\n        'Shuffle','never','MaxEpochs',50,...\r\n        'Plots','training-progress');\r\n<\/pre>\r\nThe results are much better:\r\n\"\"\r\n\r\n\r\n