{"id":686,"date":"2018-11-15T13:16:29","date_gmt":"2018-11-15T13:16:29","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=686"},"modified":"2021-04-06T15:51:26","modified_gmt":"2021-04-06T19:51:26","slug":"sudoku-solver-image-processing-and-deep-learning","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2018\/11\/15\/sudoku-solver-image-processing-and-deep-learning\/","title":{"rendered":"Sudoku Solver: Image Processing and Deep Learning"},"content":{"rendered":"<span style=\"font-size: 14px;\"> This post is from a talk given by <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/profile\/authors\/10732621-justin-pinkney\">Justin Pinkney<\/a> at a recent <a href=\"http:\/\/www.matlabexpo.com\/uk\/\">MATLAB Expo.<\/a> <\/span>\r\n<!-- [If you prefer to watch Justin explain this in his own words, please check out the video here. (Link available in december)]-->\r\n<span style=\"font-size: 14px;\">Today\u2019s example will walk through using image processing and deep learning to automatically solve a Sudoku puzzle.<\/span>\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" class=\"aligncenter wp-image-730 size-large\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/image_finalSol_zoomed-1024x768.jpg\" alt=\"\" width=\"1024\" height=\"768\" \/>\r\n<span style=\"font-size: 12px;\">Those <span style=\"color: #ff0044;\"> red numbers<\/span> in the puzzle have been automatically added to the paper by the algorithm we're about to create.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">This example highlights why deep learning <em><strong>and<\/strong><\/em> image processing are both useful for object detection and image classification.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #e67300;\"><strong>Image Processing <u>and<\/u> Deep Learning<\/strong><\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">The two technologies I want to highlight today are:<\/span>\r\n<h6><\/h6>\r\n<h6><\/h6>\r\n<table>\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 0px 5px; text-align: center; border: 1px solid black;\"><strong>Image Processing<\/strong><\/td>\r\n<td style=\"padding: 0px 5px; text-align: center; border: 1px solid black;\"><strong>Deep learning<\/strong><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 10px 5px;\"><img decoding=\"async\" loading=\"lazy\" width=\"532\" height=\"280\" class=\"size-full wp-image-732 aligncenter\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/image_processing_cropped.png\" alt=\"\" \/><\/td>\r\n<td style=\"padding: 5px 5px;\"><img decoding=\"async\" loading=\"lazy\" width=\"749\" height=\"435\" class=\"alignnone size-full wp-image-734\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/deep_learning_cropped.png\" alt=\"\" \/><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 5px 5px; border: 1px solid black; width: 50%;\">Transforming or modifying an image at the pixel level. For example, filtering, blurring, de-blurring, and edge detection (to name a few)<\/td>\r\n<td style=\"padding: 0px 5px; border: 1px solid black; width: 50%;\">Automatically identifying features in an image through learning on sample images. Deep learning has has been revolutionizing the area of image processing in the past few years.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">If you need a refresher on what deep learning is, check out this <a href=\"https:\/\/www.youtube.com\/watch?v=3cSjsTKtN9M\">intro video<\/a>! <\/span>\r\n\r\n<span style=\"font-size: 14px;\">Now, let\u2019s talk about how these two technologies relate to each other. Two common perspectives are:<\/span>\r\n<h6><\/h6>\r\n<blockquote>\u201cDeep learning has made \u2018traditional\u2019 image processing obsolete.\u201d<\/blockquote>\r\nor\r\n<h6><\/h6>\r\n<blockquote>\u201cDeep learning needs millions of examples and is only good for classifying pictures of cats anyway.\u201d<\/blockquote>\r\n(I\u2019m definitely guilty of classifying a few cats in my day).\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">The reality is:<\/span>\r\n<ul>\r\n \t<li><span style=\"font-size: 14px;\">Deep learning and image processing are effective tools to solve different problems.<\/span><\/li>\r\n<\/ul>\r\n<em><strong>and <\/strong><\/em>\r\n<ul>\r\n \t<li><span style=\"font-size: 14px;\">These tasks are complex: <em>use the right tool for the job<\/em>.<\/span><\/li>\r\n<\/ul>\r\n<span style=\"font-size: 14px;\">So keeping these ideas in mind, let\u2019s use both image processing and deep learning to solve a complete problem. <\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 18px; color: #007399;\"><strong><u>Sudoku Puzzle<\/u><\/strong><\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">For those of you not familiar, a Sudoku puzzle as below here is completed when each row, column and 3x3 square contain the numbers 1-9 exactly with no repeats. <\/span>\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"361\" height=\"361\" class=\"alignnone size-full wp-image-724\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/sudoku_raw.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">The puzzle begins with some of the numbers filled in. Which numbers and how many are filled in determines the complexity of the puzzle.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">Here, we want our algorithm to find the boxes, and fill in the missing numbers. But that\u2019s almost too easy! We also want to solve the puzzle regardless of where it is in the image. Here\u2019s an example image of where we expect the algorithm to be able to solve the puzzle.<\/span>\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"421\" height=\"316\" class=\"alignnone size-full wp-image-722\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/Picture1.jpg\" alt=\"\" \/>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">To solve this, we want to use the right tool for the job, and that means breaking this down into a few distinct parts:<\/span>\r\n\r\n<img decoding=\"async\" loading=\"lazy\" width=\"797\" height=\"206\" class=\"alignnone size-full wp-image-738\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/workflow_cropped.png\" alt=\"\" \/>\r\n\r\n&nbsp;\r\n<ol>\r\n\r\n\t<li>Find the Puzzle - locate the box in the image<\/li>\r\n\r\n\t<li>Find the Boxes \u2013 identify each box of the 9x9 squares<\/li>\r\n\r\n\t<li>Read the numbers \u2013 these numbers can be digital or handwritten<\/li>\r\n\r\n\t<li>Solve the puzzle<\/li>\r\n\r\n<\/ol>\r\n\r\n\r\n<h6><\/h6>\r\n\r\n\r\n<span style=\"font-size: 14px;\">For each of these steps, we can choose between solving this with deep learning or image processing. Which sections should use deep learning and which sections should use image processing? Read on to discover how we did it.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\"><strong>Step 1. Find the Puzzle<\/strong><\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">Overview: We have uncontrolled images, and background, and the object size and orientation could vary greatly between different images. Not to mention lighting, camera capturing conditions. There\u2019s a lot of variability.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\">Method? Deep Learning<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">Let\u2019s see if we can classify the pixels associated with the puzzle using <a href=\"https:\/\/www.mathworks.com\/help\/vision\/examples\/semantic-segmentation-using-deep-learning.html\">semantic segmentation<\/a>. To do this, we need to label training data. Here\u2019s Justin's video of what that looks like in MATLAB using Image Labeler.<\/span>\r\n\r\n<h6><\/h6>\r\n\r\n<div style=\"width: 1280px;\" class=\"wp-video\"><!--[if lt IE 9]><script>document.createElement('video');<\/script><![endif]-->\n<video class=\"wp-video-shortcode\" id=\"video-686-1\" width=\"1280\" height=\"720\" preload=\"metadata\" controls=\"controls\"><source type=\"video\/mp4\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/Media1.mp4?_=1\" \/><a href=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/Media1.mp4\">https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/Media1.mp4<\/a><\/video><\/div>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">Here\u2019s the final input data after the it has been labeled. <\/span>\r\n\r\n<img decoding=\"async\" loading=\"lazy\" width=\"1210\" height=\"600\" class=\"alignnone size-full wp-image-746\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/traininDataSem.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\n\r\n<span style=\"font-size: 14px;\">Check out the variability in the training images! Definitely a good deep learning problem. One important thing to note is the dataset is quite small - only around 100 images. Let\u2019s train a semantic segmentation network and see if this data is sufficient. <\/span>\r\n<h6><\/h6><h6><\/h6>\r\n\r\nSetup the image datastores that store pixel information needed for the semantic segmentation network.\r\n<pre>train = pixelLabelImageDatastore(imagesTrain, labelsTrain, ...\r\n                                 'OutputSize', inputSize(1:2));\r\ntest = pixelLabelImageDatastore(imagesTest, labelsTest, ...\r\n                                'OutputSize', inputSize(1:2));\r\n<\/pre>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">We can then setup the network layers. It\u2019s interesting to note that Justin made a function to balance the classes using class weighting. Justin pointed out to me that his function does exactly what this example does: <a href=\"https:\/\/www.mathworks.com\/help\/vision\/examples\/semantic-segmentation-using-deep-learning.html#d120e2892\">https:\/\/www.mathworks.com\/help\/vision\/examples\/semantic-segmentation-using-deep-learning.html#d120e2892<\/a><\/span>\r\n<h6><\/h6>\r\nSetup the network.\r\n\r\n<pre>numClasses = 2;\r\nbaseNetwork = 'vgg16';\r\nlayers = segnetLayers(inputSize, numClasses, baseNetwork);\r\nlayers = sudoku.weightLossByFrequency(layers, train);\r\n<\/pre>\r\n<h6><\/h6>\r\nSet up the training options.\r\n\r\n<pre>    opts = trainingOptions('sgdm', ...\r\n        'InitialLearnRate', 0.005, ...\r\n        'LearnRateDropFactor', 0.1, ...\r\n        'LearnRateDropPeriod', 20, ...\r\n        'LearnRateSchedule', 'piecewise', ...\r\n        'ValidationData', test, ...\r\n        'ValidationPatience', Inf, ...\r\n        'MaxEpochs', 40, ...\r\n        'MiniBatchSize', 2, ...\r\n        'Shuffle', 'every-epoch', ...\r\n        'Plots', 'training-progress', ...\r\n        'CheckpointPath', checkpointPath);\r\n\r\n<\/pre>\r\nFinally, train the network.\r\n<pre>net = trainNetwork(train, layers, opts);\r\n<\/pre>\r\n<span style=\"font-size: 14px;\">This took roughly 20 minutes to run through 40 epochs, though differing hardware\/GPUs will produce varying results.<\/span>\r\n\r\n<span style=\"font-size: 14px;\">On a new test image, we get this result from the trained network:<\/span>\r\n\r\n<img decoding=\"async\" loading=\"lazy\" class=\"aligncenter wp-image-718 size-large\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/network_semantic_result-1024x627.png\" alt=\"\" width=\"1024\" height=\"627\" \/>\r\n<span style=\"font-size: 14px;\">Not bad! It wasn\u2019t even fooled too badly by the other box-shaped figure in the image. The smaller noise can be removed in the next section. <\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\"><strong>Step 2. Find the Boxes <\/strong><\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">Here, we need to find the individual boxes in the grid. This is a well-defined problem: straight lines, always dark ink on light paper, and equally sized boxes. Also keep in mind, we already found the approximate location of the box in step 1. We can make everything besides that location black, making this a very cleanly defined problem. <\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\">Method? Image Processing<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">We talk a lot about image processing in our <a href=\"https:\/\/blogs.mathworks.com\/steve\/\">image processing blog<\/a>. In fact, Steve's blog is the reason I became confident in my image processing skills. The key thing to remember here if you\u2019re not an image processing expert is - you don\u2019t have to be! MATLAB has apps to make this process easy. Check out the <a href=\"https:\/\/www.mathworks.com\/help\/images\/ref\/imagesegmenter-app.html\">Image Segmenter<\/a> app (<a href=\"https:\/\/www.mathworks.com\/videos\/image-segmentation-app-1504291389509.html\">here's a video<\/a> that shows an overview) to explore detecting the boxes in the image. The code below was automatically generated by the app and will detect the individual squares in the image. <\/span>\r\n<h6><\/h6>\r\nFirst, we have to clean up the image, so any noise is gone.\r\n<pre>BW_out = bwpropfilt(networkMask, 'Area', [100000 + eps(100000), Inf]);\r\n<\/pre>\r\nThen we dilate the mask to ensure it covers the entire box.\r\n<pre>maskDilated = imdilate(BW_out, strel('disk', 120));\r\n<\/pre>\r\nWe only care about the location where the box is, zero everything else out\r\n<pre>grayIm = rgb2gray(im);\r\ngrayIm(~maskDilated) = 0;\r\n<\/pre>\r\n<img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-748 size-large\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/grayIm-1024x743.png\" alt=\"\" width=\"1024\" height=\"743\" \/>\r\n\r\nAnd then find only the box in the image\r\n<pre>%% Mask it\r\nBW = imbinarize(grayIm, 'adaptive', 'Sensitivity', 0.700000, 'ForegroundPolarity', 'bright');\r\n% Invert mask\r\nBW = imcomplement(BW);\r\n% Clear borders\r\nBW = imclearborder(BW);\r\n% Fill holes\r\nBW = imfill(BW, 'holes');\r\nimshowpair(im,BW)\r\n<\/pre>\r\n<img decoding=\"async\" loading=\"lazy\" class=\"aligncenter wp-image-750 size-medium\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/finalMaskCompare-300x218.png\" alt=\"\" width=\"300\" height=\"218\" \/>\r\n\r\n<span style=\"font-size: 14px;\">This video shows step 1 and 2 together.<\/span>\r\n\r\n<div style=\"width: 1280px;\" class=\"wp-video\"><video class=\"wp-video-shortcode\" id=\"video-686-2\" width=\"1280\" height=\"720\" preload=\"metadata\" controls=\"controls\"><source type=\"video\/mp4\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/vlc-record-2018-11-13-09h38m01s-media4.mp4-.mp4?_=2\" \/><a href=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/vlc-record-2018-11-13-09h38m01s-media4.mp4-.mp4\">https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/vlc-record-2018-11-13-09h38m01s-media4.mp4-.mp4<\/a><\/video><\/div>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">I find these results fascinating... and very robust!<\/span>\r\n<h6><\/h6>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\"><strong>Step 3. Read the Numbers<\/strong><\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">There are many methods to reading handwritten and typed numbers. This is a challenging problem that must deal with different fonts and styles of numbers, but we have a variety of options:<\/span>\r\n<h6><\/h6>\r\n<ul>\r\n \t\r\n \t<li><span style=\"font-size: 14px;\">Optical character recognition (OCR) is a common method<\/span><\/li>\r\n \t<li><span style=\"font-size: 14px;\">HOG with a machine learning classifier is another option. A MATLAB example is <a href=\"https:\/\/www.mathworks.com\/help\/vision\/examples\/digit-classification-using-hog-features.html\">here<\/a><\/span><\/li>\r\n\r\n\r\n<\/ul>\r\n\r\n<span style=\"font-size: 14px;\">\r\nHandwriting, however, is also a well-studied deep learning classification problem (You can see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/examples\/create-simple-deep-learning-network-for-classification.html\">this example<\/a> using the common MNIST dataset, and I\u2019ve written a semi-related post about it <a href=\"https:\/\/blogs.mathworks.com\/deep-learning\/2018\/06\/22\/deep-learning-in-action-part-1\/\">here<\/a>).<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\">Method? Deep Learning<\/span>\r\n<h6><\/h6>\r\n\r\n<span style=\"font-size: 14px;\">The goal here is to move from typed or handwritten numbers and digitize them using deep learning, as shown in this figure.<\/span>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"641\" height=\"327\" class=\"alignnone size-full wp-image-720\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/OCR_picture.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">For this, we want lots of training data to account for the different variations in the letters, and when it comes to this much data, we don\u2019t want to be spending hours writing out digits by hand for training.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">This is a great opportunity to use MATLAB to generate <strong>synthetic data<\/strong>. For the handwritten digits, this is easy \u2013 simply steal from the MNIST Dataset and add to our synthetic background shown in the figure below. For creating a variety of typed numbers, we want to vary the numbers to ensure they will be recognized regardless of the font used (Times New Roman, Verdana, etc)<\/span>\r\n<h6><\/h6>\r\n<img decoding=\"async\" loading=\"lazy\" width=\"298\" height=\"301\" class=\"alignnone size-full wp-image-700\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/background_synth.png\" alt=\"\" \/>\r\n<h6><\/h6>\r\n<pre>resolution = size(im, 1);\r\nmaxSize = resolution - 2*border;\r\nfontSize = round((maxSize - minSize)*rand(1) + minSize);\r\n \r\ntextColour = maxColour*rand(1, 3);\r\nposition = [resolution\/2, resolution\/2] + maxOffset.*(rand(1, 2) - 0.5);\r\n\r\n<\/pre>\r\n\r\n<span style=\"font-size: 14px;\">For both types of synthetically generated numbers, we want to vary the size and location of the numbers. This is what allows us to generate as much data as we want!<\/span>\r\n<table>\r\n<tbody>\r\n\r\n<tr>\r\n\r\n<td style=\"padding: 10px 20px;\">\r\n<img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-704 size-thumbnail\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/eight_synth1-150x150.png\" alt=\"\" width=\"150\" height=\"150\" \/>\r\n<\/td>\r\n<td style=\"padding: 10px 20px;\">\r\n<img decoding=\"async\" loading=\"lazy\" class=\"wp-image-706 size-thumbnail\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/eight_synth2-150x150.png\" alt=\"\" width=\"150\" height=\"150\" \/>\r\n<\/td>\r\n\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 10px 20px;\">\r\nSynthetic Image - Handwritten\r\n<\/td>\r\n\r\n<td style=\"padding: 10px 20px;\">\r\nSynthetic Image - Typed\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n\r\n\r\n\r\n<span style=\"font-size: 14px;\">Note: the line widths of the boxes are also made randomly thicker or thinner to account for any discrepancies between boxes.\r\n\r\n\r\n\r\n<em>The idea of synthetic data warrants more time than I can give it in this post. Stay tuned for more posts on this in the future!<\/em><\/span>\r\n\r\n<h6><\/h6>\r\n\r\n<span style=\"font-size: 14px;\">We now can train the network. Set the training options, create the layers, and train the network as we did before.<\/span>\r\n<pre>options = trainingOptions('sgdm', ...\r\n                          'Plots', 'training-progress', ...\r\n                          'L2Regularization', 1e-2, ...\r\n                          'MaxEpochs', 8, ...\r\n                          'Shuffle', 'every-epoch', ...\r\n                          'InitialLearnRate', 0.01, ...\r\n                          'LearnRateDropFactor', 0.1, ...\r\n                          'LearnRateDropPeriod', 3, ...\r\n                          'LearnRateSchedule', 'piecewise', ...\r\n                          'ValidationData', test, ...\r\n                          'ValidationPatience', Inf, ...\r\n                          'MiniBatchSize', 64);\r\n \r\n% This is creating a network from scratch that closely resembles VGG16\r\nlayers = sudoku.training.vggLike(initialChannels, imageSize);\r\n    \r\n% Train\r\nnet = trainNetwork(train, layers, options);\r\n<\/pre>\r\n<span style=\"font-size: 14px;\">For this example, it\u2019s looking like the network is getting roughly 97.8% accuracy. That accuracy is sufficient... for a Sudoku solution.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\"><strong>Step 4. Solve the Puzzle<\/strong><\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">We have the boxes and the numbers. Now we need to fill in the other values.<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\">Method? Neither! This is an optimization problem<\/span>\r\n<h6><\/h6>\r\n<span style=\"font-size: 14px;\">We\u2019ve written about Sudoku solvers previously: \r\n\r\n<ul>\r\n\t<li>Cleve\u2019s post on Sudoku is here: <a href=\"https:\/\/www.mathworks.com\/company\/newsletters\/articles\/solving-sudoku-with-matlab.html\">https:\/\/www.mathworks.com\/company\/newsletters\/articles\/solving-sudoku-with-matlab.html<\/a><\/li>\r\n\r\n\t<li>\r\n<span style=\"font-size: 14px;\">Or check out the optimization code in this documentation example: <a href=\"https:\/\/www.mathworks.com\/help\/optim\/examples\/solve-sudoku-puzzles-via-integer-programming.html\">https:\/\/www.mathworks.com\/help\/optim\/examples\/solve-sudoku-puzzles-via-integer-programming.html<\/a><\/span><\/li>\r\n<\/ul>\r\n\r\n\r\n<h6><\/h6>\r\n<span style=\"font-size: 16px; color: #007399;\">Putting it all together<\/span>\r\n<h6><\/h6>\r\nNow we have all 4 steps completed, and we have a Sudoku solver using deep learning and image processing to best solve the problem.\r\n\r\n\r\n\r\n\r\n<table>\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 50px 10px; border-bottom: 1px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-776 size-full\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/step1-1.png\" alt=\"\" width=\"275\" height=\"115\" \/><\/td>\r\n<td style=\"padding: 10px 10px; text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-758 size-medium\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/SolutionFinal_1-300x225.jpg\" alt=\"\" width=\"300\" height=\"225\" \/><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 50px 10px; border-bottom: 1px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-778 size-full\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/step2-1.png\" alt=\"\" width=\"275\" height=\"115\" \/><\/td>\r\n<td style=\"padding: 10px 10px; text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-760 size-medium\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/SolutionFinal_2-300x225.jpg\" alt=\"\" width=\"200\" height=\"125\" \/><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 100px 10px; border-bottom: 1px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-780 size-full\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/step3-1.png\" alt=\"\" width=\"275\" height=\"115\" \/><\/td>\r\n<td style=\"padding: 10px 10px; text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-762 size-medium\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/solutionFinal_3.jpg-300x300.png\" alt=\"\" width=\"300\" height=\"300\" \/><\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"padding: 100px 10px; border-bottom: 1px solid black;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-782 size-full\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/step4-1.png\" alt=\"\" width=\"275\" height=\"115\" \/><\/td>\r\n<td style=\"padding: 10px 10px; text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-764 size-medium\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/solutionFinal_4.jpg-300x300.png\" alt=\"\" width=\"300\" height=\"300\" \/><\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n<h6><\/h6>\r\n<h6><\/h6><h6><\/h6>\r\n\r\nKeep in mind some <strong>key tips<\/strong> when tackling a problem dealing with images and video:\r\n<ul>\r\n\r\n\r\n \t<li>Deep learning is sometimes, <em>but not always, <\/em>the right tool for the job<\/li>\r\n \t<li>Both image processing and deep learning are great tools that can be combined to form the right\r\nsolution<\/li>\r\n\r\n<h6><\/h6>\r\n\r\nYou can download the complete solution here: <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/68980-deep-sudoku-solver\">https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/68980-deep-sudoku-solver<\/a>, and leave a comment for Justin below if you have any questions.\r\n<h6><\/h6>","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2018\/11\/image_finalSol_zoomed-1024x768.jpg\" onError=\"this.style.display ='none';\" \/><\/div><p> This post is from a talk given by Justin Pinkney at a recent MATLAB Expo. \r\n\r\nToday\u2019s example will walk through using image processing and deep learning to automatically solve a Sudoku... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/deep-learning\/2018\/11\/15\/sudoku-solver-image-processing-and-deep-learning\/\">read more >><\/a><\/p>","protected":false},"author":156,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/686"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/users\/156"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/comments?post=686"}],"version-history":[{"count":14,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/686\/revisions"}],"predecessor-version":[{"id":3098,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/686\/revisions\/3098"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/media?parent=686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/categories?post=686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/tags?post=686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}