{"id":3269,"date":"2019-05-20T07:00:21","date_gmt":"2019-05-20T11:00:21","guid":{"rendered":"https:\/\/blogs.mathworks.com\/steve\/?p=3269"},"modified":"2024-01-01T16:38:09","modified_gmt":"2024-01-01T21:38:09","slug":"multiresolution-pyramids-part-4-image-blending","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/steve\/2019\/05\/20\/multiresolution-pyramids-part-4-image-blending\/","title":{"rendered":"Multiresolution pyramids part 4: Image blending"},"content":{"rendered":"
\r\n\r\nToday I want to wrap up (for now) my series on multiresolution pyramids by showing you how to make this strange-looking fruit:\r\n\r\n\r\n\r\nThis a slightly nontraditional demonstration of a technique called image blending. I first considered writing a blog about image blending when I saw an example recently that was posted inside MathWorks. I asked the engineer who wrote the example, Steve Kuznicki, for more information about the example, and he shared his method and code with me.\r\n\r\nThe method for blending two images together was based on using Laplacian pyramids. The earliest reference I know about is Burt and Adelson, \"A Multiresolution Spline with Application to Image Mosaics,\" ACM Transactions on Graphics<\/i>, vol. 2, no. 4, October 1983, pp. 217-236.\r\n\r\nThe code, though, looked a little complicated. I realized that the code was complicated because of underlying limitations in the functional design of impyramid<\/tt>, which I wrote about earlier in the series (02-Apr-2019<\/a> and 09-Apr-2019<\/a>).\r\n\r\nSo, instead of jumping right into image blending, I spent some time first writing about image tiling<\/a> and revised interfaces for computing and visualization multiresolution and Laplacian pyramids.\r\n\r\nNow, I'm ready to come back to image blending.\r\n\r\nBecause, I think, of the influence of the Burt and Adelson paper, it has become common to demonstrate the technique by blending images of an apple and orange, with the seam between the two images split right down the middle. I found a couple of public-domain images (Big red apple<\/a> and 20140119Hockenheim1<\/a>) to use for a demonstration.\r\n
apple_url = 'https:\/\/blogs.mathworks.com\/steve\/files\/apple.jpg'<\/span>;\r\nA = im2double(imread(apple_url));\r\n\r\norange_url = 'https:\/\/blogs.mathworks.com\/steve\/files\/orange.jpg'<\/span>;\r\nB = im2double(imread(orange_url));\r\n\r\nsubplot(1,2,1)\r\nimshow(A)\r\nsubplot(1,2,2)\r\nimshow(B)\r\n<\/pre>\r\n\"\"\r\n\r\nNow I will make a mask image to define which part of the blended image will come from image A (the apple).\r\n
mask_A = zeros(1024,1024);\r\nmask_A(:,1:512) = 1;\r\n\r\nclf\r\nimshow(mask_A)\r\nxticks([])\r\nyticks([])\r\naxis on<\/span>\r\n<\/pre>\r\n\"\"\r\n\r\nThe simplest way to blend the images is to just put the pieces together. We can do that with a little mask arithmetic.\r\n
C = (A .* mask_A) + (B .* (1 - mask_A));\r\nclf\r\nimshow(C)\r\nxticks([])\r\nyticks([])\r\n<\/pre>\r\n\"\"\r\n\r\nYou can see the sharp edge and contrast at the seam between the two images. The idea behind using Laplacian pyramids is to smooth the seam in a spatial-frequency-dependent way. Here is the procedure:\r\n
\r\n
    \r\n \t
  1. Construct Laplacian pyramids for the two images.<\/li>\r\n \t
  2. Make a third Laplacian pyramid that is contructed by a mask-based joining of the two original-image Laplacian pyramids at every pyramid level.<\/li>\r\n \t
  3. Reconstruct the output image from the blended Laplacian pyramid.<\/li>\r\n<\/ol>\r\n<\/div>\r\nThe overall effect is to smooth low spatial frequencies at lot at the seam, but to smooth high spatial frequencies only a little. It's a clever, deceptively simple idea, really.\r\n\r\nHere's the first step. (Note: the multiresolution pyramid formed from the mask will be used to combine the Laplacian pyramids at each level.)\r\n
    mrp_A = multiresolutionPyramid(A);\r\nmrp_B = multiresolutionPyramid(B);\r\nmrp_mask_A = multiresolutionPyramid(mask_A);\r\n\r\nlap_A = laplacianPyramid(mrp_A);\r\nlap_B = laplacianPyramid(mrp_B);\r\n<\/pre>\r\nNext, form the blended Laplacian pyramid.\r\n
    for<\/span> k = 1:length(lap_A)\r\n    lap_blend{k} = (lap_A{k} .* mrp_mask_A{k}) + ...<\/span>\r\n        (lap_B{k} .* (1 - mrp_mask_A{k}));\r\nend<\/span>\r\n<\/pre>\r\nFinally, reconstruct the output image from the blended Laplacian pyramid. (The code for reconstructFromLaplacianPyramid<\/tt> is below.)\r\n
    C_blended = reconstructFromLaplacianPyramid(lap_blend);\r\nimshow(C_blended)\r\n<\/pre>\r\n\"\"\r\n\r\nA side-by-side blend is boring, though. Note that the two regions in the mask can have any shape. Here's how I made the blended image that started this post. It is based on a mask with a circular region in the center.\r\n
    x = linspace(-1,1,1024);\r\ny = x';\r\nmask2_A = hypot(x,y) <= 0.5;\r\n\r\nmrp_mask2_A = multiresolutionPyramid(mask2_A);\r\n\r\nfor<\/span> k = 1:length(lap_A)\r\n    lap_blend2{k} = (lap_A{k} .* mrp_mask2_A{k}) + ...<\/span>\r\n        (lap_B{k} .* (1 - mrp_mask2_A{k}));\r\nend<\/span>\r\n\r\nC2_blended = reconstructFromLaplacianPyramid(lap_blend2);\r\nimshow(C2_blended)\r\n<\/pre>\r\n\"\"\r\n\r\nDoesn't that look yummy?\r\n\r\nFunctions used above:\r\n
    function<\/span> mrp = multiresolutionPyramid(A,num_levels)\r\n%multiresolutionPyramid(A,numlevels)<\/span>\r\n%   mrp = multiresolutionPyramid(A,numlevels) returns a multiresolution<\/span>\r\n%   pyramd from the input image, A. The output, mrp, is a 1-by-numlevels<\/span>\r\n%   cell array. The first element of mrp, mrp{1}, is the input image.<\/span>\r\n%<\/span>\r\n%   If numlevels is not specified, then it is automatically computed to<\/span>\r\n%   keep the smallest level in the pyramid at least 32-by-32.<\/span>\r\n\r\n%   Steve Eddins<\/span>\r\n%   Copyright The MathWorks, Inc. 2019<\/span>\r\n\r\nA = im2double(A);\r\n\r\nM = size(A,1);\r\nN = size(A,2);\r\n\r\nif<\/span> nargin < 2\r\n    lower_limit = 32;\r\n    num_levels = min(floor(log2([M N]) - log2(lower_limit))) + 1;\r\nelse<\/span>\r\n    num_levels = min(num_levels, min(floor(log2([M N]))) + 2);\r\nend<\/span>\r\n\r\nmrp = cell(1,num_levels);\r\n\r\nsmallest_size = [M N] \/ 2^(num_levels - 1);\r\nsmallest_size = ceil(smallest_size);\r\npadded_size = smallest_size * 2^(num_levels - 1);\r\n\r\nAp = padarray(A,padded_size - [M N],'replicate'<\/span>,'post'<\/span>);\r\n\r\nmrp{1} = Ap;\r\nfor<\/span> k = 2:num_levels\r\n    mrp{k} = imresize(mrp{k-1},0.5,'lanczos3'<\/span>);\r\nend<\/span>\r\n\r\nmrp{1} = A;\r\nend<\/span>\r\n\r\nfunction<\/span> lapp = laplacianPyramid(mrp)\r\n\r\n% Steve Eddins<\/span>\r\n% MathWorks<\/span>\r\n\r\nlapp = cell(size(mrp));\r\nnum_levels = numel(mrp);\r\nlapp{num_levels} = mrp{num_levels};\r\nfor<\/span> k = 1:(num_levels - 1)\r\n   A = mrp{k};\r\n   B = imresize(mrp{k+1},2,'lanczos3'<\/span>);\r\n   [M,N,~] = size(A);\r\n   lapp{k} = A - B(1:M,1:N,:);\r\nend<\/span>\r\nlapp{end} = mrp{end};\r\n\r\nend<\/span>\r\n\r\nfunction<\/span> out = reconstructFromLaplacianPyramid(lapp)\r\n\r\n% Steve Eddins<\/span>\r\n% MathWorks<\/span>\r\n\r\nnum_levels = numel(lapp);\r\nout = lapp{end};\r\nfor<\/span> k = (num_levels - 1) : -1 : 1\r\n   out = imresize(out,2,'lanczos3'<\/span>);\r\n   g = lapp{k};\r\n   [M,N,~] = size(g);\r\n   out = out(1:M,1:N,:) + g;\r\nend<\/span>\r\nend<\/span>\r\n<\/pre>\r\n