{"id":4301,"date":"2020-12-23T10:24:12","date_gmt":"2020-12-23T15:24:12","guid":{"rendered":"https:\/\/blogs.mathworks.com\/steve\/?p=4301"},"modified":"2020-12-23T10:30:15","modified_gmt":"2020-12-23T15:30:15","slug":"registering-hand-held-pictures","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/steve\/2020\/12\/23\/registering-hand-held-pictures\/","title":{"rendered":"Registering Hand-Held Pictures"},"content":{"rendered":"
The typical modern French* horn, pictured below, has about 23 total feet of tubing. At the beginning and the end, the tubing is conical. In the middle, the tubing is cylindrical.<\/span><\/div>
\"IMG_4910.jpeg\"<\/img><\/div>
Depending on which valve levers are pressed, a player might be buzzing into single tube that is anywhere from 9 feet to 17 feet long (approximately). For a personal project, I wanted to create a visual illustration of these various lengths. I used an extension pole and got some help in taking several pictures. My helper used a handheld phone.<\/span><\/div>
A = imread(<\/span>'A.jpg'<\/span>);<\/span><\/span><\/div><\/div>
B = imread(<\/span>'B.jpg'<\/span>);<\/span><\/span><\/div><\/div>
C = imread(<\/span>'C.jpg'<\/span>);<\/span><\/span><\/div><\/div>
<\/div><\/div>
tiledlayout(2,2)<\/span><\/span><\/div><\/div>
<\/div><\/div>
nexttile<\/span><\/span><\/div><\/div>
imshow(A,<\/span>'Interpolation'<\/span>,<\/span>'bilinear'<\/span>)<\/span><\/span><\/div><\/div>
title(<\/span>'Image A (length of F horn, 12 feet)'<\/span>)<\/span><\/span><\/div><\/div>
<\/div><\/div>
nexttile<\/span><\/span><\/div><\/div>
imshow(B,<\/span>'Interpolation'<\/span>,<\/span>'bilinear'<\/span>)<\/span><\/span><\/div><\/div>
title(<\/span>'Image B (length of B-flat horn, 9 feet)'<\/span>)<\/span><\/span><\/div><\/div>
<\/div><\/div>
nexttile<\/span><\/span><\/div><\/div>
imshow(C,<\/span>'Interpolation'<\/span>,<\/span>'bilinear'<\/span>)<\/span><\/span><\/div><\/div>
title(<\/span>'Image C (length of B horn, 17 feet)'<\/span>)<\/span><\/span><\/div>
<\/div><\/div><\/div><\/div><\/div>
I wanted to use these pictures to make a composite that lets you visually compare the three pole lengths. The problem is that these three pictures aren't aligned with each other, which makes creating an accurate composite challenging.<\/span><\/div>
Let me use <\/span>imshowpair<\/span> to illustrate what I mean, using images A and B.<\/span><\/div>
clf<\/span><\/span><\/div><\/div>
imshowpair(A,B)<\/span><\/span><\/div>
<\/div><\/div><\/div><\/div><\/div>
It looks like these two images differ from each other by a translation and a small rotation.<\/span><\/div>
To accomplish my goal, then, I need to get these three pictures aligned with each other first. Then I can crop them and create my composite. <\/span><\/div>
Sounds like an image registration and spatial referencing problem!<\/span><\/div>
Here's the plan:<\/span><\/div>
  1. Infer a geometric transform that aligns image B to image A.<\/span><\/li>
  2. Infer a geometric transform that aligns image C to image A.<\/span><\/li>
  3. Using the transforms found above, warp images B and C to the same spatial reference as image A.<\/span><\/li>
  4. Crop all three images closely around me holding the pole in the center.<\/span><\/li>
  5. Create a composite of the three images.<\/span><\/li><\/ol>
    I'll be using tools in the Image Processing Toolbox and the Computer Vision Toolbox.<\/span><\/div>

    Register image B to image A<\/span><\/h2>
    Convert images to grayscale for the registration procedure.<\/span><\/div>
    Ag = rgb2gray(A);<\/span><\/span><\/div><\/div>
    Bg = rgb2gray(B);<\/span><\/span><\/div><\/div><\/div>
    Use SURF feature detection (from the Computer Vision Toolbox) in both images.<\/span><\/div>
    % Detect SURF features<\/span><\/span><\/div><\/div>
    A_points = detectSURFFeatures(Ag);<\/span><\/span><\/div><\/div>
    B_points = detectSURFFeatures(Bg);<\/span><\/span><\/div><\/div><\/div>
    Extract feature info from the detected feature points.<\/span><\/div>
    [A_features,A_valid_points] = extractFeatures(Ag,A_points);<\/span><\/span><\/div><\/div>
    [B_features,B_valid_points] = extractFeatures(Bg,B_points);<\/span><\/span><\/div><\/div><\/div>
    Match up the feature points from the two images.<\/span><\/div>
    index_pairs_B_A = matchFeatures(A_features,B_features);<\/span><\/span><\/div><\/div>
    A_matched_points = A_valid_points(index_pairs_B_A(:,1));<\/span><\/span><\/div><\/div>
    B_matched_points = B_valid_points(index_pairs_B_A(:,2));<\/span><\/span><\/div><\/div><\/div>
    Infer a projective geometric transformation that will align image B with image A.<\/span><\/div>
    tform_B_A = estimateGeometricTransform(B_matched_points,A_matched_points,<\/span>'projective'<\/span>);<\/span><\/span><\/div><\/div>
    tform_B_A.T<\/span><\/span><\/div>
    ans = <\/span>3\u00d73 single matrix<\/span><\/div><\/div>
    1.0211 0.0346 0.0000\r\n -0.0196 1.0139 0.0000\r\n 31.3027 47.2352 1.0000\r\n<\/div>
    <\/div>
    <\/div><\/div><\/div><\/div><\/div><\/div><\/div>
    Align image B to image A. Note especially the use of the <\/span>OutputView<\/span> option of <\/span>imwarp<\/span>; this is used to compute the image warping result exactly in the same spatial rectangle as occupied by image A.<\/span><\/div>
    A_ref = imref2d(size(Ag));<\/span><\/span><\/div><\/div>
    B_reg = imwarp(B, tform_B_A, <\/span>'OutputView'<\/span>, A_ref);<\/span><\/span><\/div><\/div><\/div>
    Visually check the result.<\/span><\/div>
    figure<\/span><\/span><\/div><\/div>
    imshowpair(A,B_reg,<\/span>'Interpolation'<\/span>,<\/span>'bilinear'<\/span>)<\/span><\/span><\/div>
    <\/div><\/div><\/div><\/div><\/div>
    <\/span><\/div>

    Register image C to image A using the same procedure<\/span><\/h2>
    Cg = rgb2gray(C);<\/span><\/span><\/div><\/div>
    C_points = detectSURFFeatures(Cg);<\/span><\/span><\/div><\/div>
    [C_features,C_valid_points] = extractFeatures(Cg,C_points);<\/span><\/span><\/div><\/div>
    index_pairs_C_A = matchFeatures(A_features,C_features);<\/span><\/span><\/div><\/div>
    A_matched_points = A_valid_points(index_pairs_C_A(:,1));<\/span><\/span><\/div><\/div>
    C_matched_points = C_valid_points(index_pairs_C_A(:,2));<\/span><\/span><\/div><\/div>
    tform_C_A = estimateGeometricTransform(C_matched_points,A_matched_points,<\/span>'projective'<\/span>);<\/span><\/span><\/div><\/div>
    C_reg = imwarp(C, tform_C_A, <\/span>'OutputView'<\/span>, A_ref);<\/span><\/span><\/div><\/div>
    imshowpair(A,C_reg,<\/span>'Interpolation'<\/span>,<\/span>'bilinear'<\/span>)<\/span><\/span><\/div>
    <\/div><\/div><\/div><\/div><\/div>

    Crop images to a common rectangle<\/span><\/h2>
    Now that images A, B, and C have all been registered to the spatial reference of image A, they can all three be cropped using the same rectangle in the same position.<\/span><\/div>
    r = [1100 450 600 1600];<\/span><\/span><\/div><\/div>
    A_c = imcrop(A,r);<\/span><\/span><\/div><\/div>
    B_reg_c = imcrop(B_reg,r);<\/span><\/span><\/div><\/div>
    C_reg_c = imcrop(C_reg,r);<\/span><\/span><\/div><\/div><\/div>

    Make a tiled composite<\/span><\/h2>
    Finally, I'll use <\/span>imtile<\/span> to make a tile composite image that can be saved to a JPEG file.<\/span><\/div>
    ABC = imtile({B_reg_c,A_c,C_reg_c},<\/span>...<\/span><\/span><\/div><\/div>
    <\/span>'BorderSize'<\/span>,50,<\/span>...<\/span><\/span><\/div><\/div>
    <\/span>'BackgroundColor'<\/span>,<\/span>'white'<\/span>);<\/span><\/span><\/div><\/div>
    imshow(ABC,<\/span>'Interpolation'<\/span>,<\/span>'bilinear'<\/span>)<\/span><\/span><\/div>
    <\/div><\/div><\/div><\/div><\/div>
    It amazes me that this instrument that I can hold in my lap can really be a tube as long the pole pictured on the right.<\/span><\/div>
    * The modern instrument is not French in origin, so <\/span>French horn<\/span> is a misnomer, strictly speaking. Many players refer to the instrument simply as a <\/span>horn,<\/span> and this is the recommendation of the International Horn Society. In the United States,however, most people who are not orchestral musicians will be confused by the term <\/span>horn<\/span>, and so I use <\/span>French horn<\/span> here.<\/span><\/div>
    <\/div><\/div>