The last time I wrote about spatial transformations, I explained that imtransform uses the function findbounds to locate the transformed image in output space. By default, imtransform computes an output image grid that is just big enough to capture the output image, wherever it is located.
We designed imtransform this way because we thought it was what users would expect most of the time. Generally, this default behavior works out just fine. The big exception, though, is when users call imtransform to apply a pure translation. Let's try it and see.
I = imread('pout.tif');
imshow(I)
Now make an affine tform struct representing a pure translation:
T = [1 0 0; 0 1 0; 50 100 1];
tform = maketform('affine', T);Apply the translation:
I2 = imtransform(I, tform);
And compare the two images:
subplot(1,2,1) imshow(I) title('original image') subplot(1,2,2) imshow(I2) title('translated image')
They're the same! Hence, the confusion.
Here imtransform has been a little "smarter" than users want. By automatically adjusting the output-space grid to capture the output image wherever it moves, the translation gets removed.
There are a couple of things you can do to make this work better. First, you can ask imtransform to give you more information about the output space coordinates of the output image, and then you can use this information when displaying the image. Here's how that might work:
[I2,xdata,ydata] = imtransform(I,tform);
xdata is a two-element vector specifying where the first and last columns of I2 are located in output space. Similarly, ydata specifies where the first and last rows are located. You can pass this information along to the display function, imshow.
subplot(1,2,1) imshow(I) axis on axis([0 400 0 400]) subplot(1,2,2) imshow(I2,'XData',xdata,'YData',ydata) axis on axis([0 400 0 400])
A second technique is to tell imtransform what output-space rectangle to use.
I3 = imtransform(I,tform,'XData',[1 290],'YData',[1 391]); clf imshow(I3)
If you understand the XData and YData parameters, you can use these techniques separately or in combination to implement whatever translation effect you'd like.
Since translation is a regular source of calls to tech support, we are exploring ways to alleviate the confusion. If you have suggestions, please leave a blog comment. Thanks!
Get
the MATLAB code
Published with MATLAB® 7.2
I haven’t found a better explanation about the udata, vdata, xdata and ydata parameters anywhere else. Thank You very much.
abhishek – Thanks, I’m glad this was helpful to you.
I want to be able to do all my transformations (shearing, scaling and rotating) around the center of my image, or around any specified point. how do I do this with imtransform ?
Goutham – See this post.
Hey, Steve. I wanted to say thanks again for the great blog. I have picked up a ton of valuable information from here that has been invaluable in completing my PhD thesis as well as doing my real job day to day. Thank you!
JP—Thanks for the kind words!
Hey Steve. Thanks for the helpful explanation. Why not have a special imtranslation function similar to imrotate in Matlab?
Steffen—We have considered various ways to ease the confusion, including something like imtranslate. How do you think we should specify it? For example, if the input is 480-by-640, and the desired translation is -100 pixels horizontally and -400 pixels vertically, how big should the output image be, and what should it contain?
Hi Steve,
I have been reading your blog/posts for over a weeks and it has been a great place to find answers and clear explanations since I came across it. Thank for the steep learning curve.
I am still having issues when carrying out image registrations with “imtransform” using a TForm inferred with “cp2tform” from control points collected with “cpselect” and refined/fine tuned with “cpcorr”.
There is a border effect and the transformed image has a slight degradation in quality compared like pixel with like pixel in the original image (see below for further details).
I hope this is the right place to post this. I thought I’d start here since this a very relevant thread before posting on the MatlabCentral newsgroup. Please let me if this is answered elsewhere – I have been reading around and could found explanation of the artifacts I am seeing.
Anyhow, I am expecting a pure translation yet a close inspection of the registered image after transformation with tform indicates extra pixels where filled in, increasing the image by 1 pixel along the first 2 dimensions (i.e. from to ).
Also I noted that pixels values were slightly altered and slight “blur” between original and transformed image, which seems to indicate that the transformation using a Tform inferred by cp2tform is not a pure translation. I am also concerned about rounding error of pixel index double to integer (see xdata and ydata at the very bottom of this post).
I pasted below, at the end of my post, the parameters I used. Here are the questions that I hope you can help clarify.
Is the only way to impose a pure translation would be to build a custom function (I am assuming using the controls points)?
Is the fact that I collected 15 controls with cpselect an over-kill and/or the source of “confusion” by adding spurious scaling or rotation?
How can I display the Tform returned by cp2tform in a transformation matrix “form” so that it can be investigated? (I did not see any example illustrated this).
The imtransform interpolation seems to introduce pixels responsible for the slight increase in image size and border effect, even though it is minimized with ‘bound’ parameter. Is the interpolation adding this “border effect” or only contributing to it?
As a side note, the project I am working on involves image registrations before combining them for further analysis.
The images data are set of three scanned images (RGB Tiff file )which are adjacent top/bottom to one another, with a small overlap of less than 100pixels/1cm.
I apologize for the lengthy post and I hope it makes sense.
Thank you in advance for taking the time reply. Looking forward to it.
Donat
____________________________________________
%use maketform to create transformation % trasnformation only involving translation is expected. % Tried different type of transformation. %mytform = cp2tform(Im2_points, Im3_points, 'nonreflective similarity'); mytform = cp2tform(Im2_points, Im3_points, 'linear conformal'); %Besides, the defaults imtransform(Im2, mytform) which %using bilinear interpolation I also tried these. %The 'fill' option seems to worsen/create the border effect. orange = [255 127 0]'; % [registered xdata ydata] = imtransform(Im2, mytform,... % makeresampler({'cubic','nearest'},'bound'),... % 'FillValues',orange); [registered xdata ydata] = imtransform(Im2, mytform,... makeresampler({'cubic','nearest'},'bound'),... 'FillValues',orange);>>> where the derived xdata and ydata from Tform are:
xdata = 1.0e+003 *( -0.0133 1.5267)
ydata = 1.0e+003 *( -1.6028 0.0962)
Donat—Regarding “extra” pixels at the boundary – imtransform is simply trying to capture the entire set of output pixels, each of which occupies a finite area. The translated input image pixels may not exactly cover exactly one output pixel each. You can control the output grid range yourself, using input parameters to imtransform.
Regarding blurring – well, that’s just the nature of interpolation. If the amount of translation is not an integer, then the output pixels have to be interpolated. Try bicubic instead of bilinear interpolation; that will reduce the blurring effect.
Regarding the transform types – you tried nonreflective similarity and linear conformal – both of these are much more general than a pure translation, so you’re bound to get a transform that is not, in fact, a pure translation. You can, however, create an affine transform matrix yourself that is a pure translation. You’ll still see interpolation and edge effects.
The output of cp2tform is an ordinary MATLAB struct. You can examine the contents of the fields in order to find the specific transform matrix used.
Steve,
Good to hear back from you. I have been sieving through a lot of your blogs and their comments and started answering some of the questions I had and came across new ones. I apologize in advance for the lengthy reply. I cut and pasted some of my code (to show syntax) and your replies.
Regarding the matrix form of Transform matrix… I did come across one of your reply about displaying the cp2tform structure output using .tdata, thank you.
%To display the Transformation matrix
mytform.tdata.T
%To display the inverse Transformation matrix
mytform.tdata.Tinv
Until then I was applying tformfwd to an input point to derive the translation vector components.
>Donat—Regarding “extra” pixels at the boundary – imtransform is simply trying to capture the entire set of output pixels, [...] You can control the output grid range yourself, using input parameters to imtransform.
I have been making extensive use of XData and YData parameters returned from imtransform as well as controlling UData and Vdata parameters. Somehow, I managed to get rid of the extra pixels at the boundary by enforcing a pure translation affine transform. (See below)
However, after registration I am still having some issues with consistently combining the registered image with the base image. My set share a narrow overlap at the top and bottom. I got the indexing working to produce the resulting image with the appropriate averaging at the overlap area as follow:
%imlincomb is recommended for better accuracy, i.e. to avoid rouding error
Combined_Image(r1:r2,c1:c2,:)=imlincomb(0.5,Base(r1:r2,c1:c2,:),0.5,registered(r1:r2,c1:c2,:));
The problem is that I can’t seem to implement the code transforming the input image and combining it with its base so that it works consistently in both case:
the top left of base at (1,1) or the top left of input at (1,1). In other word if I switch around before registration which is the base and which is the input, I am having slight error in deriving the index of the overlap area…
Anyhow, without seeing my code this must be confusing to follow – I hope it makes sense to you. In the end I want my code to be bullet proof for others to use.
:?: Shouldn’t this be symmetrical and reciprocal?
:?: Are the any reason I could not have a versatile algorithm?
So I am back to basics on that end and going through your blogs about confusion over imtransform and translation….
>Regarding blurring – well, that’s just the nature of interpolation. If the amount of translation is not an integer, then the output pixels have to be interpolated.
My scans have different quality of details, i.e. a slight blur (out of focus, slight motion blur from vibration), which varies form one scan to another.
:?: Would that mean that considering the uncertainty on how accurate placing accurately the control points is, rounding the amount of translation to nearest integer could be acceptable on a practical level?
>Try bicubic instead of bilinear interpolation; that will reduce the blurring effect.
I used the makeresampler to have more choice in defining the interpolation, e.g. ‘bound’ seemed better to limit border effect. ‘bicubic’ is defined for imtransform but not for makeresampler.
:?: Is bicubic defined as follow with makeresampler({‘cubic’,'cubic’},’bound’) ?
>Regarding the transform types – [...] You can, however, create an affine transform matrix yourself that is a pure translation. You’ll still see interpolation and edge effects.
I did try this already. However I am not sure if my approaches are correct.
The first one is to average the translation vectors for each controls points pairs (both adjusted or not with cpcorr).
For the second approach, I thought better to let cp2tform generate from the control points selected an output transform. I then used it with tformfwd to derive the translation vector TRANSLATION that I re-used to define a new tform as follow:
tform = maketform(‘affine’,TRANSLATION);
[registered xdata ydata] = imtransform(Unregistered, tform,…
makeresampler({‘cubic’,'cubic’},’bound’),…
‘FillValues’, orange,…
‘UData’, [1 size(Base,2)],…
‘VData’, [1 size(Base,1)]);
In effect I discarded/neglected the terms corresponding to scaling and rotation as defined for “nonrecflective-similarity”.
:?: The translation terms might not corresponds to the ideal pure translation but I suspect it is would be close, isn’t it?
A third approach I tried, was to modify directly the findNonreflectiveSimilarity function in cp2tform so that it, I hope, it solves for a pure translation by imposing:
let sc = s*cos(theta)=1
let ss = s*sin(theta)=0
I renamed the modified imtool box function’scp2tform to cp2tformN (See below for details).
:?: Is this hack mathematically correct?
I apologize for the lengthy post. I haven’t been getting any follow up on the posts I wrote on MatlabCentral newsgroup – you are the first to take the time to reply. Thank you.
As a side, note, to complement my current scan images registration project, I’d like to make use of the alpha channel/transparency parameter as you described in your last blog. I thought it would help assess the registration/combination of images if with a slider you could vary transparency from image A to B, mid-point 50%A+50%B. I like imtool so I’d make use of it to implement my first GUI?! (-:
Anyhow, this past three weeks have been a very steep learning curve and I very much appreciate your inputs and the wealth of discussions from your blogs. Looking forward to your reply.
Thanks,
Donat
************* Modified findNonreflectiveSimilarity function from cp2tformN*************
function [trans, output] = findNonreflectiveSimilarity(uv,xy,options)
%
% For a nonreflective similarity:
%
% let sc = s*cos(theta)
% let ss = s*sin(theta)
%
% [ sc -ss
% [u v] = [x y 1] * ss sc
% tx ty]
%[...]
%
K = options.K;
M = size(xy,1);
x = xy(:,1);
y = xy(:,2);
X = [x y ones(M,1) zeros(M,1);
y -x zeros(M,1) ones(M,1) ];
u = uv(:,1);
v = uv(:,2);
U = [u; v];
% We know that X * r = U
if rank(X) >= 2*K
r = X \ U;
else
msg = ‘At least 2 unique points needed to infer nonreflective similarity transform.’;
eid = sprintf(‘Images:%s:twoUniquePointsReq’,mfilename);
error(eid,msg);
end
% sc = r(1);
% ss = r(2);
sc = 1; %let’s see what happens when imposing Scale=1 and Rotation theta=0;
ss = 0;
tx = r(3);
ty = r(4);
Tinv = [sc -ss 0;
ss sc 0;
tx ty 1];
T = inv(Tinv);
T(:,3) = [0 0 1]‘;
trans = maketform(‘affine’, T);
output = [];
In my last post:
the top left of base at (1,1) or the top left of input at (1,1). In other word if I switch around before registration which is the base and which is the input, I am having slight error in deriving the index of the overlap area…
Should have read:
the top left of base image at (1,1) in resulting (larger) image or the top left of input at (1,1) in the resulting image (i.e. after combination). In other word if I switch around before registration which (top or bottom scans) is the base and which is the input, I am having sleight error in deriving the index of the overlap area…
Thanks.
Donat
Donat—
It’s up to you to decide whether rounding translations to the nearest integer is acceptable for your application.
Yes, makeresampler({‘cubic’, ‘cubic’}, …) gets you bicubic.
I doubt that inferring a nonreflective similarity transformation and then just discarding the nontranslation terms is a sound method. I think you should formulate the problem of estimating the translation parameters directly as a least squares problem. See how cp2tform does it for ideas about how to proceed.
Thanks Mr. Eddins,
Your help so far has been much appreciated and the blogs have been very informative. It is a really great thing you are doing and I know that personally it has given much relief and understanding so far! I look forward to future blogs. I’m off to try your latest lessons and hopefully perfect my code.
Best Regards,
Colin
Colin—You’re welcome!
Mr. Eddins,
Your suggested solutions worked beautifully for my problem! Note: anyone using contours to evaluate your alignment/registration, you can use the xdata, ydata so long as you use imcontour(xdata,ydata,I).
Best Regards,
Colin
Hi,
When most people want a pure translation, they want to see movement. Therefore, the output space should be the same as the input space. In the example you gave (For example, if the input is 480-by-640, and the desired translation is -100 pixels horizontally and -400 pixels vertically, how big should the output image be, and what should it contain?), the result would be black for 400/640 of the output vertically and 100/480 horizontally. I think that would make the most sense. And if the user moves the image by more htan the image? The result would be all black.
Thanks for the great explanation, I couldn’t understand why the image didn’t change!
Thanks,
Danielle
Danielle—Thanks for the input. Not everyone agrees about what would make the most sense for a general spatial transformation function like imtransform. We’ve talked about writing a new function that is specifically for translations.
I second Danielle’s recommendation about specialized imtranslate – with provision for an optional argument whether user wants to let the image grow to occupy translated image or stay the same size.
Additionally, I wish to note an interesting feature I came across that I first thought was a bug.
I was computing affine transform based on correlation. I was using ‘FillValues’ of NaN as these registered images were to be arithmetically combined and the ‘surroundings’ were to be thrown out. To my puzzlement, when the affine transform was unity (nothing to do), the first row and the first column of the transformed image were set to NaN! This can be seen from following
i1=double(imread('cameraman.tif')); [height width]=size(i1); yi=[1 height]; xi=[1 width]; T=[1 0 0; 0 1 0; 0 0 1]; tform=maketform('affine',T); [i2 xo yo]=imtransform(i1,tform,'FillValues',0,'XData',xi,'YData',yi); [i3 xo2 yo2]=imtransform(i1,tform,'FillValues',NaN,'XData',xi,'YData',yi); subplot(131); imagesc(i1); title('Original'); axis equal; subplot(132); imagesc(i2); title('Zero translation-fill by zero'); axis equal; subplot(133); imagesc(i3); title('Zero translation-fill by NaN'); axis equal;It turns out that the reason why the first row and first column are NaN is the default bilinear interpolation used by imtransform in computing output space pixels. When I change the interpolation to ‘nearest’ everything is just fine.
did someone tried to compare this function with intel ippiResize c++ function of intel ipp librairy
Michael—It would be more reasonable to compare the imresize function to ippiResize. Also, there are several different “flavors” of ippiResize you might need to look at. One difference I know about is that the ippiResize functions do not do antialiasing, except for a supersampling option that isn’t as good as what imresize does.
Steve thanks for your continued posts on this blog. Apparently it is still an active topic of confusion. I am still confused as to how you can put a custom transform into cp2tform. In particular, how can I make cp2tform perform only a search for translations? If I try to make my own tform with maketform, I cannot put this into cp2tform since that function requires a string input for an algoritm it already knows. I understand imtransform can be made to translate an unregistered image, but is there an image processing toolbox function that can match up control points with just translations (or rotations solely or scaling solely)? Why must nonreflective similarity include all 3? Thanks so much for your help!!
Dean—The function cp2tform does not support finding a translation-only tform. I suggest that you take a look at the subfunction FindNonreflectiveSimilarity inside cp2tform.m. There are comments at the top of this function that detail the math for setting up a least-squares problem. You could do something similar to this for inferring a translation.
Thanks Steve for your response. I found the math in the subfunction and will see what I can do. Have a great weekend!
Hi Steve…Your posts helped me a lot in developing a knowledge about spatial transformation…I want to know if it is possible to translate an irregularly shaped object within an image by keeping the background of the image intact.If yes,then how???
Kaushik—Is the background constant? Or do you know what the background pixels “behind” the object are supposed to be? If not, then I don’t think there’s a way.
If the background is constant then how can i translate such an image??
Kaushik—Suppose you have a binary mask, bw, whose foreground pixels belong to the object in image A that you want to translate. Then try something like this:
Hi Steve,
Thanks a lot for your wonderful post!
I am working on Image Stitching. I now have 2 images which I want to stitch together. Using the imtransform command, I am able to get the xdata and ydata parameters which are the starting points of the second image with respect to the first image. Whats the best way to add the 2 images to produce one image?
Regards,
Anand
Anand—See my post on controlling the input and output grids. You’ll need to transform both images into an identical output grid so that you can blend them.
Another vote for either imtranslate.m, or at least a blurb in the imtransform help why pure translation doesn’t change the image. I wasted a fair bit of time before finding http://blogs.mathworks.com/steve/2006/07/07/spatial-transformations-translation-confusion/
Carsten—Thanks for your input.
Hi Steve,
I have a question about simultaneously registering two images to one another; I thought it was vaguely related to this thread, so I figured I’d post here.
Supposed I have two images imA and imB that overlap somewhat (e.g. they’re two halves of a panorama). I want to register them. If I follow the Matlab Image Registration example, I get on that stays unmodified, and one that gets registered, often getting distorted a lot in the case of a projective or affine transformation.
So, is there a way to calculate two transforms AB and BA to register A to B and B to A respectively, then simply apply part of those two transformations so that the images “meet in the middle”? Do I just divide the two transformation matrices in two?
Thanks.
Hi Steve…Your posts helped me a lot in developing a knowledge about spatial transformation…I want to know if it is possible to translate an irregularly shaped object within an image by keeping the background of the image intact.If yes,then how???
Simon—I don’t know much about panorama stitching in practice. But I do know you can’t just average the transform matrices.
If you can successfully segment the object, then you can assign its pixel values somewhere else in the image. Of course, you’ll have to fill in the gap in the original object location. You might take a look at roifill.
The code below illustrates a problem I’m having with imtransform. I do not see why I get a correct transform with the first set of input points, but not with the second set. Also, I don’t know what it is about the four pairs of control points that make the condition number so high. Any help or ideas would be greatly appreciated.
close all;clear all; % User inputs inputSet = 2; % 1 for a set of input points that works, 2 for a set that does not work numRows = 378; % Rows in original image numCols = 570; % Cols in original image featureRow = 370; % Row location of a feature featureCol = 560; % Col location of a feature % Define basePoints and inputPoints basePoints =[143 95; 428 95; 143 284; 428 284]; if inputSet == 1 inputPoints = [-2715.562638664548 388.123635347757; -2125.110248447010 397.956744564281; ... -2687.235992046404 670.251779320847; -2109.030835350265 646.991296648817]; elseif inputSet == 2 inputPoints = [-2841.364192843382 392.571575290416; -2232.168660713694 402.270514363645; ... -2811.962711649763 679.351410803620; -2215.607991122505 654.591857089487]; % Note: inputSet 2 works if you remove the average row and column locations in inputPoints % (commented out below). I don't understand why this should make a difference % inputPoints(:,1) = inputPoints(:,1) - mean(inputPoints(:,1)); % inputPoints(:,2) = inputPoints(:,2) - mean(inputPoints(:,2)); end % Plot basePoints and inputPoints figure;plot(basePoints(:,1),basePoints(:,2),'b+'); hold on; plot(inputPoints(:,1),inputPoints(:,2),'g+'); % Compute tform tform = cp2tform(inputPoints,basePoints,'projective'); % Compute condition number. It is very high, even though this appears to be a straightforward % transform (see plot above) conditionNum = cond(tform.tdata.T) % Create image. Outlined with 1's. Feature is a 3x3 region of 2's. originalImage = zeros(numRows,numCols); originalImage(1,:) = 1; originalImage(numRows,:) = 1; originalImage(:,1) = 1; originalImage(:,numCols) = 1; originalImage(featureRow-1:featureRow+1,featureCol-1:featureCol+1,:) = 2; % Transform the image [tformedImage,colData,rowData] = imtransform(originalImage,tform,'nearest'); % Locate the feature [rowIndices,colIndices] = find(tformedImage(:,:,1)==2); tformedFeatureCol = colData(1) + mean(colIndices) - 1 tformedFeatureRow = rowData(1) + mean(rowIndices) - 1 % Now perform a manual calculation of where the feature should be located tformedPoint = tform.tdata.T'*[featureCol;featureRow;1]; expectedTformedCol = tformedPoint(1)/tformedPoint(3) expectedTformedRow = tformedPoint(2)/tformedPoint(3) % Compute error in feature location colError = tformedFeatureCol - expectedTformedCol rowError = tformedFeatureRow - expectedTformedRowChris—Please contact technical support.
Of course, I figured it out shortly after sending that message. The relatively large distance between my basePoints and inputPoints increases the condition number and triggers imtransform to stop mapping pixels 1-to-1 in order to reduce output image size. By using
the problem is fixed. Documented feature.
I noticed you are looking for recommendations on how to implement imtransform. I would recommend an optional parameter in imtransform that causes the returned image to be exactly what is expected, including the expected translation and one-to-one pixel mapping. I know there is a concern about exceeding memory limitations, but if this occurs, the error Matlab gives is very clear on the cause of the error.
Chris—Thanks for your input. We recognize that the automatic scale change behavior in imtransform is frequently confusing and we are taking steps to address this.
hi
I am trying to model simple translation on the image by step of non-integer value (eg 2.5 pixel). I was hoping if you can provide me some guidance as I tried imtransform of that translation matrix and using bicubic interpolation but the image is different that expected. There are shades that wasn’t supposed to be there. Hope to hear from you soon.
Thanks.
Regards,
Chew
Chew—You’ll need to say more. What exactly did you try, and why were the results not satisfactory?
Finally found this solution! Been looking for this for so long! thanks. Would be nice if it was mentioned as part of the Help for “cpselect” and “imtransform” :)
Einat—In the imtransform documentation, see the “Pure translation” section under “Tips.”