Introduction
My last significant development project before retiring from MathWorks was to improve the algorithm for imregcorr, which is used to register images based on translation, rigid, or similarity transformation types. This enhancement shipped with Image Processing Toolbox in R2024b, and I want to tell you about it. If you have ever used imregcorr before and gotten unsatisfying results, then you should definitely try it again in R2024b or later.
I will demonstrate the improvements using several image registration questions that I found on MATLAB Answers. When you see filenames in the code below, those files were originally uploaded in the MATLAB Answers post. You can find them there if you want to reproduce these examples yourselves.
I’ll start with one example, then I’ll talk about the algorithm changes, and finally I’ll show more examples.
Example: Star-Field Image Alignment
Nathan posted on 03-Jul-2024 that imregcorr failed to align these two star-field images.
load srcimg SourceImageAdj
load srcref.mat SourceRef
load targetim.mat TargetImageAdj6
load tgtref.mat TargetRef
tiledlayout(1,2)
nexttile
imshow(SourceImageAdj)
title("SourceImageAdj")
nexttile
imshow(TargetImageAdj6)
title("TargetImageAdj6")
The registration result using imregcorr in R2024a is poor. In this visualization, created using imshowpair, misregistered objects show up in color. As you can see, everything is in color.
In R2024b and later, the output of imregcorr is much better.
tform = imregcorr(SourceImageAdj, SourceRef, ...
TargetImageAdj6, TargetRef, ...
TransformType = "similarity")
tform =
simtform2d with properties:
Dimensionality: 2
Scale: 0.9981
RotationAngle: -1.0487
Translation: [0.0617 -0.0056]
R: [2x2 double]
A: [ 0.9979 0.0183 0.0617
-0.0183 0.9979 -0.0056
0 0 1.0000]
Nathan said that he expected the scale to be $1$, the rotation angle to be $-1$, and the translation to be $(0.0640,0)$. You can see from the display of tform above that the output from imregcorr is very close to the expected values.
Visualize the results using imshowpair:
registered_image = imwarp(SourceImageAdj, SourceRef, tform, "linear", ...
OutputView = TargetRef);
figure
imshowpair(TargetImageAdj6, TargetRef, registered_image, TargetRef)
New Algorithm: Normalized Gradient Correlation
In R2024a and earlier, imregcorr was based on Reddy, B. S. and Chatterji, B. N. “An FFT-Based Technique for Translation, Rotation, and Scale-Invariant Image Registration,” IEEE Transactions on Image Processing, Vol. 5, No. 8, August 1996. Roughly speaking, this method looks for the peak in the correlation of the log-polar frequency-domain phases of the two images to estimate the scale and rotation from one image to the other. In many cases, this method fails because there is not a single, well-defined peak at an easily estimated location. With the star-field image pair above, for example, the matrix of correlation values looks like this:
Starting in R2024b, imregcorr uses a technique called normalized gradient correlation (NGC), mostly as described in Tzimiropoulos, Argyriou, Zafeiriou, and Stathaki. “Robust FFT-Based Scale-Invariant Image Registration with Image Gradients,” IEEE Transactions on Pattern Analysis and Machine Intelligence 32, no. 10 (October 2010): 1899–1906, https://doi.org/10.1109/TPAMI.2010.107.
This method directly incorporates gradient information by forming a complex gradient image from each input image:
$$ G(\mathbf{x})=G_x (\mathbf{x})+jG_y (\mathbf{x}) $$where $G_x (\mathbf{x})$ is the horizontal gradient of the input image, and $G_y (\mathbf{x})$ is the vertical gradient.
The paper argues that this form of correlation weights edge pixels more strongly, suppresses contributions from slowly-varying image regions, and produces a sharp peak even when the images contain dissimilar regions, as is often the case in practice. Here is the matrix of normalized gradient correlation values from the star-field image pair above, displayed as an image. The narrow peak is barely visible. The second image shows a zoomed-in view of the peak.
Assessing the New IMREGCORR Algorithm
Changing the output of an existing function in a MathWorks product is risky. Such changes could cause compatibility issues in user code, so they must be very well motivated and thoroughly tested. We were considering such a change, despite the risk, because we had evidence that many users have been unsuccessful trying to apply imregcorr to their data.
To assess the new imregcorr algorithm, I gathered a test suite of 163 images, chosen from a variety of sources and containing a variety of content. Here are thumbnails of the test suite images.
I then ran these images through about a dozen different experiments, covering the similarity, rigid, and translation transformation types, with a range of parameters, as well as speed and memory comparisons. I think there may have been somewhere around 10,000 image registration pairs tested in all.
From the test results, I concluded that the new algorithm’s output was significantly more accurate and precise than the old algorithm in roughly 95% of test cases. The two algorithms produced comparable results in about 4% of test cases. In a very small number of cases, the old algorithm produced a better result than the new one, according to the error metric I was using. However, in almost all of those cases, both algorithms had essentially failed to find a usable registration result.
I also observed that the new algorithm could produce a usable registration result with much less overlap between the two input images. Also, it was mostly faster (but not always), and it used less memory most of the time (but not always).
I put all this data in front of the Image Processing Toolbox team, and everyone agreed with the plan to change the underlying algorithm for imregcorr.
Additional Examples from MATLAB Answers
Registering a High Resolution Image with a Blurry Image
Jeff asked on 07-Dec-2020 for help with registering images having different sizes and resolutions. They were taken with different devices. Here’s what the originals look like.
moving = imread("Smp180_VIS_1.png");
fixed = imread("Smp180_VIS_ref.png");
imshowpair(fixed, moving, "montage")
When you use imregcorr to register these two images in R2024a or earlier, you get a warning message:
Warning: Phase correlation did not result in a strong peak. Resulting registration could be poor.
Indeed, the resulting registration computed in R2024a is poor.
Here is what you get with the improved imregcorr in R2024b or later:
tform = imregcorr(moving, fixed)
tform =
simtform2d with properties:
Dimensionality: 2
Scale: 0.3675
RotationAngle: 8.4500
Translation: [87.4448 63.7681]
R: [2x2 double]
A: [0.3635 -0.0540 87.4448
0.0540 0.3635 63.7681
0 0 1.0000]
[moving_reg, moving_reg_ref] = imwarp(moving, imref2d(size(moving)), tform);
figure
imshowpair(moving_reg, moving_reg_ref, fixed, imref2d(size(fixed)))
Registering a Rotated Camerman
Eirik posted on 28-Oct-2021 that imregcorr could not detect the correct rotation in a synthetically generated example using the cameraman image.
fixed_image = imread("cameraman.tif");
moving_image = imrotate(fixed_image, 125);
tiledlayout(1,2)
nexttile
imshow(fixed_image), title("fixed image")
nexttile
imshow(moving_image), title("moving image")
In R2024a and earlier, imregcorr estimates a rotation angle of about 17 degrees, obviously very far off. In R2024a and later, the function produces a good result.
tform = imregcorr(moving_image, fixed_image)
tform =
simtform2d with properties:
Dimensionality: 2
Scale: 1.0001
RotationAngle: 124.9967
Translation: [377.8203 84.5295]
R: [2x2 double]
A: [-0.5736 -0.8193 377.8203
0.8193 -0.5736 84.5295
0 0 1.0000]
[moving_image_reg,moving_reg_ref] = imwarp(moving_image, imref2d(size(moving_image)), tform);
clf
imshowpair(fixed_image,imref2d(size(fixed_image)),moving_image_reg,moving_reg_ref)
Simple Patterns
MATLAB user cr posted on 19-Aug-2020 that they had trouble getting a good registration on images with simple patterns using feature-based methods in Computer Vision Toolbox. The new version of imregcorr handles the given example correctly.
A2 = imread("A2.png");
A0 = imread("A0.png");
tiledlayout(1,2)
nexttile
imshow(A2), title("A2")
nexttile
imshow(A0), title("A0")
tform = imregcorr(A2,A0)
tform =
simtform2d with properties:
Dimensionality: 2
Scale: 1.0000
RotationAngle: -32.5015
Translation: [317.9212 1.4153e+03]
R: [2x2 double]
A: [1.0e+03 *
0.0008 0.0005 0.3179
-0.0005 0.0008 1.4153
0 0 0.0010]
[A2_reg,A2_reg_ref] = imwarp(A2,tform);
figure
imshowpair(A2_reg, A2_reg_ref, A0, imref2d(size(A0)))
