Image Stitching (Java API)
Hi, I'm trying to stitch two images together, using the OpenCV Java API. I have been following this C++ tutorial here http://ramsrigoutham.com/2012/11/22/panorama-image-stitching-in-opencv/ However, I get the wrong output and I cannot work out the problem.
FAULTY CODE
public class ImageStitching {
static Mat image1;
static Mat image2;
static FeatureDetector fd;
static DescriptorExtractor fe;
static DescriptorMatcher fm;
public static void initialise(){
fd = FeatureDetector.create(FeatureDetector.BRISK);
fe = DescriptorExtractor.create(DescriptorExtractor.SURF);
fm = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);
//images
image1 = Highgui.imread("room2.jpg");
image2 = Highgui.imread("room3.jpg");
//structures for the keypoints from the 2 images
MatOfKeyPoint keypoints1 = new MatOfKeyPoint();
MatOfKeyPoint keypoints2 = new MatOfKeyPoint();
//structures for the computed descriptors
Mat descriptors1 = new Mat();
Mat descriptors2 = new Mat();
//structure for the matches
MatOfDMatch matches = new MatOfDMatch();
//getting the keypoints
fd.detect(image1, keypoints1);
fd.detect(image1, keypoints2);
//getting the descriptors from the keypoints
fe.compute(image1, keypoints1, descriptors1);
fe.compute(image2,keypoints2,descriptors2);
//getting the matches the 2 sets of descriptors
fm.match(descriptors2,descriptors1, matches);
//turn the matches to a list
List<DMatch> matchesList = matches.toList();
Double maxDist = 0.0; //keep track of max distance from the matches
Double minDist = 100.0; //keep track of min distance from the matches
//calculate max & min distances between keypoints
for(int i=0; i<keypoints1.rows();i++){
Double dist = (double) matchesList.get(i).distance;
if (dist<minDist) minDist = dist;
if(dist>maxDist) maxDist=dist;
}
System.out.println("max dist: " + maxDist );
System.out.println("min dist: " + minDist);
//structure for the good matches
LinkedList<DMatch> goodMatches = new LinkedList<DMatch>();
//use only the good matches (i.e. whose distance is less than 3*min_dist)
for(int i=0;i<descriptors1.rows();i++){
if(matchesList.get(i).distance<3*minDist){
goodMatches.addLast(matchesList.get(i));
}
}
//structures to hold points of the good matches (coordinates)
LinkedList<Point> objList = new LinkedList<Point>(); // image1
LinkedList<Point> sceneList = new LinkedList<Point>(); //image 2
List<KeyPoint> keypoints_objectList = keypoints1.toList();
List<KeyPoint> keypoints_sceneList = keypoints2.toList();
//putting the points of the good matches into above structures
for(int i = 0; i<goodMatches.size(); i++){
objList.addLast(keypoints_objectList.get(goodMatches.get(i).queryIdx).pt);
sceneList.addLast(keypoints_sceneList.get(goodMatches.get(i).trainIdx).pt);
}
System.out.println("\nNum. of good matches" +goodMatches.size());
MatOfDMatch gm = new MatOfDMatch();
gm.fromList(goodMatches);
//converting the points into the appropriate data structure
MatOfPoint2f obj = new MatOfPoint2f();
obj.fromList(objList);
MatOfPoint2f scene = new MatOfPoint2f();
scene.fromList(sceneList);
//finding the homography matrix
Mat H = Calib3d.findHomography(obj, scene);
//LinkedList<Point> cornerList = new LinkedList<Point>();
Mat obj_corners = new Mat(4,1,CvType.CV_32FC2);
Mat scene_corners = new Mat(4,1,CvType.CV_32FC2);
obj_corners.put(0,0, new double[]{0,0});
obj_corners.put(0,0, new double[]{image1.cols(),0});
obj_corners.put(0,0,new double[]{image1.cols(),image1.rows()});
obj_corners.put(0,0,new double[]{0,image1.rows()});
Core.perspectiveTransform(obj_corners, scene_corners, H);
//structure to hold the result of the homography matrix
Mat result = new Mat();
//size of the new image - i.e. image 1 + image 2
Size s = new ...
Actually half is not a function but a new matrix based on the result matrix which takes half of the dimensions. Half is the name of the variable, not a function. This is a basic matrix constructor. Understand?
The constructor would be: Mat half = new Mat(result, new Rect(0,0,image2.cols(),image2.rows())); Also have a look at Features2d.drawMatches to see if your matches are okay.
I figured it out after posting the question, and I used the constructor you specified. However, my image stitching doesn't actually work :/ I updated the code. Maybe you can spot something wrong? Is it the way I'm combing the 2 images?
For FeatureDetection I use (and the guy from the tutorial) also Surf like the extractor and the matcher is flannbased. If you have a look at your detected features than it should be clear what happen. There are diagonal matches in a horizontal panorama. They are wrong! you must sort them out.
Indeed, before applying the stitiching, first apply for example ransac, in order to remove all these false matches and only keep horizontal ones.
Thanks for the responses. I have used this version of the homography function: Mat H = Calib3d.findHomography(obj, scene, Calib3d.RANSAC, 1) ...unfortunately, it hasn't helped - in the documentation the re-projection error (4th parameter) is recommended to be 1-10 - i have tried them all to no avail. I have experimented with a number of algorithm combinations for detection, extraction and matching - again, no success. I tried SURF,SURF and FLANN combination, but no success - the number of 'good' matches I got was 20. The diagonal lines tell me something is wrong, yes - but I don't know how to fix it.
When you separate the matches into good and bad ones also have a look where they are located. If you take a horizontal panorama than there can´t be a match at pixel(y,x) (5, 20) and (70, 40). --> Point point1 = keyPoints1[matchesList.get(i).queryIdx].pt; Point point2 = keyPoints2[matchesList.get(i).trainIdx].pt; if (Math.abs(point1.y - point2.y) > 10) // cant be a good match Try also to use grayscale images for detector.
@mayday hey we have similar problem and we cant figure out did u find a solution can u help us?