solvePnP fails in a specific case
I am using OpenCV's solvePnPRansac function to estimate the pose of my camera given a pointcloud made from tracked features. My pipeline starts with forming a point cloud from matched features between two cameras, and use that as a reference to estimate the pose of one of the cameras as it starts moving. I have tested this in multiple settings and it works as long as there are enough features to track while the camera is in motion.
Strangely, during a test I did today, I encountered a failure case where solvePnP would just return junk values all the time. What's confusing here is that in this data set, my point cloud is much denser, it's reconstructed pretty accurately from the two views, the tracked number of points (currently visible features vs. features in the point cloud) at any given time was much higher than what I usually have, so theoretically it should have been a breeze for solvePnP, yet it fails terribly. I tried with CV_ITERATIVE, CV_EPNP and even the non RANSAC version of solvePnP. I was just wondering if I am missing something basic here? The scene I am looking at can be seen in these images (image 1 is the scene and feature matches between two perspectives, image 2 is the point cloud for reference)
From my limited knowledge on PnP problem, failure cases could be due to:
What I would do is:
Thanks for the ideas, Eduardo. I am trying them out and will let you know if I have any luck. Just one omore question, you mentioned that the singular points config. would be a failure case, which I am assuming is due to most of them being coplanar? When I was first developing this pipeline, I was running some tests using a simulator, where I could place models of buildings and look at it through simulated cameras. In that case, the cloud was almost flat, yet PNP worked perfectly even without initial estimates. Although the number of points I had in those point clouds were upwards of a 1000. Do you see anything strange in that case?
Coplanar 3D object points are not a problem since we can detect this configuration and use the appropriate method (for example coplanar posit). You can easily estimate the pose of a chessboard pattern with solvePnP for example or when you calibrate a camera.
An example of singular configuration would be a pose estimation with 4 points and on these 4 points, 3 points are collinear in the image plane.
It could be great if you could check if the pose estimation fails because of the point configuration / a limitation in the methods or an issue in OpenCV.
An update:
I think I got confused because I was trying a bunch of different things when I mentioned it in the question, CV_EPNP did work when I tried now with solvePnPRansac. The accuracy is not as good as what CV_ITERATIVE used to give me with other datasets though. I have a Kalman filter running alongside, so I am trying to see if I can tweak it to make the estimates better.
I grabbed a few more datasets today of the same scene with my reference at different baselines etc. CV_ITERATIVE still fails, RANSAC or not. If I give it an initial estimate of the extrinsics, it just returns [0 0 0]. POSIT fails too.
http://imgur.com/a/FP6Y