1 | initial version |
I was utterly wrong. I don't know what happened in my mind at this time but I looked at the wrong function to compute the error.
The part of code I quoted comes from the function (computeError) for Affine3DEstimator and compute the error distance vector for two sets of 3D points and with the estimated 3D affine transformation matrix.
The correct function called to compute the distance error to find inliers with Ransac findFundamentalMat is this one. In this case, the distance error is the maximum squared distance between the 2D image point and the corresponding epipolar line between the left and the right images.