Poor Rectification Results After Good(?) Intrinsic/Extrinsic Parameter Estimation

edit

Problem: I have been trying to calibrate 4 cameras to use for 3D reconstruction in our lab but my results so far aren't great.

Physical Setup Details: The cameras are mounted to the ceiling of the lab and spaced in an arc. My camera pairs have a large baseline ranging from 1.2m --> 1.4m, they are also angled to point at a focal point near the centre of my desired capture volume. The cameras are 1440x1080 (4:3), frame rate is currently set to 60FPS, synchronisation of the cameras relative to a master camera has been tested and verified to be within 6us.

Software Details: As far as I can tell my intrinsic parameters estimation with calibrateCameraCharuco is good. I am generally getting a root mean square pixel reprojection error of less than 0.5 for each camera. Also, visual inspection of the undistorted images shows no visible distortion. Next, when I calibrate the stereo pairs with stereoCalibrate I get OK results with an rms pixel reprojection error of about 0.9. Inspection of the translation matrix matches the setup in the lab fairly well. My problems begin when I attempt to use stereoRectify to obtain the projection/fundamental matrices for triangulation. My results from stereoRectify are mostly black, and show a lot of warping. In the next sections here I will show my results at each stage.

calibrateCameraCharuco Method/Results: Here is a snippet of my code to show how I am using the functions provided in OpenCV::aruco. Results are from 100 images.

for(int i = 0; i < nFrames; i++) {

        int r = calibPositions.at(i);

        // interpolate using camera parameters
        Mat currentCharucoCorners;
        vector < int > currentCharucoIds;
        aruco::interpolateCornersCharuco(FD.allCorners[r], FD.allIds[r], FD.allImgs[r],
                                            charucoboard, currentCharucoCorners, currentCharucoIds);

        CD.allCharucoCorners.push_back(currentCharucoCorners);
        CD.allCharucoIds.push_back(currentCharucoIds);
        CD.filteredImages.push_back(FD.allImgs[r]);

    }

    // Create imaginary objectPoints for stereo calibration
    WP.objPoints = charucoboard->chessboardCorners;

    if(CD.allCharucoCorners.size() < 8) {
        cerr << "Not enough corners for calibration" << endl;
    }

    // calibrate camera using charuco
    cout << "Calculating reprojection error..." << endl;

    double repError_all =
        aruco::calibrateCameraCharuco(CD.allCharucoCorners, CD.allCharucoIds, charucoboard, FD.imgSize,
                                      CP.cameraMatrix, CP.distCoeffs, CP.rvecs, CP.tvecs, CD.stdDevInt,
                                      CD.stdDevExt, CD.repErrorPerView,
                                      CALIB_RATIONAL_MODEL, TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 1000, 1e-8));

    cout << "Reprojection error for all selected images --> " << repError_all << endl;
    cout << "Calculating reprojection error based on best of calibration images (remove if repErr > thresh)..." << endl;

    int count = 0;
    for (int i = 0; i < nFrames; i++) {
        if (CD.repErrorPerView.at<double>(i,0) > 0.8) {

            CD.allCharucoCorners.erase(CD.allCharucoCorners.begin() + i);
            CD.allCharucoIds.erase(CD.allCharucoIds.begin() + i);
            CD.filteredImages.erase(CD.filteredImages.begin() + i);
            cout << "Removed frame [ " << i << " ] due to poor reprojection error" << endl;
            count++;

        }
    }
    cout << count << " frames removed." << endl;

    CD.repError =
            aruco::calibrateCameraCharuco(CD.allCharucoCorners, CD.allCharucoIds, charucoboard, FD.imgSize,
                                          CP.cameraMatrix, CP.distCoeffs, CP.rvecs, CP.tvecs, CD.stdDevInt,
                                          CD.stdDevExt, CD.repErrorPerView, 0, TermCriteria ...