unexpected results from undistort using charuco camera calibration

edit

Hi there,

I'm in some trouble getting the charuco camera calibration to work. To the best of my knowledge, I do everything right, but the final undistorted images are very much more warped than expected. If there's anyone that can see what I'm doing wrong, help is very much appreciated, I'm a bit stuck at the moment. So here's the case:

I have a setup of 4 camera's, each of which needs to be calibrated. I have 11 pictures of the charuco board 7x7_1000 for each camera, so 44 images in total

these are the raw images (for all camera's):

From what I understood in the tutorial* not all the markers need to be visible for the charuco camera calibration to work (which is the whole idea of the charuco bord) So as far as I can tell, the source images are fine.

I retrieve the markers and the interpolated chessboard corners for the set of images for each camera, and feed these to the v2.aruco.calibrateCameraCharuco function. Everything seems fine, as this image with all but one marker shows:

So then I go on to call the cv2.undistort function, but the result is not what I expect:

this is the code that I wrote, based on the examples in the tutorial:

def draw_charuco_board( filename, board, size=(2000, 2000) ):
    imboard = board.draw(size)
    cv2.imwrite(filename, imboard)


def detect_charuco_corners( full_board_image_gray, board ):
    parameters = cv2.aruco.DetectorParameters_create()
    return cv2.aruco.detectMarkers(full_board_image_gray, board.dictionary, parameters=parameters)


def charuco_camera_calib( board, filename_glob_pattern, do_flip=False, flip_axis=0 ):
    """
    calibrates the camera using the charuco board
    @see https://docs.opencv.org/trunk/d9/d6a/group__aruco.html#ga54cf81c2e39119a84101258338aa7383
    @see https://github.com/opencv/opencv_contrib/blob/master/modules/aruco/samples/calibrate_camera_charuco.cpp
    """
    charuco_corners = []
    charuco_ids = []
    calib_corners = []
    calib_ids = []
    fns = glob.glob(filename_glob_pattern)
    size = None
    for fn in fns:
        image = cv2.imread(fn, flags=cv2.IMREAD_UNCHANGED)
        image_size = tuple(image.shape[:2][::-1])
        if size is None:
            size = image_size
        elif not image_size == size:
            raise RuntimeError( "charuco_camera_calib:images are not the same size. previous: {} last: {}\n\tlast image: {}".format(size,image_size,fn))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        if do_flip:
            image = cv2.flip(image, flip_axis)
        corners, ids, _ = detect_charuco_corners( image, board )
        charuco_corners.append(np.array(corners))
        charuco_ids.append(np.array(ids))

        if len(corners):
            # refine the detection
            for i, corner in enumerate(corners):
                criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_COUNT, 30, 0.1)
                cv2.cornerSubPix(image, corner,
                                winSize = (10,10),
                                zeroZone = (-1,-1),
                                criteria = criteria)
            # interpolate to find all the chessboard corners
            retval, chessboard_corners, chessboard_ids = cv2.aruco.interpolateCornersCharuco( corners, ids, image, board )
            if chessboard_corners is not None and chessboard_ids is not None:
                calib_corners.append(np.array(chessboard_corners))
                calib_ids.append(np.array(chessboard_ids))
        else:
            raise RuntimeError( "charuco_camera_calib:could not get any markers from image: {}".format(fn))

    retval, camera_matrix, dist_coeffs, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco( np.array(calib_corners), np.array(calib_ids), board, size, None, None )

    for i, fn in enumerate(fns):
        image = cv2.imread(fn, flags=cv2.IMREAD_UNCHANGED)
        image = cv2.aruco.drawDetectedMarkers(image, charuco_corners[i], charuco_ids[i])
        image = cv2 ...