Angle between wall and camera using pnpRansac and its precision

edit

I am looking to calculate the angle between my camera(ORBBEC Astra) and the wall. I am implementing this through SolvePnp using OpenCV. My procedure is as follows : I have a chessboard on the wall. I acquire the pixels of the chessboard and the corresponding 3D coordinates, input them to pnp. I am getting rvec and tvec. I go on to calculate the rotation vector and the pose of the camera.

I need the rotation angles obtained to be very precise. I use the following methods for finding the angle and subsequently the accuracy obtained. I run into some problems with my methods..

Here's what I've tried. I stuck my chessboard to a wall in front of the camera. And then stuck the chessboard to an adjacent wall. Here my angle is known, it's going to be 90°(so basically, I move the chessboard not the camera). I use SolvePnp to find tvec and rvec at these two positions. Here's where I get stuck. I get results I don't understand. My code's here

flag = cv.SOLVEPNP_ITERATIVE
rvec = np.zeros((3, 1))
tvec = np.zeros((3, 1))
_, rvec, tvec, inliers= cv.solvePnPRansac(object_points, img_points, mtx, dist)
Rt = cv.Rodrigues(rvec)
Rt = np.transpose(Rt[0])
sy = math.sqrt(Rt[0, 0] * Rt[0, 0] + Rt[1, 0] * Rt[1, 0])
singular = sy < 1e-6

# rotation matrix to Euler Angles
if not singular:
    x = math.atan2(Rt[2,1] , Rt[2,2])* (180 / np.pi)
    y = math.atan2(-Rt[2,0], sy)* (180 / np.pi)
    z = math.atan2(Rt[1,0], Rt[0,0])* (180 / np.pi)

else:
    x = math.atan2(-Rt[1,2], Rt[1,1])* (180 / np.pi)
    y = math.atan2(-Rt[2,0], sy)* (180 / np.pi)
    z = 0

R = np.array([x, y, z])
imagePoints, jacobian = 
v cv.projectPoints(object_points,rvec,tvec,mtx,dist)
pix_r = np.subtract(img_points,imagePoints)
cv.waitKey()

Here, the distance between the camera and wall is 4m. I get rotation matrices for each image using Solvepnp, but the relative angle between the walls is not 90. When chessboard is right in front, I get R = [7.37, 9.32, 0.37] degrees, yaw pitch roll. When chessboard is on an adjacent wall; I get R = [1.62, 2.98, -0.08]. My tvec seems pretty consistent with [46, -71, 3937] and [40,-61,4142] respectively. Using the cv.projectPoints I get and error of about 100 pixels at times.

Is there any other approach I could use for finding the angle between the wall and camera?

Note : the chessboard points are found using Canny edge detection and Hough. It detects some points in image other than the chessboard but I assume the outliers aren't taken into account using pnpRANSAC.

Thanks!

[Edit] Here are the images I'm using, these are depth images from the Orbbec

My matrix array and distortion array are : mtx = np.array([(576.254, 0, 313.154), (0, 577.558, 249.936), (0 ... (more)