1 | initial version |
solvePnP should be fine, though you may want to set the flags to use the method that specifically works with 4 point sets. Or use more points.
I think you're doing the projection wrong though. Take a look HERE. This method shows how to get the line of sight from a point in and image and the camera intrinsics and extrinsics. Except the last section, which you'll need to write, where you scale the LOS to be the negative of the cameraTranslation z value. Then add that to the translation and get your result.