1 | initial version |
You need to find first the Homography. Have a look at this tutorial. Then, it would be easier to reverse the usage of scene points and object points to get the transformation that gives you the rectangle in the way you want (instead of finding the orientation of the rectangle in the scene, you try to find the fronto-parallel orientation, if that does make sense).
You could also have a look at the affine transformation estimation made by the estimageRigidTransform function or the findTransformEEC function.