Get object transform instead of camera pose

edit

I've successfully gotten an animated camera in 3D space using:

retval, rvec, tvec = cv2.solvePnP(objp, corners, K, dist_coef)
camera_pose = (-np.matrix(r_mat).T * np.matrix(tvec)).reshape(-1,3)

In my original shot the camera was still and the object was moving. How do I go from an animated camera and a still object to an animated object and a still camera? I'm using four points (corners) for the solve, and I would like to get the 3D coordinates of those points.

Thanks