How to detect camera rotation

edit

You've asked a fairly complex question, so instead of trying to give you the complete answer, I'll point you in the right direction (based on my limited perspective on the problem you are trying to solve).

Using Camshift, you can find an object center, size, and orientation. http://docs.opencv.org/modules/video/doc/motion_analysis_and_object_tracking.html?highlight=camshift (and you'll see other operations here that will be of benefit)

1. Find a reference object. This assumes you can detect an object with OpenCV (or will investigate how to do it).
2. Once you rotate the camera (in one axis I'm assuming), find the same object and run Camshift on it. The orientation should give the value you can utilize.

I don't know if you are trying to solve a more difficult problem and may need to bring in Jacobian Matrices (though probably not if you take the approach of finding a reference object again (could be the same one in some circumstances)).

You can check this paper,"Using vanishing points to correct camera rotation in images". Obviously, it's time consuming to estimate the rotation between two images, and the result is not accurate.Maybe you should consider use another device.

I don't think there is an image processing method to determine camera rotation, not a reliable one at least. Will Stwart's answer assumes you'll always have detected objects in your image and that you can always compare them between frames to determine relative camera rotation. It also assumes that this object is not moving, or else the rotation change you'd get would be of the object and not of the camera.

It wouldn't even make much sense to use image to determine rotation, since that is a task that can be completed by just adding a 100$ sensor in your setup. IMU sensors can determine rotation very robustly with a lot less error than any image processing method would. They are even used for autonomous navigation, which seems to be your goal exactly. There are many low cost solutions that are easy to integrate in a custom project.

Read more in http://en.wikipedia.org/wiki/Inertial_measurement_unit