Calibrating a PTZ camera

edit

I'm trying to retrieve the camera matrix for an Axis PTZ(Pan-Tilt-Zoom) camera in order to project a pixel back to 3D world space.

The camera is placed at a certain height, viewing at a certain angle at an object, and I want to find the 3D coordinate of that detected object.

Calibrating a PTZ camera doesn't really make sense to me as the focal length and possibly the distortion would change if the camera would e.g. zoom. How do you go about calibrating such cameras (i.e. cameras where the focal length can change)? Do there exist any existing projects that tackles this problem?