360° lens for panoramic view and depth

edit

I will need this application for inspections, so I will have to record almost static scenes. So, it would be better to use two cameras and try to implement the stereo vision, thank you for your tip. Do I need to consider some other aspects, or will it be exactly the same thing even if now I'm using 360 degree lens?

In theory you could stack two 360 degree lenses, stereo baseline distance between them, and perform a narrow stereo disparity calculation (and resultant depth map) on them across much of their image extent. These cameras are vertically offset so they are out of each other's view. Note the image information returned is a bit different information than two eyes would return; eyes are horizontally offset; the "shadowed" areas where there is no depth information will be above and below items rather than left or right.I'd also be concerned that it may take a more complex distortion model and calibration for the 360 degree lenses, than with conventional simple convex lenses or pinhole model.

You might find this stereo rig calculator to be helpful in determining desired baseline, etc. (to achieve required depth resolution and other parameters) https://nerian.com/support/resources/...

I'm not sure what you mean by 360 deg lens. Is this a fisheye lens which images a half sphere, or something more sophisticated (can the camera see "behind" itself?).

Have you thought about how you are going to calibrate the cameras?

I have done a lot of testing with various wide FOV lenses (including fisheye) and have found the rational distortion model to do very well up to 150deg or so FOV, even with significant distortion. However for true fisheye lenses the fisheye model works better (particularly at the extremes of the image). Unfortunately (at least with older versions of OpenCV) the calibration was less stable (didn't converge as reliably) and the undistort functions were specific to the fisheye model.

I have been able to achieve very good reprojection error (~0.1 pixel rms) by using a high quality calibration Charuco calibration target. In my experience the Charuco target is the only way to go with high distortion / wide FOV lenses (because it works without having to see the entire target).

I typically have to bootstrap the calibration with Aruco calibration, plus 2 or 3 passes of iterative filtering - the initial Aruco calibration just isn't good enough to predict the image locations of the other features (chessboard corners), so you end up with bad measurements at the extremes of the image. After 2 or 3 iterations I typically am able to get the majority of the visible points.

Thank you for all your suggestions! I added more information on my question so you can get more information about what I would like to do. The lens is like a fisheye and the camera is mounted at the end of the scope like showed in the above figures. I hope you can help me :)

Ah, then my comments about narrow camera stereo disparity are not relevant. With wide FOV cameras, it would seem to me, the depth information will be most accurate in the volume closest to the plane perpendicular to the baseline between the cameras.The farther you get away from that plane, toward the poles pointed to by the ends of the baseline segment, the less is the range of the disparity - at those poles you have effectively zero baseline. However, if you are able to rotate the cameras with respect to each other, rotating the baseline, you may be able to accumulate and merge point cloud data over time to map the entire volume. This would require external calibration at each rotation value.

Thank you for your support! Since the scope is very tiny and small, I think I won't be able to use a wide baseline. At maximum, I can setup a baseline of about 10 mm. Do you think it is too low? Is there any restriction on the length of the baseline and the size of the lens?