Copying my github issue here, I am guessing this is the correct way to do this.
https://github.com/opencv/opencv/blob/86a8ff61293ca5e8d3b5a28cb623b4ca36aab74a/modules/calib3d/src/fisheye.cpp#L436
On this line the assert (and another one below) in initUndistortRectifyMap allows an empty matrix to be passed as the new camera matrix P.
Question is why is it okay to pass an empty P matrix and then later on line 469 it is basically set to identity? And then generate a map from distorted to undistorted image with focal lengths set to 1 and camera center 0? What's the point/ is there any use case where this is a valid thing to do? I think it would be more user friendly to force users to supply a valid P matrix, right?
I wasted a whole day getting black images, which was kind of my fault for not reading the documentation more carefully. I think the intent was to: given empty rotation and newcamera matrix, skip rectification and undistort using the other provided camera matrix.
Btw undistort.dispatch.cpp seems to do the intended aforementioned behavior: https://github.com/opencv/opencv/blob/199ddff13b73071529a42cff97ee8286cfc15d02/modules/calib3d/src/undistort.dispatch.cpp#L112
(P.S. what's the meaning of dispatch btw? looks like some parallelized methods but when do they get called?)