Stereo Rectify doesn't rectify even with correct M and D

edit

I cannot figure out a problem with stereoRectify or initUndistortRectifyMap. It might be related to this question but the user didn't post enough information for me to determine it. I do stereo calibration in two passes, because then we can split the intrinsic and extrinsic calibration, but it doesn't seem to work. These are inputs:

You can see radial lens distortion that needs to be corrected. So after each camera calibration I get a pretty good rectified images:

Left Camera Matrix: [1.0316489177476062e+003, 0., 6.5285947709947743e+002, 0., 1.0306889512875423e+003, 5.2195206055917618e+002, 0., 0., 1.]

Left Camera D: [-3.6474261084702730e-001, 1.7160954305339343e-001, 1.3998315284130826e-004, -5.7337307704477671e-005, -4.3398439779963091e-002]

Right Camera Matrix: [1.0294741711514341e+003, 0., 6.5282835377635274e+002, 0., 1.0285664286763802e+003, 5.2393079228919225e+002, 0., 0., 1..]

Right Camera D: [-3.8333204303394736e-001, 2.7030722876029428e-001, 1.2659691660140408e-004, -6.2717216712805863e-004, -1.4868158456944205e-001]

Now I do calibration together. I call stereoCalibrate like this:

double rms = stereoCalibrate(objectPoints, imagePoints[0], imagePoints[1],
                    cameraMatrix[0], distCoeffs[0],
                    cameraMatrix[1], distCoeffs[1],
                    Size(width,height), R, T, E, F,
                    cv::CALIB_USE_INTRINSIC_GUESS + cv::CALIB_FIX_INTRINSIC, TermCriteria(cv::TermCriteria::COUNT+cv::TermCriteria::EPS, 100, 1e-6));

RMS error: 0.716278 Average reprojection error: 0.932500

I get quite sane results. R is almost identity, showing that the cameras are very parallel:

9.9825297964692494e-001, -5.6272429338183161e-002, -1.8011172155504525e-002,
5.6492686748686502e-002,  9.9833134239913945e-001,   1.1962739128096707e-002,
1.7307945283909901e-002, -1.2959339485918969e-002,   9.9976621794804499e-001

And T is also showing correct results:

-2.8507174779791137e+002, -5.9310534218857187e+000,  9.6822612661982728e+000

It shows that the distance between cameras is 285mm, which is as precise as I can physically measure. So it seems all the inputs are correct. Now I call stereoRectify() like so:

    stereoRectify(cameraMatrix[0], distCoeffs[0],
          cameraMatrix[1], distCoeffs[1],
          Size(width,height), R, T, R1, R2, P1, P2, Q,
          cv::CALIB_ZERO_DISPARITY, 1.0, Size(width,height), &validRoi[0], &validRoi[1]);

R1:

9.9804912529642753e-001, -3.5033737187826153e-002, -5.1677662037772229e-002,
3.5384693276990765e-002,  9.9935639816774224e-001,   5.8917673825730115e-003,
5.1437991569745038e-002, -7.7088715031055979e-003,  9.9864643709544143e-001

R2:

9.9920772192907659e-001, 2.0788992329479110e-002, -3.3937386982939709e-002,
-2.1019454438531062e-002, 9.9975827258716765e-001, -6.4481725030493993e-003,
3.3795132377480969e-002, 7.1564091168283793e-003,  9.9940315930866519e-001

P1:

-4.7663940994070654e+003, 0., 4.8584126815795898e+003, 0., 
0.,  -4.7663940994070654e+003, 3.7089497299194336e+003, 0., 
0., 0., 1.,  0.

P2:

-4.7663940994070654e+003, 0., 4.8584126815795898e+003,  1.3598416693461742e+006,
0., -4.7663940994070654e+003, 3.7089497299194336e+003, 0., 
0., 0., 1., 0.

Q:

1., 0., 0., -4.8584126815795898e+003,
0., 1., 0., -3.7089497299194336e+003, 
0., 0., 0., -4.7663940994070654e+003,
0., 0., 3.5051096071344805e-003, 0.

Q also seems correct (1/B = 1/235 = 3.505e-003). But now when I use the remap function I get this:

I use OpenCV3.0. Maybe this was fixed in 3.1? I have calibrated stereo cameras with OpenCV before, and while it was painful it wasn't impossible. I have calibrated numerous times and I always get the same result. When I set Alpha to 0.0 in stereoRectify I get image like this:

But it is wrong as well ... (more)