In the opencv guide there's the explanation of the formulas behind the process of calibration where radial distortion coefficients are multiplied for "r", the distance between the point and the center, is that distance in pixels or mm?
I just had this problem today and asked myself the same question. If you look at the math you find that the radial distortion is (just showing x component) where r^2 = x^2 + y^2 is:
x' = x( 1 + k1 r^2 + k2 r^4 + k3 r^6)
I originally thought that x was in pixels and therefore if x were merely 10 pixels from center then the r^6 term would dominate no matter what the other terms.
But looking through the math more closely ,x = X/Z and is not in pixels. It is dimensionless. In fact I can imagine that with most cameras with a FOV less than 90 degrees, that Z will be greater than X so X/Z will be less than one and y=Y/Z will be less than one. Now when you use those fractions as inputs to the distortion formulas you get much more reasonable results. The higher order terms are now less dominant than more dominant and it makes more sense.
In other words, r^2 = (X/Z)^2 + (Y/Z)^2 where X, Y, and Z are the 3D camera space coordinates.
This may not help your overall problem with your cameras and this question is very old so you've probably moved on but I hoped to answer your specific question.
