Technique to find corresponding objects across stereo views

edit

you understand the math well. go with your intuition.

consider a 3D ray/line for an object in the image. lines for the same object, from both camera views, will intersect (or pass closely) in space. lines that intersect/come close may not be of the same object, due to the correspondence problem, as you understand.

you'll need to calculate those lines for your objects. then, to check feasible correspondence, calculate the closest distance for each pair and check if it's small. that's a little bit of linear algebra involving cross products. you'll know that you have an unresolvable ambiguity when multiple candidates come close.

to resolve ambiguities, you can try giving objects identity, and track them over time. give objects position and velocity. associate detections in a new video frame to objects given their predicted position over the time step, then adjust position and velocity estimate. you can formalize this as a Kalman filter, or go with the simple formulation I gave.

as for object detection, since you have white balls on dark background, simple "blob detection" (thresholding, contours/connected components, cv::moments for centroids) is probably good enough. I agree that feature matching entirely unsuitable here.

calculating these lines given image pixel coordinates should involve the camera projection matrix and its pose matrix. "invert" the projection matrix and give it a screen coordinate (inverting is numerically bad, so this is done in one "solve" step). you'll get a point on the line (and image plane), in the camera frame. you can turn that into a vector (4x1 homogeneous, x/y/z as you got them, w-coordinate is 0) and pass it into the pose matrix (4x4), to get that vector in world space. combine with camera origin (0 vector, 4x1) mapped through pose matrix to get the origin point of the line/ray in world space.

offhand I can't list the specific APIs in OpenCV that will all of solve this for you. if OpenCV happens to not have a function you need, maybe for linear algebra, add the Eigen library, or use python+numpy. opencv does do cross product, norms (L2 norm in particular), and matrix multiplication (dot product).