Use stereo matching algorithms with more than one image pair?

stereo

asked 2012-09-08 02:10:42 -0600

seveque
21 ●1

I would like to use the OpenCV stereo matching algorithms (such as StereoSGBM) with more than one pair of images. I have several pairs of images taken from the same position, but with varying illumination conditions.

Is there an easy way to use the stereo matching algorithms with more than one image pair? I realize that some of the stereo algorithms work on 3-channel images, which would allow encodings of up to 3 1-channel images, but my images are already 3-channel RGB, and I have more than 3 pairs that I'd like to use to compute the matching costs, if possible.

I'd appreciate any ideas on the simplest way to use multiple image pairs with the stereo algorithms. Thank you!

edit retag flag offensive close merge delete

add a comment

3 answers

Sort by » oldest newest most voted

answered 2012-09-12 02:09:10 -0600

Jacek

306 ●1 ●7

updated 2012-09-12 02:11:12 -0600

I don't think any OpenCV stereo algorithms allows to use more than 1 pair of images. StereoSGBM is designed to work with 2 images only. I'm also not aware of any available stereo software that works with many image pairs taken from the same locations but with varying lighting conditions.

Potentially SGBM can be changed to use more than 1 pair of images (assuming all pairs are taken from the same camera positions). One step in SGBM algorithm is to compute correspondence score between a block of pixels on the left image with a block of pixels on the right. It should be sufficient to change only this step of the algorithm, so that correspondence between blocks of pixel on each image pair is taken into account. And it can be good research topic to find out what's the best way to combine correspondence scores from many image pairs (e.g. take minimum score, or maybe sum of scores from each image pair?). But this requires some code change on SGBM algorithm.

edit flag offensive delete link

Comments

Jacek, thanks for your response. I guess you are right that the stereo algorithms currently available in OpenCV do not allow mulitple image pairs as input. As you point out, it may be possible to modify the SGBM implementation to do multiple passes across each image pair during the computation of the pixel matching score.

As far as I know, most people who implement this technique use the sum of scores across all the image pairs. Taking the minimum score doesn't seem like it would be appropriate - in a set of RGB image pairs, for example, two pixels may be very different colors but still match perfectly within a single spectral channel.

seveque ( 2012-09-18 22:36:31 -0600 )edit

add a comment

answered 2012-09-08 13:27:16 -0600

Victor1234
134 ●1 ●8

updated 2012-09-08 13:28:15 -0600

I think your can find some criteria to get mask for each pair for undesirable location, process them (pairs) and merge.

edit flag offensive delete link

Comments

Victor,

Thanks for your idea about masking parts of the image pairs and merging the results after processing.

However, I'd like to use all of the image pairs simultaneously in the computation of the stereo matching cost metric for each pixel (ala "space-time stereo" techniques). Each image pair contains potentially useful information for each pixel, the image pairs just vary in the wavelength of light or pattern of projected illumination.

seveque ( 2012-09-08 23:34:48 -0600 )edit

add a comment

answered 2012-09-12 10:37:06 -0600

Kirill Kornyakov

2792 ●13 ●25 ●52

Am I right that you want to match 3 pairs of RBG images with a single OpenCV call? OpenCV can't do this, but gpu module supports processing of 3-channel images: http://docs.opencv.org/modules/gpu/doc/camera_calibration_and_3d_reconstruction.html.

And I want to clarify something:

Why do you want to process RGB images? If you think that it will give you better quality, do you have any links?
Why don't call matching several times? In fact if you want to build a depth map for 2 pairs, you can't reuse any data. That means that performance will not be affected.

IMHO you can just convert your images to grayscale and then call OpenCV several times. Otherwise please explain where a single call may help you...

edit flag offensive delete link

Comments

I don't want to use just 3 channels of RGB images. Rather, I would like to be able to use an arbitrary number of image pairs to compute the stereo matching score that is used to generate the final disparity map. This technique is known as "spacetime stereo", and there are a number of research papers that discuss the method and demonstrate its effectiveness.

Calling the OpenCV stereo matching function multiple times does not accomplish what I want. I want to use multiple image pairs as input to the stereo pixel matching metric and then to create a single disparity map as output.

seveque ( 2012-09-18 22:22:59 -0600 )edit

So, in fact you need some special behaviour from the stereo matching function. In this situation you should modify the source code, create your custom implementation. Actually it is possible to run OpenCV's stereo on more than one image pair, InputArray allows you to do this without modification of the API. But this is only a trick, and of course you need to modify the method to understand your input and to produce required output.

Kirill Kornyakov ( 2012-09-21 10:04:01 -0600 )edit

Thanks, Kirill, for pointing out that the InputArray type in the current API will allow passing an array of image pairs as input to the stereo matching algorithms. I'll consider modifying the source code to handle this situation as you suggest!

seveque ( 2012-09-26 01:30:46 -0600 )edit

add a comment

Use stereo matching algorithms with more than one image pair?

3 answers

Comments

Comments

Comments

Links

Question Tools

Stats

Related questions

Use stereo matching algorithms with more than one image pair? edit

3 answers

Comments

Comments

Comments

Links

Question Tools

Stats

Related questions

Use stereo matching algorithms with more than one image pair?