Revision history [back]

The problem with this particular scene is that almost all matched points lie on the same plane in 3d space. This is a known degenerate case for fundamental matrix estimation. Take a look these slides for explanation. I tried to increase 2nd nearest neighbor threshold in this code from 0.7 to 0.8:

for i,(m,n) in enumerate(matches):
    if m.distance < 0.8*n.distance: # was 0.7
        good.append(m)

and got the following result:

Note the corrctly matched out-of-plane background points.

In general fundamental matrix estimation is very sensitive to quality of matches and number of outliers, since opencv uses 8-point algorithm for model estimation. Try to work with higher resolution images with rich textures and non-planar scenes.