You can prune your results, in multiple ways, some of them are:
- Cross-check: Make a cross-check of your matches (if you use BruteForce-matcher you can set this option)
- Ratio-Test of your matcher: See e.g. http://answers.opencv.org/question/4829/how-to-filter-freakbruteforcematcher-result
- Trickier: Compute transformation (homography) matrix, warp the image according to it and see how many overlap you get, either by computing directly the root-means-square-error or by computing the cross-correlation (matchTemplate()) from your object and the warped image and see if you get a high peak or not.