Restrict OpenCV descriptor matcher to a ROI for each query KeyPoint

edit

My goal is to find keypoint matches in two images.

I've tried using only ORB. It is able to find good matches but it is too slow for any real-time application(This is for a mobile device). So I want to give an initial approximation of where the KeyPoint might be using KLT(Using only KLT is not accurate enough) so that the descriptors considered when matching reduces.the steps would be

1) find keypoints and descriptors of the first image

2) use KLT to find an approximation of the found keypoints in the next image

3) use the tracked point list to find regions of interest in the next image for each tracked point

5) Consider only the ROI found for a point when matching descriptors of keypoints in the two images

I've tried doing this in a loop where I give the matcher only the selected keypoint and the keypoints found in its ROI.

detector->detect(prevGray,pointsL);
KeyPointsToPoints(pointsL,leftP);
getKLTapproximation(prevGray,leftP,nextGray,RightP);
PointsToKeyPoints(RightP,pointsRapp);
PointsToKeyPoints(leftP,pointsL);
for(int i=0;i<pointsL.size();i++){
    tempHolder.push_back(pointsL[i]);
    //Mat ROI= the calculated ROI around the tracked point pointsRapp[i]
    extractor->compute(prevGray,tempHolder,desc);
    detector->detect(ROI,pointsR);
    extractor->compute(prevGray,tempHolder,desc);
    //do match on the two descriptor lists
    tempHolder.clear();
}

But it is too slow possibly because of the multiple calls to keypoint detection and extraction. Is there an existing OpenCV method that does this?