Is it possible to get a pointer to two different frames in a video sequence at the same time with OpenCV?

edit

You can access a specific frame in a video sequence like this:

capture.set(CV_CAP_PROP_POS_FRAMES, frame_num);
capture >> frame;

You can then access a separate frame like this:

capture.set(CV_CAP_PROP_POS_FRAMES, frame_num - 20);
capture >> frame2;

However, when you set the capture property with CV_CAP_PROP_POS_FRAMES, it actually moves the first pointer and frame and frame2 end up pointing to the same data, containing the same values.

I want a pointer to both, so I do not have to clone the data (expensive computation) but I can perform comparisons between the two frames. How can this be done with OpenCV?