The first thing to check is whether you compile with full optimizations: set optimization level to -O2/-O3
If that doesn't help you enough, you should play with the stitcher parameters:
double registrationResol()
void setRegistrationResol(double resol_mpx)
double seamEstimationResol()
void setSeamEstimationResol(double resol_mpx)
double compositingResol()
void setCompositingResol(double resol_mpx)
They control the quality and the speed of your stitching process.
Third, you should try it on a Tegra3 device - that's an Android, btw ;)
Finally, if you still do not like the result, why not try to optimize it yourself more? Here is a starting point into image processing optimizations. But beware that you may cut your fingers. http://answers.opencv.org/question/755/object-detection-slow/#760
Which features are you using? Did you try Orb or other features?
Yes, I tried Orb, it works fast, but sometimes it finds it difficult to detect enough features. I came to a conclusion that by changing registrationResol I get much better results with the default feature detector.