Try to use LBP features instead of Haar. It may give you ~3x speedup. Cascade file for the LBP face detector is available in the OpenCV repository.
Try to increase haar_minSize parameter, because it greatly affects the speed.
Right now you can try CV_HAAR_FIND_BIGGEST_OBJECT, on mobiles it is sometimes true that you need only the largest face.
You can also try to use DetectionBasedTracker class which does tracking. It greatly reduces apparent processing time, because slow detection is performed in a background thread.