Multi object tracking with Haar cascade detection

edit

I suppose you are working on a video with high temporal resolution (at least 10 images / seconds). Indeed, below this, you will have a lot of problems to track your objects.

The solution you can implement depends also on the processing time you can spend to track objects. If you need a close-to-realtime solution, I suggest using sparse Lukas-Kanade tracking algorithm. You will first take point in the center of each objects detected by your classifier, then use calcopticalflowpyrlk to track those points along the sequence.

If you can spend more CPU, you can use dense optical-flow algorithm which are more robust. You will then have to compute the mean translation of the flow field inside the bounding box of each objects to have the movement of your object.