Hi,
This is not a very technical question per se, but rather one regarding the limitations of OpenCV as a framework. I'm currently looking into designing a bot which can play rhythm games (think Guitar Hero and games with similar mechanics), utilizing computer vision to determine when inputs should be made.
My current design involves using a multi-tracker to track "notes" as objects as they scroll down the lane, and then press the correct button when they reach the lower area of the lane. I'd like to adapt this to use machine learning once I have the computer vision part done, but that's in the future.
For the first game which I'm planning on using this bot with, capturing the entire game window isn't necessary. I figure that capturing just the note lane(s) is a better option, since a smaller area would lead to faster processing (288x482 px in this case, notes have 3 different colors, don't change form as they scroll down, will scroll at different speeds depending on song). The game itself runs at 60 frames per second, so being able to process images at that rate seems necessary in this case.
Now, before I begin even thinking about implementing this design using OpenCV, I have two fundamental inquiries:
1. Would a multi-tracker be capable of performing a task like the one I've stated here? Given that there could be many notes present on-screen at any given time, would a multi-tracker still be able to accurately track all those notes?
2. With what degree of precision would OpenCV be able to track the motion of the notes and detect when they reach the designated "hit-area"? Many modern rhythm games have very small timing windows, one of the main ones that I'd like to use this bot with has a timing window of ~32ms to achieve the highest score for a single note, for instance.
Any answers/pointers are appreciated. If OpenCV would be unsuitable for the task I've specified above, I'd appreciate any advice regarding similar frameworks that I could use for my project here. Or perhaps a different OpenCV-powered approach, rather than using a tracker.