OpenCV/EmguCV Get Color inside Shapes

edit

Hello. I need to be able to detect multiple features in a image and I would like to ask for some advice/help/suggestions.

In this image:

I need to detect:

What I've DONE already: HSV image, Canny, Contours of the squares, Centroid of the squares , Contours of the circles, Centroid of the circles, BoundingBoxes,

What I still NEED TO DO:

Find which colors (Red,Blue,Green,Yellow,etc...) are inside each square. Sometimes there are more than one color inside one square. After that,

if(square have blue and green color) { } else if(square have red and yellow color) { } ...

I was trying to do like centroid.X - (a number) and centroid.Y - (a number) to detect color inside each quadrant but it's not working well because the program can crash if the centroid is near the border of the image.

Can you please help me in C code or just suggestions of an algortim to develop? Is SIFT a viable option??