Revision history [back]

I would like to use OpenCV to track an animal's pupil in a video recording of a mouse viewing naturalistic visual scenes. Here is an example of what a given image might look like under the best of circumstances.

And here is an example of what an image might look like under worse circumstances (i.e., during blinking).

My impression from browsing the Q&A is that ellipse detection is the problem I will need to solve for this task. So far I have tried this process:

1. Process the image into a binary format

   import cv2 as cv
   image = cv.imread(<image file path>)
   gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY)
   blur = cv. medianFilter(gray, 11)
   (threshold, binary) = cv.threshold(blur, 50, 255, cv.THRESH_BINARY_INV)
   

2. Extract contours from the binary image

   (image2, contours, hierarchy) = cv.findContours(binary, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_NONE)
   

3. Identify the contour most likely to represent the pupil

   import numpy as np
   (xc, yc) = np.array(gray.shape) / 2 
   dxy = [np.sqrt((yc - cnt.squeeze().mean(0)[1]) ** 2 + (xc - cnt.squeeze().mean(0)[0]) ** 2 for cnt in contours]
   best = np.argsort(dxy)[0]
   pupil = contours[best]
   

4. Fit the best contour with an ellipse

   (xc, yc), (a, b), theta = cv.fitEllipse(pupil)
   

This approach works well for images in which the pupil is entirely visible. Here is an example of the processing and result of this procedure.

Like I said this works pretty well for cases in which the pupil is entirely visible, but for cases in which the pupil is partially occluded by the animal's eyelid, the algorthim breaks down.

I've also tried using a brute force approach (i.e., Hough circle transform), but it seems like this approach depends on the assumption that the object is perfectly circular which the pupil is not because of the angle of the camera. This approach is also computationally expensive which makes it suboptimal for processing long video recordings acquired at a high-frequency sampling rate (1 hour long recordings at 120 - 200 fps in my case).

I came across some more recently developed procedures for ellipse detection which are specifically robust to the case of partially occluded shapes. This paper is from 2006 and this paper is from 2014. These algorithms seem particuarly effective for detection of partially occluded ellipses; however, they don't have an implementation in OpenCV.

My question is two parts (and some). Is it possible to create an algorithm that makes detection of ellipses robust to partial occlusion using only the existing tools availbale in OpenCV, and if so, how would you approach this aim? Second, Is it feasible to implement these alternative algorithms referenced in the papers I linked for ellipse detection in OpenCV? To this second point, I would love to contribute this additional functionality to OpenCV, but I have never programmed in c++; however, I have had some experience contributing to existing projects written in Python. Is it possible to contribute to OpenCV using something written in Python?

I would like to use OpenCV to track an animal's pupil in a video recording of a mouse viewing naturalistic visual scenes. Here is an example of what a given image might look like under the best of circumstances.

And here is an circumstances and another example of what an image might look like under worse circumstances (i.e., during blinking).

poor quality.

My impression from browsing the Q&A is that ellipse detection is the problem I will need to solve for this task. So far I have tried this process:

1. Process the image into a binary format

   import cv2 as cv
   image = cv.imread(<image file path>)
   gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY)
   cv.COLOR_BGR2GRAY) # convert to grayscale
   blur = cv. medianFilter(gray, 11)
   11) # blur out some of the edges
   (threshold, binary) = cv.threshold(blur, 50, 255, cv.THRESH_BINARY_INV)
   cv.THRESH_BINARY_INV) # binarize
   

2. Extract contours from the binary image

   (image2, contours, hierarchy) = cv.findContours(binary, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_NONE)
   

3. Identify the contour most likely to represent the pupil

   import numpy as np
   (xc, yc) = np.array(gray.shape) / 2 
   dxy = [np.sqrt((yc - cnt.squeeze().mean(0)[1]) ** 2 + (xc - cnt.squeeze().mean(0)[0]) ** 2 for cnt in contours]
   best = np.argsort(dxy)[0]
   np.argsort(dxy)[0] # this is the contour closest to the center of the image
   pupil = contours[best]
   

4. Fit the best contour with an ellipse

   (xc, yc), (a, b), theta = cv.fitEllipse(pupil)
   

This approach works well for images in which the pupil is entirely visible. Here is an example of the processing and result of this procedure.

Like I said this works pretty well for cases in which the pupil is entirely visible, but for cases in which the pupil is partially occluded by the animal's eyelid, the algorthim breaks down.

I've also tried using a brute force approach (i.e., Hough circle transform), but it seems like this approach depends on the assumption that the object is perfectly circular which the pupil is not because of the angle of the camera. This approach is also computationally expensive which makes it suboptimal for processing long video recordings acquired at a high-frequency sampling rate (1 hour long recordings at 120 - 200 fps in my case).

I came across some more recently developed procedures for ellipse detection which are specifically robust to the case of partially occluded shapes. This paper is from 2006 and this paper is from 2014. These algorithms seem particuarly effective for detection of partially occluded ellipses; however, they don't have an implementation in OpenCV.

My question is two parts (and some). a half) parts. Is it possible to create an algorithm that makes detection of ellipses robust to partial occlusion using only the existing tools availbale in OpenCV, and if so, how would you approach this aim? Second, Is would it be feasible to implement these alternative the geometric algorithms referenced in the papers I linked for ellipse detection (referenced in the papers I linked) in OpenCV? To this second point, I would love to contribute this additional functionality to OpenCV, but I have never programmed in c++; however, I have had some experience contributing to existing projects written in Python. Is it possible to contribute to OpenCV using something written in Python?