Specular Highlights detection

asked 2016-04-15 11:44:43 -0600

4133 ●12 ●41 ●101

updated 2016-08-14 06:36:40 -0600

Hi guys, I am interested to find a way to detect and remove the specularities from a given indoors image. For example give the following image:

you can see that there is the main specular spot in the middle of the image and some minor ones around. I am mostly interested for the main cases but if I could extract the other smaller cases as well that would be perfect.

Searching around the literature I found the following 3 interesting papers: Real-time highlight removal using intensity ratio, Efficient and Robust Specular Highlight Removal and Fast and High Quality Highlight Removal from A Single Image. The first two that they provide the code as well gave not that good results.

you see that it detects the main specular spot but at the same time anything else white-ish is a candidate for detected as specularity as well.

Moreover, I have some special cases:

where it seems to be a specularity but in reality it's an overexposure. Therefore, I would like to hear your ideas regarding how I could optimize the detection or if you have any other implementation/paper that might help. The idea is to create a specular hightlights map and recover afterwards the intensity of pixels, pointed from the map. Thanks.

update: I needed to remove the images due to rights

edit retag flag offensive close merge delete

add a comment

answered 2016-04-16 05:56:21 -0600

pklab

4284 ●7 ●30 ●60

updated 2016-08-14 06:37:18 -0600

theodore

4133 ●12 ●41 ●101

You can't get back information you lost due to pixel saturation. All you can do is to try some synthetic reconstruction using information from neighbour like cv::inpaint function does. Just to show, below is a simple code that use uses cv::inpaint.

//helper function
void Morphology(const Mat &imgIn,Mat &imgOut,int morpOp=MORPH_CLOSE,
                           int minThickess=2,int shape=MORPH_ELLIPSE)
{
  int size = minThickess / 2;
  Point anchor = Point(size, size);
  Mat element = getStructuringElement(shape, Size(2 * size + 1, 2 * size + 1), anchor);
  morphologyEx(imgIn, imgOut, morpOp, element, anchor);
}
void Main_Inpaint()
{
  Mat src,blur,mask,bkMask,fgMask,dst;
  vector<vector<Point> > contours;
  src = imread("pixel-saturation1.png");
  // remove noise
  cv::GaussianBlur(src,blur,Size(),2,2);
  //CREATE A MASK FOR THE SATURATED PIXEL
  int minBrightness=253;
  int dilateSize=20;
  //convert to HSV
  Mat src_hsv,brightness,saturation;
  vector<Mat> hsv_planes;
  cvtColor(blur, src_hsv, COLOR_BGR2HSV);
  split(src_hsv, hsv_planes);
  brightness = hsv_planes[2];
  //get the mask
  threshold(brightness,mask,minBrightness,255,THRESH_BINARY);
  //dialte a bit the selection
  Morphology(mask,mask,MORPH_DILATE,dilateSize);
  //INPAINTING
  float radius=5.0;
  inpaint(src,mask,dst,radius,INPAINT_NS);
  imshow("Navier-Stokes based method",dst);

  inpaint(src,mask,dst,radius,INPAINT_TELEA);
  imshow("Method by Alexandru Telea ",dst);
  //show the selection on src
  findContours(mask, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
  for (int i = 0; i < contours.size(); i++)
    drawContours(src,contours,i,Scalar(0,0,255),2);
  imshow("Inpaint mask",src);
  waitKey(0);
}

The result is pretty nice with 1st image but fails with your 2nd case.

I know it's a completely total different approach, but if you can control camera/acquisition, you could try to reduce the brightness/gain/shutter down to remove spotlight...

It would be easier to recover dark region if a bit of information will survive see Automated Removal of Partial Occlusion Blur for reference.

Another option is High Dynamic Range...

edit flag offensive delete link

Comments

@pklab thanks for the answer, however my main problem is not recovering the specular spot but first detecting it. For recovering, inpaint() could be one option but I have come upon some nice techniques (I do not remember now the sources) that it was used for recovering pixel intensity from shadow areas. Also the paper that you suggest could be an option. However, detecting the specular hotspots seems not be that easy as it seems. Moreover, unfortunately hardware-wise there is not the possibility something to be done. You get my upvote, though ;-).

theodore ( 2016-04-16 09:27:42 -0600 )edit

@theodore sorry for miss understanding... I'm focused on "recovering" despite of title says "detection".

I don't have a ready to use solution... sure it isn't easy but your could start from @Guanta answer here, this paper Real-Time Specular Highlight Removal Using Bilateral Filtering and this also is nice A Survey of Specularity Removal Methods.

pklab ( 2016-04-16 12:31:30 -0600 )edit

@pklab no problem, no need for worries :-). I am aware of @Guanta 's answer as well as the papers that you suggest, though they seem a bit outdated. As I mentioned in my initial post I have already tried the following papers Real-time highlight removal using intensity ratio and here is the code and Efficient and Robust Highlight Removal and code here which seem to be included in the state of the art at the moment but without that good results. Then there is also this work:

theodore ( 2016-04-17 04:53:42 -0600 )edit

Fast and High Quality Highlight Removal from A Single Image which says that outperforms the previous both but I unfortunately they do not share the code at the moment. I am in doubts, because the datasets that they use of evaluation are not that close to a real life case like the one I am showing above for that reason I was asking in case someone has come up with something else.

theodore ( 2016-04-17 04:59:56 -0600 )edit

It would be nice to see here some "battery included" cutting edge algorithm for spot light detection.

Using more than one image is another approach, this Improved seeded region growing for detection of water bodies in aerial images is a bit different but could be a starting point.

Spotlights have low saturation and high brightness such as overexposure or some pure white objects. May be shape and gradient could be useful to do basic segmentation... it would be interesting to build a machine learning approach around this problem. have fun :)

pklab ( 2016-04-18 05:03:09 -0600 )edit

@theodore Just to know, why did you edited my answer ? maybe I'm missing something but 2 versions looks identical.

pklab ( 2016-08-25 12:01:51 -0600 )edit

@pklab just removed the images due to some rights. Otherwise except that, I did not change anything else ;-)

theodore ( 2016-09-01 04:06:10 -0600 )edit

@theodore you are welcome ;-)

pklab ( 2016-09-01 04:57:44 -0600 )edit

add a comment

Specular Highlights detection

1 answer

Comments

Links

Question Tools

Stats

Related questions

Specular Highlights detection edit

1 answer

Comments

Links

Question Tools

Stats

Related questions

Specular Highlights detection