How to remove the unnecessary line segments from the image?

edit

I want to turn my original image "Original Image" into the target image "Target Image". And I manage to use OpenCV's Hough Transform to remove the line segments. However, the results come out not as expected (Referring to "Resulting image"). From the resulting image, the line segments are not extracted as I want.

I have looked into this web page, but since some line segments are not in a horizontal direction. I find it is unsuitable to apply in my work.

From the resulting image, the left vertical line is divided into two parts. But what I hope is to extract a whole vertical line segments seen in the original image. For the extracted upper line and the skew line segments, they are not shaped as the actual lines in the original image. I wonder how to perfectly remove those lines segments? What is the efficient way to get my work done? Thanks.

My code:

    Mat tempImg,binaryImg, edges;

Mat im = imread("C:\\apple.bmp");

cvtColor(im, tempImg, CV_BGR2GRAY); 

adaptiveThreshold( tempImg, binaryImg, 200, CV_ADAPTIVE_THRESH_MEAN_C , CV_THRESH_BINARY , 7, 15 );

imwrite("binaryImg.png",binaryImg);

Canny(binaryImg, edges, 40, 120, 3);    

threshold(edges, edges, 128, 255.0, THRESH_BINARY);

vector<Vec4i> lines;
HoughLinesP(edges, lines, 1, CV_PI/180, 50, 20, 10 );
cv::Mat outImg = cv::Mat::zeros(binaryImg.rows, binaryImg.cols, CV_8UC3);

for( size_t i = 0; i < lines.size(); i++ )
{
    Vec4i l = lines[i];
    line( outImg, Point(l[0], l[1]), Point(l[2], l[3]), Scalar(255,255,255), 3, CV_AA);
}

imwrite("outImg.bmp",outImg);

Original Image

Target Image

Resulting image

I have tried to use OpenCV's inpainting, but I find the inpainting is not perfect enough. Though I have the mask, the resulting image has still a large area of black region on the left hand side. This area is not removed under OpenCV's impainting. I don't really know why the mask doesn't take into effect. I wonder how to make the impainting better.

My code:

Mat tempImg,binaryImg, edges,eroded;
    Mat im = imread("C:\\apple.bmp");

    cvtColor(im, tempImg, CV_BGR2GRAY); 

    adaptiveThreshold( tempImg, binaryImg, 128, CV_ADAPTIVE_THRESH_MEAN_C , CV_THRESH_BINARY , 7, 15 );


    imwrite("binaryImg.png",binaryImg);

    Canny(binaryImg, edges, 40, 120, 3);    

    threshold(edges, edges, 128, 255.0, THRESH_BINARY);

    vector<Vec4i> lines;
    HoughLinesP(edges, lines, 1, CV_PI/180, 50, 20, 10 );
    cv::Mat outImg = cv::Mat::zeros(binaryImg.rows, binaryImg.cols, CV_8UC3);

    for( size_t i = 0; i < lines.size(); i++ )
    {
        Vec4i l = lines[i];
        line( outImg, Point(l[0], l[1]), Point(l[2], l[3]), Scalar(255,255,255), 3, CV_AA);
    }

    Mat mask,newImg;
    Mat element(10,10,CV_8U,Scalar(1));  
    dilate(outImg,eroded,element); 


    cv::cvtColor(eroded, mask, CV_BGR2GRAY);    
    std::vector<std::vector<cv::Point> > contours;
    cv::findContours(tempImg.clone(), contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);

    for (int i = 0; i < contours.size(); i++)
    {
        cv::Rect r = cv::boundingRect(contours[i]);

        if (std::abs(1 - (tempImg.cols/r.width)) > 0.2)
            cv::drawContours(mask, contours, i, CV_RGB(0,0,0), CV_FILLED);
    }

    imwrite("mask.bmp",mask);
    inpaint(im, mask, newImg, 1, cv::INPAINT_TELEA ...