Extracting A4 sheet out of troubling backgrounds

edit

So, my challenge is to extract a A4 paper document out of a mobile phone photo. I'm currently doing it using by applying gaussian blur and canny, find the contours and then pick the biggest most rectangle-ish contour. That works great for photos in which the document is having a uniform dark background but if the background is white or containing other shapes then the whole solution goes bananas.

Any suggestion how to get rid of this problem and be less vulnerable to unexpected backgrounds?

Below you can find two troubling images and my current implementation. I'm using python 2.7.9 and opencv 2.4.9.1

    image = cv2.imread(args["image"])
  ratio = image.shape[0] / 500.0
  orig = image.copy()
  image = imutils.resize(image, height = 500)

  # convert the image to grayscale, blur it, and find edges
  # in the image
  gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  gray = cv2.GaussianBlur(gray, (5, 5), 0)
  edged = cv2.Canny(gray, 75, 200)



  (cnts, _) = cv2.findContours(edged.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
  cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:5]

  # loop over the contours
  for c in cnts:
    # approximate the contour
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.02 * peri, True)

    # if our approximated contour has four points, then we
    # can assume that we have found our screen
    if len(approx) == 4:
      screenCnt = approx
      break

  cv2.drawContours(image, [screenCnt], -1, (0, 255, 0), 2)