Trouble extracting number in Sudoku grid

edit

I am trying to build a Sudoku grabber using OpenCV. I have so far been able to get the boundary of the puzzle and apply a perspective transform to get a bird's eye view. In the next step, I am having trouble detecting the numbers in the individual boxes. Note: I do not get a perfect square after the perspective transformation. I have some convexity defects at the top. I am not overly worried about that.

In each individual box, I am trying to extract the number using cv2.findContours and filtering out the number based on aspect ratio and area. The problem I face during this step is that the contours are so disjoint, I am certain they are going to wrongly recognized by the OCR system which is the next step. I have attached the image after the perspective transform and also the contours which I have obtained in a few cases.

import numpy as np
import cv2
from matplotlib import pyplot as plt

thresh = cv2.adaptiveThreshold(warp,255,1,1,11,2)
kernel = np.array([0, 1, 0, 1, 1, 1, 0, 1, 0], dtype = "uint8")
kernel = np.reshape(kernel, (3,3))
thresh = cv2.erode(thresh,kernel,iterations = 1)
thresh = cv2.dilate(thresh,kernel,iterations = 2)
# I assume that the perspective transformed image (PTI) is evenly divided into 81 square blocks
# The size of the PTI is 450x450
for x in xrange(0,450,50):
  for y in xrange(0,450,50):
    mid_point_y = ((2*y)+50)/2
    mid_point_x = ((2*x)+50)/2
    number_selected = thresh[mid_point_y-15:mid_point_y+15,mid_point_x-15:mid_point_x+15]
    all_contour, hierarchy = cv2.findContours(number_selected.copy(),cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
   if len(all_contour)!=0:
     cv2.drawContours(number_selected,all_contour,-1,(0,255,0),3)
   plt.figure(1)
   plt.imshow(number_selected, cmap = 'gray')
   plt.title('Number')
   plt.show()

Edit: 1) I have added the relevant code 2) I have not described the aspect ratio and area code. I wanted to see why there was a single contour for some numbers and multiple contours for some others.

figure_1.png figure_1_actual.png figure_2.png figure_2_actual.png figure_3.png figure_3_actual.png

sudoku_contour.jpg