Stabilizing landmark points

edit

Read here and there that calcOpticalFlowPyrLK can be used to normalize the jitters. Was trying something like this

lk_params = dict(winSize  = (600,600),
                 maxLevel = 3,
                 criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 30, 0.01))

# calculate optical flow
p1, st, err = cv2.calcOpticalFlowPyrLK(prevGray, image_gray, old_landmarks[0], landmarks[0], **lk_params)

old_landmarks = [p1]
prevGray = cv2.bitwise_and(image_gray,image_gray)

At starts it seems to work, but then goes bananas and gives very weird results. I wonder if there is somebody mastered stabilization and can suggest something. Thank you!