OpenCV predictions much worse compared to Tensorflow

edit

I have a ResNet type of model, that I have simplified and that I would like to use with OpenCV.

Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 50, 50, 1)]  0                                            
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 50, 50, 1)    4           input_1[0][0]                    
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 50, 50, 20)   200         batch_normalization[0][0]        
__________________________________________________________________________________________________
re_lu (ReLU)                    (None, 50, 50, 20)   0           conv2d[0][0]                     
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 50, 50, 20)   80          re_lu[0][0]                      
__________________________________________________________________________________________________

...

conv2d_13 (Conv2D)              (None, 13, 13, 80)   57680       batch_normalization_12[0][0]     
__________________________________________________________________________________________________
add_5 (Add)                     (None, 13, 13, 80)   0           conv2d_14[0][0]                  
                                                                 conv2d_13[0][0]                  
__________________________________________________________________________________________________
re_lu_12 (ReLU)                 (None, 13, 13, 80)   0           add_5[0][0]                      
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, 13, 13, 80)   320         re_lu_12[0][0]                   
__________________________________________________________________________________________________
average_pooling2d (AveragePooli (None, 3, 3, 80)     0           batch_normalization_13[0][0]     
__________________________________________________________________________________________________
flatten (Flatten)               (None, 720)          0           average_pooling2d[0][0]          
__________________________________________________________________________________________________
dense (Dense)                   (None, 2)            1442        flatten[0][0]                    
==================================================================================================

I followed the tutorial to freeze the model and to optimize it using Tensorflow 1.5 tensorflow.python.tools.optimize_for_inference tool.

• https://medium.com/@sebastingarcaacos...

My code:

# Load Tensorflow model
tf_model = keras.models.load_model("path/to/model-dir/")

# Load OpenCV model
cv_model = cv2.dnn.readNetFromTensorflow("model.pb", "model.pbtxt")

# Normalize image
IMG_SIZE = 50
def normalize(img: np.ndarray) -> np.ndarray:
    blob = cv2.resize(img, (IMG_SIZE, IMG_SIZE))
    blob = blob.astype("float32") / 255

    sigma = math.sqrt(np.sum((blob - np.mean(blob))**2) / blob.size)
    blob = (blob - np.mean(blob)) / sigma

    return blob

# Forward for Tensorflow
img = normalize(img)
tf_predictions = tf_model(img).numpy()

# Forward for OpenCV
img = normalize(img)
cv_model.setInput(img.reshape(1, 1, IMG_SIZE, IMG_SIZE))
cv_predictions = model.forward()


# Output results
print("OpenCV:", cv_predictions)
print("TF    :", tf_predictions)

It works great and OpenCV brings 10 times better performance! Just amazing!

But comparing the prediction results between Tensorflow (not freezed) model with OpenCV (freezed and optimized) model I found, that a large percentage (5% or 10%) of predictions differs by a lot. Some predictions even flips from Yes to No. Like an example here

OpenCV: [[0.4270926  0.57290745]]
TF    : [[0.9662978  0.03370225]]

My questions:

• What is the reason for this inaccuracy?

• Can I get my accuracy back by losing some of the amazing performance?

• May it be the precision drop from float32 to float16 and if yes, how can I force OpenCV to use float32?

Infos:

• Tensorflow: 2.3.0 (CPU only version)
• OpenCV: 4.4.0.44
• Python: 3.6.9