Ask Your Question
4

readNetFromTensorflow fails on retrained NN

asked 2017-10-04 11:57:49 -0600

Robb gravatar image

updated 2017-10-04 11:59:02 -0600

Hi, I hope someone can help me.

My plan is to train a CNN in Tensorflow and use it in a app that uses OpenCV3.3 . For testing purposes i used the retrain script delivered with Tensorflow and expanded it (Inception V3) with the Flowers. (their Tutorial https://www.tensorflow.org/tutorials/...) I trained two models with it (one mobile the other just a normal one)

The resulting graph can be used in Tensorflow, but it fails to load in Opencv (Python, OpencCv 3.3, Win 7 64 ). The script does not create any Checkpoints so i can't freeze the graph (is it Frozen and can ich check this?) But i used the optimize_for_intereference to remove jpeg decoding.

When i try to load those graphs in OpenCV it throws the following errors:

mobile version:

"...tensorflow\tf_importer.cpp:465: error: (-2) More than one input is Const op in function cv::dnn::experimental_dnn_v1::`anonymous-namespace'::TFImporter::getConstBlob"

non mobile:

"tensorflow\tf_importer.cpp:447: error: (-2) Input layer not found: conv/batchnorm in function cv::dnn::experimental_dnn_v1::`anonymous-namespace'::TFImporter::connect"

Thanks for any help in advance ;-)

edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted
5

answered 2017-10-04 14:51:56 -0600

dkurt gravatar image

Hi, @Robb! It's a quite easy:

Let's retrained graph is called graph.pb. Call optimize_for_inference.py tool to remove an Identity nodes, some training-only nodes, make some fusion (conv+bn):

python ~/tensorflow/tensorflow/python/tools/optimize_for_inference.py \
  --input graph.pb \
  --output opt_graph.pb \
  --frozen_graph True \
  --input_names DecodeJpeg/contents \
  --output_names final_result

Before: image description After: image description

Then remove PlaceholderWithDefault node and preprocessing subgraph by

~/tensorflow/bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
  --in_graph=opt_graph.pb \
  --out_graph=final_graph.pb \
  --inputs=Mul \
  --outputs=final_result \
  --transforms="remove_nodes(op=PlaceholderWithDefault) strip_unused_nodes(type=float, shape=\"1,299,299,3\") sort_by_execution_order"

PlaceholderWithDefault PlaceholderWithDefault

preprocessing subgraph preprocessing subgraph

Enjoy!

import cv2 as cv
import numpy as np

net = cv.dnn.readNetFromTensorflow('final_graph.pb')
inp = np.random.standard_normal([299, 299, 3]).astype(np.float32)
net.setInput(cv.dnn.blobFromImage(inp))
out = net.forward()

Thank you for the interest in OpenCV! All images are done using TensorBoard.

edit flag offensive delete link more

Comments

@dkurt This solution remove batchnorm, would it affect the inference results? May you help me solve this problem, thanks

tham gravatar imagetham ( 2017-10-04 19:15:03 -0600 )edit

@tham, it fuses convolution weights with batch normalization multipliers. I don't think that tool are called optimize_for_inferene would break the inference.

dkurt gravatar imagedkurt ( 2017-10-04 23:25:38 -0600 )edit

First thanks for the detailed answer. I hope this will solve my problems but i can't use bazel to build anything on my machine (windows 7) it always throws errors. I think i will have to solve those problems first.

Robb gravatar imageRobb ( 2017-10-06 07:07:01 -0600 )edit
1

@Robb, May be that can help you in this case https://github.com/tensorflow/tensorf...

dkurt gravatar imagedkurt ( 2017-10-06 11:42:02 -0600 )edit

@dkurt Thanks for your answer. I also did the retraining exercise via https://codelabs.developers.google.co... , and then followed your directions above, but I can't seem to make the DNN module read the retrained MobileNet after I did the suggestions in your answer though... I still get the error "More than one input is Const op in function". Do you have any other tips for deciding which nodes need to be deleted in order to get OpenCV to read the graph? I am using the Java version of OpenCV.

tomdke gravatar imagetomdke ( 2017-10-06 15:49:57 -0600 )edit

@tomdke, try to follow guides at https://github.com/opencv/opencv/pull... or at the header of this PR.

dkurt gravatar imagedkurt ( 2017-10-07 01:23:08 -0600 )edit

@dkurt I could load the retrained graph after doing everything exactly as described, but it seems to be empty? It is not returning any results for any input anymore.

Robb gravatar imageRobb ( 2017-11-01 05:02:43 -0600 )edit

@Robb, you mean net.empty() is True or out is None or every bounding box at out has less confidence than threshold? Have you tried to run the same graph in TensorFlow?

dkurt gravatar imagedkurt ( 2017-11-01 08:36:32 -0600 )edit

@dkurt Not in tf yet, i only tried to load it there (was loaded) in OpenCv, in python, it throws on net.Forward() = error \opencv-3.3.0\modules\dnn\src\layers\reshape_layer.cpp:86: error: (-215) total(srcShape, srcRange.start, srcRange.end) == maskTotal in function cv::dnn::computeShapeByReshapeMask maybe that is a information that helps you help me ;-)

Robb gravatar imageRobb ( 2017-11-01 08:52:05 -0600 )edit

@Robb, The sample at answer works correctly or produces the same error?

dkurt gravatar imagedkurt ( 2017-11-01 08:56:33 -0600 )edit
0

answered 2018-04-05 01:56:11 -0600

zidek gravatar image

updated 2018-04-05 07:08:41 -0600

conversion working but wrong cca lower accurency 20% down output from C++ opencv :-(

result from opencv c++ read

Probability: 75.1135% daisy6.75558e-06,dan0.000468269,ros0.2483,sun9.00434e-05,tul0.751135

result from standard label function from tensorflow for tulip image is OK tulips 0.9127952 roses 0.0797129 sunflowers 0.0060887286 daisy 0.0010076686 dandelion 0.00039551986

maybee is something wrong with read blob I use modified googlenet c++ source, I only changed resolution from 224,244 to 299, 299.

Mat inputBlob = blobFromImage(img, 1.0f, Size(299, 299), Scalar(), true, false); //Convert Mat to batch
//! [Set input blob] net.setInput(inputBlob, inBlobName); //set the network input

files used if you want to reproduce, this error: input image C:\fakepath\tulip.jpg

optimized and trasformed model http://193.87.95.129/inf_predn/final_...

c++ example http://193.87.95.129/inf_predn/opencv...

-retrained model output_graph.pb is OK python_label tested -retrained model final_graph.pb is OK python_label tested some problem to reading model by C++ with same precission?

used tensorflow 1.7 gpu gtx 1050 for retrain

edit flag offensive delete link more

Comments

@zidek, please attach a reference to model to make it easier to reproduce. Test images are also would be useful. Thank you!

dkurt gravatar imagedkurt ( 2018-04-05 03:20:28 -0600 )edit

I had the same problems haven't read about someone getting accurate results (compareable to those in Python) with a retrained net yet.

Robb gravatar imageRobb ( 2018-04-05 03:47:19 -0600 )edit

seems tensorflow 1.7 broke retrained nets completely can't even load one trained in 1.7 with tensorflow . Which version are you using?

Robb gravatar imageRobb ( 2018-04-05 04:38:34 -0600 )edit

@zidek, Both OpenCV and TensorFlow produce the same results using this model. However you try to compare an origin TensorFlow graph which involves preprocessing nodes and a truncated one without it. So we need to reproduce these procedures (Sub and Mul) out of the graph. Print out two y values used in preprocessing subgraph. In example, your image normalized to [-1, 1] (subtract 127.5 and divide by 127.5) produces a prediction [[5.44129289e-04 1.32429996e-04 1.01533994e-01 6.84944959e-03 8.90940011e-01]]. Unfortunately, I don't know the order of classes but the last one has the highest score (>0.89).

dkurt gravatar imagedkurt ( 2018-04-05 07:10:08 -0600 )edit

@dkurt looked into his graph and wrote a script in python that takes a picture and gives the output of the detection for Tensorflow (in this case 1.5) and opencv 3.4.1 . While the numbers are correct they seem to be out of position. (wrong index for them out of Net.forward() no idea if this is my fault will post as an Answer below) edit: the answer appeared above no idea why.

Robb gravatar imageRobb ( 2018-04-06 04:21:02 -0600 )edit

a little improovement of precission with this configuration in c++

Mat inputBlob = blobFromImage(img, 1.0f/ 127.5, Size(299, 299), 127.5, true, false);

now is: Probability: 90.2093% daisy0.00174798,dandelion0.000868017,roses0.0915152,sunflower0.00377592,tulip0.902093

zidek gravatar imagezidek ( 2018-04-09 14:35:46 -0600 )edit
0

answered 2018-04-06 05:01:22 -0600

Robb gravatar image

updated 2018-04-06 06:56:39 -0600

edit: Fixed it it was my error (but the lines below could have some use for others) it was the wrong blob: blob= cv2.dnn.blobFromImage(image,1.0/127.5,(299,299),(127.5,127.5,127.5), True, False)

I threw zideks graph into tensorflow and Opencv with the following scripts (some test functions i copy pasted and changed ) basically you only need to look at OpenCVTest and TensorflowResult.

import sys
import tensorflow as tf
import numpy as np
import cv2
import time
import locale
import argparse

def load_graph(model_file):
  graph = tf.Graph()
  graph_def = tf.GraphDef()

  with open(model_file, "rb") as f:
    graph_def.ParseFromString(f.read())
  with graph.as_default():
    tf.import_graph_def(graph_def)

  return graph


def OpenCVTest(picture,graphToLoad, labels): 
   load_graph(graphToLoad)
   image = cv2.imread(picture)
   print("Size of Picture " + str(sys.getsizeof(image))) 
   blob= cv2.dnn.blobFromImage(image,1,(299,299))
   print("size of blob " + str(sys.getsizeof(blob)) )

   Net=  cv2.dnn.readNetFromTensorflow(graphToLoad)

   print("size of Net " + str(sys.getsizeof(Net)) )
   Net.setInput(blob)
   start = time.time()

   preds = Net.forward()
   end = time.time()
   print("[INFO] classification took {:.5} seconds".format(end - start))
   top = np.argsort(preds[0])[::-1][:5]

   print("OpenCV: ")
   for i in top:
     print(i)
     print(labels[i],preds[0][i])

def TensorflowResult(pic, graphName, labels):
  print("Tensorflow:")  
  file_name = pic
  model_file = graphName
  input_height = 299
  input_width = 299
  input_mean = 128
  input_std = 128
  input_layer = "Mul"
  output_layer = "final_result"

  parser = argparse.ArgumentParser()
  parser.add_argument("--image", help="image to be processed")
  parser.add_argument("--graph", help="graph/model to be executed")
  parser.add_argument("--labels", help="name of file containing labels")
  parser.add_argument("--input_height", type=int, help="input height")
  parser.add_argument("--input_width", type=int, help="input width")
  parser.add_argument("--input_mean", type=int, help="input mean")
  parser.add_argument("--input_std", type=int, help="input std")
  parser.add_argument("--input_layer", help="name of input layer")
  parser.add_argument("--output_layer", help="name of output layer")
  args = parser.parse_args()

  if args.graph:
    model_file = args.graph
  if args.image:
    file_name = args.image
  if args.labels:
    label_file = args.labels
  if args.input_height:
    input_height = args.input_height
  if args.input_width:
    input_width = args.input_width
  if args.input_mean:
    input_mean = args.input_mean
  if args.input_std:
    input_std = args.input_std
  if args.input_layer:
    input_layer = args.input_layer
  if args.output_layer:
    output_layer = args.output_layer

  graph = load_graph(model_file)
  t = read_tensor_from_image_file(file_name,
                                  input_height=input_height,
                                  input_width=input_width,
                                  input_mean=input_mean,
                                  input_std=input_std)

  input_name = "import/" + input_layer
  output_name = "import/" + output_layer
  input_operation = graph.get_operation_by_name(input_name);
  output_operation = graph.get_operation_by_name(output_name);

  with tf.Session(graph=graph) as sess:
    results = sess.run(output_operation.outputs[0],
                      {input_operation.outputs[0]: t})
  results = np.squeeze(results)

  top_k = results.argsort()[-5:][::-1]
  #labels = load_labels(label_file)
  for i in top_k:
    print(i)
    print(labels[i], results[i])

print("prefferred encoding: "+ locale.getpreferredencoding(False))
print("Tensorflow Version: "+tf.__version__)
print("Opencv Version: "+cv2.__version__)


labels = ["daisy","dandelion","roses","sunflower","tulip"]
graphToLoad = "final_graph.pb"
picture = "sun.jpg"
OpenCVTest(picture,graphToLoad, labels)
TensorflowResult(picture, graphToLoad, labels)

The following picture was used and is one of the training pictures of that Tutorial. The positions of the labels i have from his printout in his c++ script. .C:\fakepath\sun.jpg ... (more)

edit flag offensive delete link more

Question Tools

Stats

Asked: 2017-10-04 11:57:49 -0600

Seen: 9,380 times

Last updated: Apr 06 '18