How to create a new thread for object detection task？
I deploy a DNN in my android app, but the FPS is just 4 that makes the frames looks discontinuous, so i intend to create a new thread to handle the object detection task, Here is part of the code:
private Thread mDetectingThread = null;
private boolean startdect = false;
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
mRgba = inputFrame.rgba();
if (mDetectingThread == null) {
mDetectingThread = new ObjectDectThread();
}
if (startdect == false) {
mDetectingThread.start();
}
startdect = true;
return mRgba;
}
class ObjectDectThread extends Thread {
final int IN_WIDTH = 300;
final int IN_HEIGHT = 300;
final double IN_SCALE_FACTOR = 0.007843;
final double MEAN_VAL = 127.5;
final double THRESHOLD = 0.3;
@Override
public void run() {
super.run();
Imgproc.cvtColor(mRgba, mRgba, Imgproc.COLOR_RGBA2RGB);
Mat blob = Dnn.blobFromImage(mRgba, IN_SCALE_FACTOR,
new Size(IN_WIDTH, IN_HEIGHT),
new Scalar(MEAN_VAL, MEAN_VAL, MEAN_VAL), /*swapRB*/false, /*crop*/false);
net.setInput(blob);
Mat detections = net.forward();
int cols = mRgba.cols();
int rows = mRgba.rows();
detections = detections.reshape(1, (int) detections.total() / 7);
for (int i = 0; i < detections.rows(); ++i) {
double confidence = detections.get(i, 2)[0];
if (confidence > THRESHOLD) {
int classId = (int) detections.get(i, 1)[0];
int left = (int) (detections.get(i, 3)[0] * cols);
int top = (int) (detections.get(i, 4)[0] * rows);
int right = (int) (detections.get(i, 5)[0] * cols);
int bottom = (int) (detections.get(i, 6)[0] * rows);
Imgproc.rectangle(mRgba, new Point(left, top), new Point(right, bottom),
new Scalar(0, 255, 0), 2);
String label = classNames[classId] + ": " + confidence;
Imgproc.putText(mRgba, label, new Point(left, top - 5),
Core.FONT_HERSHEY_SIMPLEX, 1, new Scalar(255, 0, 0), 2);
}
}
}
}
However when i run app on my phone, it crashes and report
"java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:724)
at com.example.iguardonxiaomi8se.MainActivity.onCameraFrame(MainActivity.java:236)
at org.opencv.android.CameraBridgeViewBase.deliverAndDrawFrame(CameraBridgeViewBase.java:392)
at org.opencv.android.JavaCameraView$CameraWorker.run(JavaCameraView.java:373)
at java.lang.Thread.run(Thread.java:764)"
Who can give me some suggestions ? Thanks in advance !
Do you attempt to start a new thread for every frame the comes in? At least the code looks like this to me.
Also make sure you dont share opencv instances across threads and use them concurrently . Open CV is not thread safe.
Yes, you are right. Above code will start a new thread for every frame, actually that was not what I want. That was a mistake. I should add an ‘ if’ statement and flag to make the thread start only once, just like the code now after I changed it. What I really want to do is copy and pass every frame to the background thread, then the background return the boundingbox's coordinate to UI thread, UI thread draws a rectangular box. But i don't know how to do it ...
Hmm this is basic programming stuff and not really related to opencv ^^. I try to help anyway.
Have 1 thread which will do the frame grabbing. Have n worker threads where each thread has an instance of the object detector. Distribute the grabbed frames across your worker threads with a coordinator class. Each worker thread should tell the coordinator when its ready to receive a frame and should also notify the coordinator when it has done its work. You dont have to program this by hand - take a look here https://docs.oracle.com/javase/tutori...
So far so good.But I read opencv is not thread safe. So calling opencv methods from multiple threads can fail i am afraid. In this case do the same like above - but with mutiple process/programs instead of threads
@berak If you run several object detector instances on different threads on different frames/mats - do you think there could be a problem(as long as you dont work on the same mat object i dont think there will be problems - but i am not sure)?
Yes, this is mainly a programming stuff (in fact, my major is automotive engineering...). I feel that it is not practical to create one worker thread with one frame, because many threads will be created in a few seconds. I see someone on the Internet saying that using asynctask and handlerthread, I am going to try these two. By the way, now i am looking through this repo
uff, your ObjectDectThread uses (global) class members like mRGBA ;(
and HELL, the
net, -- did you really expect, you could share it between threads ? -- :[ouch yea. naive multithreading, no locks or such
use an AsyncTask, not a thread.