OpenCV camera calibration is lagging on RPi2 with IR camera
I have a system running Arch Linux ARM with opencv-2.4.10-3. The Raspberry Pi 2 is connected to a Raspberry Pi Noir Camera. When i run the calibration the camera is lagging with what looks like 15-20s delay.
Any suggestions why this is occurring?
I noticed in top that it is using a nasty amount of resources:
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
708 root 20 0 102.9m 34.5m 124.4 4.0 1:00.95 R `- calibration
config.txt:
start_x=1
cma_lwm=
cma_hwm=
cma_offline_start=
start_file=start_x.elf
fixup_file=fixup_x.dat
gpu_mem=128
calibration.cpp:
#include <iostream>
#include <sstream>
#include <time.h>
#include <stdio.h>
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/highgui/highgui.hpp>
#ifndef _CRT_SECURE_NO_WARNINGS
# define _CRT_SECURE_NO_WARNINGS
#endif
using namespace cv;
using namespace std;
static void help()
{
cout << "This is a camera calibration sample." << endl
<< "Usage: calibration configurationFile" << endl
<< "Near the sample file you'll find the configuration file, which has detailed help of "
"how to edit it. It may be any OpenCV supported file format XML/YAML." << endl;
}
class Settings
{
public:
Settings() : goodInput(false) {}
enum Pattern { NOT_EXISTING, CHESSBOARD, CIRCLES_GRID, ASYMMETRIC_CIRCLES_GRID };
enum InputType {INVALID, CAMERA, VIDEO_FILE, IMAGE_LIST};
void write(FileStorage& fs) const //Write serialization for this class
{
fs << "{" << "BoardSize_Width" << boardSize.width
<< "BoardSize_Height" << boardSize.height
<< "Square_Size" << squareSize
<< "Calibrate_Pattern" << patternToUse
<< "Calibrate_NrOfFrameToUse" << nrFrames
<< "Calibrate_FixAspectRatio" << aspectRatio
<< "Calibrate_AssumeZeroTangentialDistortion" << calibZeroTangentDist
<< "Calibrate_FixPrincipalPointAtTheCenter" << calibFixPrincipalPoint
<< "Write_DetectedFeaturePoints" << bwritePoints
<< "Write_extrinsicParameters" << bwriteExtrinsics
<< "Write_outputFileName" << outputFileName
<< "Show_UndistortedImage" << showUndistorsed
<< "Input_FlipAroundHorizontalAxis" << flipVertical
<< "Input_Delay" << delay
<< "Input" << input
<< "}";
}
void read(const FileNode& node) //Read serialization for this class
{
node["BoardSize_Width" ] >> boardSize.width;
node["BoardSize_Height"] >> boardSize.height;
node["Calibrate_Pattern"] >> patternToUse;
node["Square_Size"] >> squareSize;
node["Calibrate_NrOfFrameToUse"] >> nrFrames;
node["Calibrate_FixAspectRatio"] >> aspectRatio;
node["Write_DetectedFeaturePoints"] >> bwritePoints;
node["Write_extrinsicParameters"] >> bwriteExtrinsics;
node["Write_outputFileName"] >> outputFileName;
node["Calibrate_AssumeZeroTangentialDistortion"] >> calibZeroTangentDist;
node["Calibrate_FixPrincipalPointAtTheCenter"] >> calibFixPrincipalPoint;
node["Input_FlipAroundHorizontalAxis"] >> flipVertical;
node["Show_UndistortedImage"] >> showUndistorsed;
node["Input"] >> input;
node["Input_Delay"] >> delay;
interprate();
}
void interprate()
{
goodInput = true;
if (boardSize.width <= 0 || boardSize.height <= 0)
{
cerr << "Invalid Board size: " << boardSize.width << " " << boardSize.height << endl;
goodInput = false;
}
if (squareSize <= 10e-6)
{
cerr << "Invalid square size " << squareSize << endl;
goodInput = false;
}
if (nrFrames <= 0)
{
cerr << "Invalid number of frames " << nrFrames << endl;
goodInput = false;
}
if (input.empty()) // Check for valid input
inputType = INVALID;
else
{
if (input[0] >= '0' && input[0] <= '9')
{
stringstream ss(input);
ss >> cameraID;
inputType = CAMERA;
}
else
{
if (readStringList(input, imageList))
{
inputType = IMAGE_LIST;
nrFrames = (nrFrames < (int)imageList.size()) ? nrFrames : (int)imageList.size();
}
else
inputType = VIDEO_FILE;
}
if (inputType == CAMERA)
inputCapture.open(cameraID);
if (inputType == VIDEO_FILE)
inputCapture.open(input);
if (inputType != IMAGE_LIST && !inputCapture.isOpened())
inputType = INVALID;
}
if (inputType == INVALID)
{
cerr << " Inexistent input: " << input;
goodInput = false;
}
flag = 0;
if(calibFixPrincipalPoint) flag |= CV_CALIB_FIX_PRINCIPAL_POINT;
if(calibZeroTangentDist) flag |= CV_CALIB_ZERO_TANGENT_DIST;
if(aspectRatio) flag |= CV_CALIB_FIX_ASPECT_RATIO;
calibrationPattern = NOT_EXISTING;
if (!patternToUse.compare("CHESSBOARD")) calibrationPattern = CHESSBOARD;
if (!patternToUse.compare("CIRCLES_GRID")) calibrationPattern = CIRCLES_GRID;
if (!patternToUse.compare("ASYMMETRIC_CIRCLES_GRID")) calibrationPattern = ASYMMETRIC_CIRCLES_GRID;
if (calibrationPattern == NOT_EXISTING)
{
cerr << " Inexistent camera calibration mode: " << patternToUse << endl;
goodInput = false;
}
atImageList = 0;
}
Mat nextImage()
{
Mat result;
if( inputCapture.isOpened() )
{
Mat view0;
inputCapture >> view0;
view0.copyTo(result);
}
else if( atImageList < (int)imageList.size() )
result = imread(imageList[atImageList++], CV_LOAD_IMAGE_COLOR);
return result;
}
static ...
At which step it is lagging ? If it is lagging only when all the images have been captured and so during the calibration step, it is normal as it is a process that takes many times. One way to reduce the time is to reduce the image resolution (you will have to multiply the intrinsic parameters according to the image resolution used, the distorsion coefficients remain the sames).
If it is lagging when you display the chessboard corners, you can also try to reduce the image resolution. Also, no need to perform subpixel accuracy as it is just a user display, you can do it later.
You can also try to use CALIB_CB_FAST_CHECK.
The rendering of the image is extremely laggy, i cant really see what i am doing in front of the camera with my chess paper. At a point it freezes and unfreezes when it writes "Calibrated", but what i have exactly done with the paper in front of it iis unknown. I see in
topthat there is a massive CPU usage, basically on all cores at a point. How do i reduce the resolution? I am unable to find the configuration for this...I am not sure to understand well what you have done in your code. Anyway the pipeline shoud be in my opinion:
To resize the image, you can use: cv::resize(). You could use only the resize image for the display and calibrate using the plain resolution.
I am basically using the code example from the
samplesdirectory, so i have not altered anything. I am running the live calibration mode where i simply pressgand the calibration begins, this is where the real lagging begins, obviously because it is doing some heavy calculations while streaming. But even before the calibration has started the stream is really laggy.I simply press
gand move the chessboard around, but i can't really see whats happening, due to the lagging.Can i find an example anywhere with the image calibration method instead of the live stream? I read that i have to create a XML file defining the image paths?
So you don't think anything is wrong with OpenCV and my RPi camera with all the lagging?
My processing time using a laptop: video.
As you can see, even with a modern computer and with images with a resolution of 640x480, there can be massive framerate drops when the chessboard pattern is partially viewed.
So, with your Raspberry PI 2, in my opinion:
From my own laptop i get the same results as yours (only a bit lower FPS). Its only my RPi2 which is struggling in video-mode.
Can i use the current implementation to calibrate with the images, or do i need to modify a lot?