Revision history [back]

Hi all,

I am currently working with huge matrices and get into trouble when the matix reaches a size of approximately 5GB while using successive push_back()-calls from cv::Mat. At a given point the resident memory shown for my app drops from approximately 5GB to a approximately 0.5GB. This is exactly at the same time, when a resize of the matrix is performed. Now the data within the matrix is corrupted.

Interestingly I can make further push_back() calls without any problems. With the exception, that the memory size of the program is 5GB to low, the memory size increases as if nothing has happend. Then, when the matrix reaches the point where the next resize of the matrix is required, i.e., at approximately 8GB, the resident memory flips back to the right size. But the data remains invalid, of course.

Here are some facts:

• System: Linux (Debian GNU 8)
• OpenCV 3.0.0-rc1
• System memory: 16GB RAM

You should be able to reproduce this behaviour with the following toy code:

int main(int argc, char **argv)
{
  // create a dummy line with random data
  float random_data[153];
  std::random_device rd;
  std::uniform_real_distribution<float> dist(0.0, 1.0);
  for (int i = 0; i < 153 ; i++) random_data[i] = dist(rd);

  // create a matrix with random data
  cv::Mat random_mat(1,153,CV_32F,random_data);

  // create a destination matrix with one row for the beginning
  cv::Mat dest_mat = random_mat.clone();

  // add rows
  double vm, rss;
  for (int i = 0; i < 12000000; i++) {
    dest_mat.push_back(random_mat);
    if ( (i%10000) == 0 ) {
      process_mem_usage(vm, rss);
      printf("%9d: virtual %8.2fM resident %8.2fM\n",i, vm/1024, rss/1024);
    }
  }
  return 0;
}

The method void process_mem_usage(double& vm_usage, double& resident_set) gets the memory stats from /proc/self/stat can be found multiple times on github and stackoverflow.

The critical lines from output are:
7960000: virtual 6069.54M resident 4768.39M
7970000: virtual 6069.54M resident 4774.32M
7980000: virtual 8396.09M resident 684.29M
7990000: virtual 8396.09M resident 689.96M
... where the memory usage drops and:
11940000: virtual 8396.09M resident 2995.32M
11950000: virtual 8396.09M resident 3001.25M
11960000: virtual 11885.91M resident 7086.56M
11970000: virtual 11885.91M resident 7092.23M
... where the memory usage flips back to the expected size.

I hope that helps to find the Bug.

Cheers, Jens