Hi! I had a lot of difficulty managing to get OpenCV to work nicely with H.264 streams on Ubuntu, particularly with my ELP camera with on-board H.264 encoding. I am writing up everything that I found out in the hopes that it'll help someone else out, the advice should be portable to other Linux distros with little extra effort.
First thing is please read the whole post here before following any links to other guides that I reference as I may provide an alteration to what is there. Let me know if anything is unclear and I'll edit accordingly.
So, OpenCV can't use the H.264 codecs used by ffmpeg, namely libx264, unless it's all compiled using shared libraries. I found doing this a lot more difficult than I felt that it needed to be mainly because I couldn't find any solid documentation around.
So the first solution is the easiest, and that's to use gstreamer. When you compile OpenCV with cmake, use -DWITH_FFMPEG=OFF and -DWITH_GSTREAMER=ON. I haven't done much testing but this should sort you out. I also wasn't able to encode for some reason, only decode. I was still determined to get it working with ffmpeg however!!
DO NOT FOLLOW EXACTLY but here is the ffmpeg compilation guide where we'll be starting from:
https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
Follow as normal until you reach the libx264 section, here we will doing the manual compilation option but adding the flag --enable-shared to the call to configure. The modified command looks similar to this:
cd ~/ffmpeg_sources && \ git -C x264 pull 2> /dev/null || git clone --depth 1 https://git.videolan.org/git/x264 && \ cd x264 && \ PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --enable-static --enable-shared --enable-pic && \ PATH="$HOME/bin:$PATH" make && \ make install
This is actually sufficient for the purposes here, but if you want the other libraries then it's the same deal, the essential flags are --enable-shared and --enable-pic. This will force the compilation of shared libraries and also position independent code (pic). I ended up also removing the --enable-static flags for safety but I am unsure if it's required.
Now onto the actual ffmpeg compilation. We are again adding --enable-shared and --enable-pic but also --cc="gcc -m64 -fPIC".
This last one tells ffmpeg to use the flags -m64 and -fPIC in calls to gcc. -m64 ensuring 64-bit environment and -fPIC will force position independent code. Perhaps sometimes --enable-pic is sufficient but it wasn't for me.
I also removed --pkg-config-flags="--static" and added --disable-static though I can't guarantee that these are essential but they appeared necessary in my case. The command that I used is so (note no libx265, add --enable-libx265 if you need):
cd ~/ffmpeg_sources && \ wget -O ffmpeg-snapshot.tar.bz2 https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2 && \ tar xjvf ffmpeg-snapshot.tar.bz2 && \ cd ffmpeg && \ PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure \ --prefix="$HOME/ffmpeg_build" \ --cc="gcc -m64 -fPIC" \ --extra-cflags="-I$HOME/ffmpeg_build/include" \ --extra-ldflags="-L$HOME/ffmpeg_build/lib" \ --extra-libs="-lpthread -lm" \ --bindir="$HOME/bin" \ --enable-gpl \ --enable-libass \ --enable-libfdk-aac \ --enable-libfreetype \ --enable-libmp3lame \ --enable-libopus \ --enable-libtheora \ --enable-libvorbis \ --enable-libvpx \ --enable-libx264 \ --enable-nonfree \ --enable-pic \ --disable-static \ --enable-shared && \ PATH="$HOME/bin:$PATH" make && \ make install && \ hash -r
Now you might get an error regarding certain libraries not being shared and needing recompilation with -fPIC. If it's one of the libraries from the ffmpeg guide then you need to recompile from source with --enable-shared as above. If you have any issues here let me know and I will help out.
It also may be from avlib, I had a massive problem with avcodec/mqc.o for a very long time for some reason. The first solution in this case is to completely remove the libraries and recompile. Recompilation is not enough, you need to remove them.
rm -r ~/ffmpeg_sources/ffmpeg
Now try the ffmpeg compilation again. Hopefully that should be enough, I also had to export -fPIC as a global flag like so:
export CXXFLAGS="$CXXFLAGS -fPIC"
So maybe you will have to also.
Now if compilation succeeds, you may still have problems opening ffmpeg, claiming it can't find the required libraries or something similar. In my case that the installation didn't correctly place them in /usr/local/lib/. To solve this simply execute this command:
sudo cp ~/ffmpeg_build/lib/* /usr/local/lib/
You may also need to tell ffmpeg where to look by adding the line:
/usr/local/lib
to the file:
/etc/ld.so.conf
You'll need to use sudo to save. Then run:
sudo ldconfig
Now hopefully all is well, again let me know about any problems or questions and I will help as best as I can.
You should be able to compile OpenCV as normal! Make sure to use the flag -DWITH_FFMPEG=ON.
Also see the guide regarding extra modules at:
https://github.com/opencv/opencv_contrib
I again can't verify whether it is necessary or not for this task but I have always compiled with them so it could be the case.
Good luck!!