writing file from Gstreamer pipleine in a VideoCapture on the Tx2

edit

I'm reading a camera in a GStreamer pipeline on an NVidia TX2. I want the output to be h264 encoded and written to a file. Next to that, I want to have the frames available in OpenCV for further processing.

To do this I made the following pipeline that works using gst-launch:

gst-launch-1.0 nvcamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1280, 
format=(string)I420, framerate=(fraction)30/1' ! tee name=t \
t. ! queue! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! \
h264parse ! qtmux ! \
filesink location=test.mov \
t. ! queue! nvvidconv ! video/x-raw ! xvimagesink -e

This all works and there are no issues with this pipeline. It shows the frames on screen (this part will be replaced by an appsink in OpenCV) and a moviefile is created that can be played.

Please note the "-e" at the end of of the command. To stop recording I press ctrl-C. If I leave the "-e" out, I get a corrupt moviefile that is not playable. This also makes sense when checking the man-page for gst-launch-1.0, since it mentions specifically this case.

No when I translate this to OpenCV in Python I get this:

import cv2

pipeline = ("nvcamerasrc sensor-id=0 ! "
        "video/x-raw(memory:NVMM), width=(int)640, height=(int)360, format=(string)I420, framerate=(fraction)30/1 ! tee name=t "
        "t. ! queue! omxh264enc ! video/x-h264, stream-format=(string)byte-stream ! "
        "h264parse ! qtmux ! "
        "filesink location=test.mov "
        "t. ! queue! nvvidconv ! video/x-raw, format=(string)BGRx ! "
        "videoconvert ! video/x-raw, format=(string)BGR ! appsink")


vc1 = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)
while(True):
    _, vc1_f = vc1.read()

    cv2.imshow('vc1', vc1_f)
    if not cv2.waitKey(1) == -1:
        break

vc1.release()
cv2.destroyAllWindows()

This also runs fine without any errors, the only thing that happens is that I end up with a corrupt movie file on exit. Exactly the same as when leaving out the "-e" option in the gst-launch version.

In all of this I realise I'm misusing the VideoCapture a bit, because that normally isn't used to write files, but only to get frames into OpenCV through the appsink element. Here, however, it is also used to write a file directly by using the gstreamer pipeline.

I can of course create a separate pipeline:

"appsrc ! videoconvert ! omxh264enc ! h264parse ! qtmux ! filesink location=test.mov "

and feed back the frames I get in OpenCV into that by using a VideoWriter (I tested that and it works properly) but that creates some overhead, so I would prefer to not have to do this and keep it clean and stick to my one VideoCapture pipeline.

So basically my question is: is it possible what I want to do here? Or am I just abusing VideoCapture too much? I.e. is there a Python OpenCV equivalent to the gst-launch "-e" option for a VideoCapture, because VideoCapture::release() doesn't do the job.

Thanks!