Streaming Video¶
Introduction to GStreamer¶
GStreamer is a powerful pipeline-based multimedia framework. It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing. This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products. This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Major GStreamer commands¶
There are two major commands: gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element. gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect¶
Lists all available elements:
Lists all available sink elements:
For example:
# gst-inspect-1.0 | grep sink
pulseaudio: pulsesink: PulseAudio Audio Sink
hls: hlssink2: HTTP Live Streaming sink
hls: hlssink: HTTP Live Streaming sink
bluez: avdtpsink: Bluetooth AVDTP sink
bluez: a2dpsink: Bluetooth A2DP sink
coreelements: filesink: File Sink
coreelements: fdsink: Filedescriptor Sink
coreelements: fakesink: Fake Sink
alsa: alsasink: Audio sink (ALSA)
gdkpixbuf: gdkpixbufsink: GdkPixbuf sink
…
Detailed help information on a specific element:
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank primary + 2 (258)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <[email protected]
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.14.4
License LGPL
Source module gst-plugins-bad
Source release date 2018-10-02
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
GstWaylandVideo
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBG
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBG
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
max-lateness : Maximum number of nanoseconds that a buffer can be late
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 2000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 922337203685477
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Defa
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Defa
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Defa
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
window-width : Wayland sink preferred window width in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
window-height : Wayland sink preferred window height in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
display : Wayland display name to connect to, if not supplied via
flags: readable, writable
String. Default: null
alpha : Wayland surface alpha value, apply custom alpha value to
flags: readable, writable
Float. Range: 0 - 1 Default:
enable-tile : When enabled, the sink propose VSI tile modifier to VPU
flags: readable, writable
Boolean. Default: false
H.264 video decoder element¶
# gst-inspect-1.0 v4l2h264dec
Factory Details:
Rank primary + 1 (257)
Long-name V4L2 H264 Decoder
Klass Codec/Decoder/Video
Description Decodes H264 streams via V4L2 API
Author Nicolas Dufresne <[email protected]>
Plugin Details:
Name video4linux2
Description elements for Video 4 Linux
Filename /usr/lib/gstreamer-1.0/libgstvideo4linux2.so
Version 1.14.4
License LGPL
Source module gst-plugins-good
Source release date 2018-10-02
Binary package GStreamer Good Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoDecoder
+----GstV4l2VideoDec
+----v4l2h264dec
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h264
stream-format: byte-stream
alignment: au
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: NV12
width: [ 1, 32768 ]
height: [ 1, 32768 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "v4l2h264dec0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
device : Device location
flags: readable
String. Default: "/dev/video12"
device-name : Name of the device
flags: readable
String. Default: null
device-fd : File descriptor of the device
flags: readable
Integer. Range: -1 - 2147483647 Default: -1
output-io-mode : Output side I/O mode (matches sink pad)
flags: readable, writable
Enum "GstV4l2IOMode" Default: 0, "auto"
(0): auto - GST_V4L2_IO_AUTO
(1): rw - GST_V4L2_IO_RW
(2): mmap - GST_V4L2_IO_MMAP
(3): userptr - GST_V4L2_IO_USERPTR
(4): dmabuf - GST_V4L2_IO_DMABUF
(5): dmabuf-import - GST_V4L2_IO_DMABUF_IMPORT
capture-io-mode : Capture I/O mode (matches src pad)
flags: readable, writable
Enum "GstV4l2IOMode" Default: 0, "auto"
(0): auto - GST_V4L2_IO_AUTO
(1): rw - GST_V4L2_IO_RW
(2): mmap - GST_V4L2_IO_MMAP
(3): userptr - GST_V4L2_IO_USERPTR
(4): dmabuf - GST_V4L2_IO_DMABUF
(5): dmabuf-import - GST_V4L2_IO_DMABUF_IMPORT
extra-controls : Extra v4l2 controls (CIDs) for the device
flags: readable, writable
Boxed pointer of type "GstStructure"
H.264 video encoder element¶
# gst-inspect-1.0 v4l2h264enc
Factory Details:
Rank primary + 1 (257)
Long-name V4L2 H.264 Encoder
Klass Codec/Encoder/Video
Description Encode H.264 video streams via V4L2 API
Author ayaka <[email protected]>
Plugin Details:
Name video4linux2
Description elements for Video 4 Linux
Filename /usr/lib/gstreamer-1.0/libgstvideo4linux2.so
Version 1.14.4
License LGPL
Source module gst-plugins-good
Source release date 2018-10-02
Binary package GStreamer Good Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----GstV4l2VideoEnc
+----GstV4l2H264Enc
+----v4l2h264enc
Implemented Interfaces:
GstPreset
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: NV12
width: [ 1, 32768 ]
height: [ 1, 32768 ]
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
stream-format: byte-stream
alignment: au
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "v4l2h264enc0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
device : Device location
flags: readable
String. Default: "/dev/video13"
device-name : Name of the device
flags: readable
String. Default: null
device-fd : File descriptor of the device
flags: readable
Integer. Range: -1 - 2147483647 Default: -1
output-io-mode : Output side I/O mode (matches sink pad)
flags: readable, writable
Enum "GstV4l2IOMode" Default: 0, "auto"
(0): auto - GST_V4L2_IO_AUTO
(1): rw - GST_V4L2_IO_RW
(2): mmap - GST_V4L2_IO_MMAP
(3): userptr - GST_V4L2_IO_USERPTR
(4): dmabuf - GST_V4L2_IO_DMABUF
(5): dmabuf-import - GST_V4L2_IO_DMABUF_IMPORT
capture-io-mode : Capture I/O mode (matches src pad)
flags: readable, writable
Enum "GstV4l2IOMode" Default: 0, "auto"
(0): auto - GST_V4L2_IO_AUTO
(1): rw - GST_V4L2_IO_RW
(2): mmap - GST_V4L2_IO_MMAP
(3): userptr - GST_V4L2_IO_USERPTR
(4): dmabuf - GST_V4L2_IO_DMABUF
(5): dmabuf-import - GST_V4L2_IO_DMABUF_IMPORT
extra-controls : Extra v4l2 controls (CIDs) for the device
flags: readable, writable
Boxed pointer of type "GstStructure"
Test Pattern Stream¶
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer. Default test pattern:
Test pattern with specific parameters on the input element:
Camera Loopback¶
Stream video from a camera to a display:
Replace /dev/videoX with the actual camera device. By default, the MIPI camera device is /dev/video0 & the parallel camera device is /dev/video1.
Video Recording¶
The i.MX8QM can encode video to H.264 at 1080p30 using its internal hardware encoder. For example, you can see the src pad capabilities in the v4l2h264enc element details for the complete list of features supported by the H.264 encoder.
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! v4l2h264enc ! avimux ! filesink location=./test_h264.avi
Video Playback¶
The i.MX8QM can decode and playback H.265 encoded video at 4Kp60 and H.264 encoded video at 1080p60 using its internal hardware decoder. For example, you can see the sink pad capabilities in the v4l2h264dec element details for the complete list of features supported by the H.264 decoder.
Example of a video file playback using the playbin element (making use of the i.MX8QM VPU decoder):
# gst-launch-1.0 playbin uri=file:/run/media/sda1/big_buck_bunny.mp4 video-sink="imxvideoconvert_g2d ! queue ! autovideosink"
Example of H.264 video playback while explicitly specifying a pipeline (making use of the i.MX8QM VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! autovideosink
Example of H.265 video playback while explicitly specifying a pipeline (making use of the i.MX8QM VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux ! queue ! h265parse ! v4l2h265dec ! imxvideoconvert_g2d ! queue ! autovideosink
RTP Video streaming¶
Encode video stream from camera using a hardware H.264 encoder and stream it over RTP:
# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! v4l2h264enc ! rtph264pay ! udpsink host=192.168.20.60 port=1223
References¶
- The GStreamer website
- NXP BSP Linux Users Guide, Multimedia section download link