Streaming Video¶
Introduction to GStreamer¶
GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Major GStreamer commands¶
There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect¶
Lists all available elements:
Lists all available sink elements:
For example:
# gst-inspect-1.0 | grep sink
alsa: alsasink: Audio sink (ALSA)
app: appsink: AppSink
autodetect: autoaudiosink: Auto audio sink
autodetect: autovideosink: Auto video sink
bluez: a2dpsink: Bluetooth A2DP sink
bluez: avdtpsink: Bluetooth AVDTP sink
coreelements: fakesink: Fake Sink
coreelements: fdsink: Filedescriptor Sink
coreelements: filesink: File Sink
curl: curlfilesink: Curl file sink
curl: curlftpsink: Curl ftp sink
curl: curlhttpsink: Curl http sink
curl: curlsmtpsink: Curl smtp sink
dash: dashsink: DASH Sink
debug: testsink: Test plugin
debugutilsbad: checksumsink: Checksum sink
debugutilsbad: fakeaudiosink: Fake Audio Sink
debugutilsbad: fakevideosink: Fake Video Sink
debugutilsbad: fpsdisplaysink: Measure and show framerate on videosink
debugutilsbad: videocodectestsink: Video CODEC Test Sink
decklink: decklinkaudiosink: Decklink Audio Sink
decklink: decklinkvideosink: Decklink Video Sink
edge: edgesink: EdgeSink
fbdevsink: fbdevsink: fbdev video sink
gdkpixbuf: gdkpixbufsink: GdkPixbuf sink
gio: giosink: GIO sink
gio: giostreamsink: GIO stream sink
hls: hlssink: HTTP Live Streaming sink
hls: hlssink2: HTTP Live Streaming sink
...
…
Detailed help information on a specific element:
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank primary + 2 (258)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <[email protected]>, George Kiagiadakis <[email protected]>
Documentation https://gstreamer.freedesktop.org/documentation/waylandsink/#waylandsink-page
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.24.7
License LGPL
Source module gst-plugins-bad
Documentation https://gstreamer.freedesktop.org/documentation/waylandsink/
Source release date 2024-08-21
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)AYUV, (string)RGBA, (string)ARGB, (string)BGRA, (string)ABGR, (string)P010_10LE, (string)v308, (string)RGBx, (string)xRGB, (string)BGRx, (string)xBGR, (string)RGB, (string)BGR, (string)Y42B, (string)NV16, (string)
NV61, (string)YUY2, (string)YVYU, (string)UYVY, (string)I420, (string)YV12, (string)NV12, (string)NV21, (string)Y41B, (string)YUV9, (string)YVU9, (string)BGR16, (string)RGB16 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: DMA_DRM
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
alpha : Wayland surface alpha value, apply custom alpha value to wayland surface
flags: readable, writable
Float. Range: 0 - 1 Default: 0
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
display : Wayland display name to connect to, if not supplied via the GstContext
flags: readable, writable
String. Default: null
drm-device : Path of the DRM device to use for dumb buffer allocation
flags: readable, writable, can be set only at object construction time
String. Default: null
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
enable-tile : When enabled, the sink propose VSI tile modifier to VPU
flags: readable, writable
Boolean. Default: false
fullscreen : Whether the surface should be made fullscreen
flags: readable, writable
Boolean. Default: false
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 5000000
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
processing-deadline : Maximum processing time for a buffer in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 15000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
render-rectangle : The render rectangle ('<x, y, width, height>')
flags: writable
Default: "< >"
GstValueArray of GValues of type "gint" Write only
rotate-method : rotate method
flags: readable, writable
Enum "GstVideoOrientationMethod" Default: 0, "identity"
(0): identity - GST_VIDEO_ORIENTATION_IDENTITY
(1): 90r - GST_VIDEO_ORIENTATION_90R
(2): 180 - GST_VIDEO_ORIENTATION_180
(3): 90l - GST_VIDEO_ORIENTATION_90L
(4): horiz - GST_VIDEO_ORIENTATION_HORIZ
(5): vert - GST_VIDEO_ORIENTATION_VERT
(6): ul-lr - GST_VIDEO_ORIENTATION_UL_LR
(7): ur-ll - GST_VIDEO_ORIENTATION_UR_LL
(8): auto - GST_VIDEO_ORIENTATION_AUTO
(9): custom - GST_VIDEO_ORIENTATION_CUSTOM
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
stats : Sink Statistics
flags: readable
Boxed pointer of type "GstStructure"
average-rate: 0
dropped: 0
rendered: 0
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
window-height : Wayland sink preferred window height in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
window-width : Wayland sink preferred window width in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
The i.MX8M-PLUS VPU video decoder element¶
# gst-inspect-1.0 vpudec
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based video decoder
Klass Codec/Decoder/Video
Description Decode compressed video to raw data
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.9.2
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoDecoder
+----GstVpuDec
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h265
video/x-vp9
video/x-vp8
video/x-h264
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
automatic-request-sync-point-flags: Flags to use when automatically requesting sync points
flags: readable, writable
Flags "GstVideoDecoderRequestSyncPointFlags" Default: 0x00000003, "corrupt-output+discard-input"
(0x00000001): discard-input - GST_VIDEO_DECODER_REQUEST_SYNC_POINT_DISCARD_INPUT
(0x00000002): corrupt-output - GST_VIDEO_DECODER_REQUEST_SYNC_POINT_CORRUPT_OUTPUT
automatic-request-sync-points: Automatically request sync points when it would be useful
flags: readable, writable
Boolean. Default: false
disable-reorder : disable vpu reorder when end to end streaming
flags: readable, writable
Boolean. Default: false
discard-corrupted-frames: Discard frames marked as corrupted instead of outputting them
flags: readable, writable
Boolean. Default: false
frame-drop : enable adaptive frame drop for smoothly playback
flags: readable, writable
Boolean. Default: true
frame-plus : set number of addtional frames for smoothly playback
flags: readable, writable
Unsigned Integer. Range: 0 - 16 Default: 3
max-errors : Max consecutive decoder errors before returning flow error
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
name : The name of the object
flags: readable, writable
String. Default: "vpudec0"
output-format : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
flags: readable, writable
Enum "GstVpuDecOutputFormat" Default: 0, "auto"
(0): auto - enable chroma interleave. (default)
(1): NV12 - NV12 format
(2): I420 - I420 format
(3): YV12 - YV12 format
(4): Y42B - Y42B format
(5): NV16 - NV16 format
(6): Y444 - Y444 format
(7): NV24 - NV24 format
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: true
use-vpu-memory : use vpu allocate video frame buffer
flags: readable, writable
Boolean. Default: true
The i.MX8M-PLUS VPU H.264 video encoder element¶
# gst-inspect-1.0 vpuenc_h264
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based AVC/H264 video encoder
Klass Codec/Encoder/Video
Description Encode raw data to compressed video
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.9.2
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----vpuenc_h264
Implemented Interfaces:
GstPreset
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16, (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
width: [ 64, 1920 ]
height: [ 64, 1088 ]
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
stream-format: { (string)avc, (string)byte-stream }
alignment: { (string)au, (string)nal }
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
bitrate : set bit rate in kbps (0 for automatic)
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 0
force-idr : force incoming frame to be encoded as IDR frame
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 0
gop-size : How many frames a group-of-picture shall contain
flags: readable, writable
Unsigned Integer. Range: 0 - 32767 Default: 30
level : VC8000E/H.264 level. 51 = Level 5.1 [51], default -1 makes wrapper to calculate h264 level
flags: readable, writable
Integer. Range: 10 - 99 Default: -1
min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
name : The name of the object
flags: readable, writable
String. Default: "vpuenc_h264-0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
profile : VC8000E/H.264 supports Baseline, Main, High, High 10 profiles, default -1 makes wrapper to set 9:Baseline
flags: readable, writable
Integer. Range: 9 - 12 Default: -1
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
qp-max : maximum QP for any picture, default 0 makes wrapper to set 51
flags: readable, writable
Integer. Range: 0 - 51 Default: 0
qp-min : minimum QP for any picture
flags: readable, writable
Integer. Range: 0 - 51 Default: 0
quant : set quant value: H.264(0-51) (-1 for automatic)
flags: readable, writable
Integer. Range: -1 - 51 Default: -1
stream-multislice : the number of slices a picture contains
flags: readable, writable
Integer. Range: 1 - 2147483647 Default: 1
The i.MX8M-PLUS VPU H.265 video encoder element¶
# gst-inspect-1.0 vpuenc_hevc
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based HEVC video encoder
Klass Codec/Encoder/Video
Description Encode raw data to compressed video
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.9.2
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----vpuenc_hevc
Implemented Interfaces:
GstPreset
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16, (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
width: [ 64, 1920 ]
height: [ 64, 1088 ]
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h265
variant: itu
width: [ 64, 1920 ]
height: [ 64, 1088 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
bitrate : set bit rate in kbps (0 for automatic)
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 0
force-idr : force incoming frame to be encoded as IDR frame
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 0
gop-size : How many frames a group-of-picture shall contain
flags: readable, writable
Unsigned Integer. Range: 0 - 32767 Default: 30
level : VC8000E/HEVC level. 180 = level 6.0*30 [180], default -1 makes wrapper to set 153:LEVEL_5_1
flags: readable, writable
Integer. Range: 30 - 180 Default: -1
min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
name : The name of the object
flags: readable, writable
String. Default: "vpuenc_hevc0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
profile : VC8000E/HEVC supports Main, Main Still Picture, Main 10, default -1 makes wrapper to set 0:Main
flags: readable, writable
Integer. Range: 0 - 2 Default: -1
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
qp-max : maximum QP for any picture, default 0 makes wrapper to set 51
flags: readable, writable
Integer. Range: 0 - 51 Default: 0
qp-min : minimum QP for any picture
flags: readable, writable
Integer. Range: 0 - 51 Default: 0
quant : set quant value: HEVC(0-51) (-1 for automatic)
flags: readable, writable
Integer. Range: -1 - 51 Default: -1
stream-multislice : the number of slices a picture contains
flags: readable, writable
Integer. Range: 1 - 2147483647 Default: 1
Test Pattern Stream¶
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Camera Loopback¶
Stream video from a camera to a display:
Video Recording¶
The i.MX8M-PLUS can encode videos to AVC(H.264) & HEVC(H.265) at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the src pad capabilities in the [vpuenc_264]#the-imx8m-plus-vpu-h264-video-encoder-element) and vpuenc_265 elements details for the complete list of features supported by the i.MX8M-PLUS VPU encoder.
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi
Example of encoding and saving a short video stream from a camera to an H.265-encoded TS file:
# gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_hevc ! mpegtsmux ! filesink location=./test_h265.ts
Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi
Video Playback¶
The i.MX8M-PLUS can decode and playback AVC(H.264), HEVC(H.265), VP9, VP8, etc. encoded videos with at resolutions of up to 1080p60 (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M-PLUS VPU decoder.
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux name=demux demux.video_0 ! vpudec ! autovideosink
RTP Video streaming¶
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
# gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=1280,height=720 ! \
jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223
References¶
- The GStreamer website
- NXP BSP Linux Users Guide, Multimedia section download link