Streaming Video¶
Introduction to GStreamer¶
GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Major GStreamer commands¶
There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect¶
Lists all available elements:
Lists all available sink elements:
For example:
# gst-inspect-1.0 | grep sink
alsa: alsasink: Audio sink (ALSA)
waylandsink: waylandsink: wayland video sink
shm: shmsink: Shared Memory Sink
decklink: decklinkaudiosink: Decklink Audio Sink
decklink: decklinkvideosink: Decklink Video Sink
ximagesink: ximagesink: Video sink
soup: souphttpclientsink: HTTP client sink
autodetect: autovideosink: Auto video sink
autodetect: autoaudiosink: Auto audio sink
gdkpixbuf: gdkpixbufsink: GdkPixbuf sink
playback: playsink: Player Sink
video4linux2: v4l2sink: Video (video4linux2) Sink
fbdevsink: fbdevsink: fbdev video sink
tcp: tcpclientsink: TCP client sink
tcp: tcpserversink: TCP server sink
…
Detailed help information on a specific element:
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank primary + 2 (258)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <[email protected]>, George Kiagiadakis <[email protected]>
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.14.0
License LGPL
Source module gst-plugins-bad
Source release date 2018-03-19
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
GstWaylandVideo
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
(string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
(string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
(string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
(string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
(string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
(string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 20000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
window-width : Wayland sink preferred window width in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
window-height : Wayland sink preferred window height in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
display : Wayland display name to connect to, if not supplied via the GstContext
flags: readable, writable
String. Default: null
alpha : Wayland surface alpha value, apply custom alpha value to wayland surface
flags: readable, writable
Float. Range: 0 - 1 Default: 0
enable-tile : When enabled, the sink propose VSI tile modifier to VPU
flags: readable, writable
Boolean. Default: false
The i.MX8M-PLUS VPU video decoder element¶
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based video decoder
Klass Codec/Decoder/Video
Description Decode compressed video to raw data
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.5.6
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoDecoder
+----GstVpuDec
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h265
video/x-vp9
video/x-vp8
video/x-h264
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
disable-reorder : disable vpu reorder when end to end streaming
flags: readable, writable
Boolean. Default: false
frame-drop : enable adaptive frame drop for smoothly playback
flags: readable, writable
Boolean. Default: true
frame-plus : set number of addtional frames for smoothly playback
flags: readable, writable
Unsigned Integer. Range: 0 - 16 Default: 3
name : The name of the object
flags: readable, writable
String. Default: "vpudec0"
output-format : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
flags: readable, writable
Enum "GstVpuDecOutputFormat" Default: 0, "auto"
(0): auto - enable chroma interleave. (default)
(1): NV12 - NV12 format
(2): I420 - I420 format
(3): YV12 - YV12 format
(4): Y42B - Y42B format
(5): NV16 - NV16 format
(6): Y444 - Y444 format
(7): NV24 - NV24 format
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
use-vpu-memory : use vpu allocate video frame buffer
flags: readable, writable
Boolean. Default: true
The i.MX8M-PLUS VPU H.264 video encoder element¶
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based AVC/H264 video encoder
Klass Codec/Encoder/Video
Description Encode raw data to compressed video
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.5.6
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----vpuenc_h264
Implemented Interfaces:
GstPreset
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
stream-format: { (string)avc, (string)byte-stream }
alignment: { (string)au, (string)nal }
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16, (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
width: [ 64, 1920 ]
height: [ 64, 1088 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
bitrate : set bit rate in kbps (0 for automatic)
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 0
gop-size : How many frames a group-of-picture shall contain
flags: readable, writable
Unsigned Integer. Range: 0 - 32767 Default: 30
name : The name of the object
flags: readable, writable
String. Default: "vpuenc_h264-0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
quant : set quant value: H.264(0-51) (-1 for automatic)
flags: readable, writable
Integer. Range: -1 - 51 Default: -1
stream-multislice : the number of slices a picture contains
flags: readable, writable
Integer. Range: 1 - 2147483647 Default: 1
The i.MX8M-PLUS VPU H.265 video encoder element¶
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based HEVC video encoder
Klass Codec/Encoder/Video
Description Encode raw data to compressed video
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.5.6
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----vpuenc_hevc
Implemented Interfaces:
GstPreset
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h265
variant: itu
width: [ 64, 1920 ]
height: [ 64, 1088 ]
framerate: [ 0/1, 2147483647/1 ]
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YUY2, (string)UYVY, (string)RGBA, (string)RGBx, (string)RGB16, (string)RGB15, (string)BGRA, (string)BGRx, (string)BGR16 }
width: [ 64, 1920 ]
height: [ 64, 1088 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
bitrate : set bit rate in kbps (0 for automatic)
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 0
gop-size : How many frames a group-of-picture shall contain
flags: readable, writable
Unsigned Integer. Range: 0 - 32767 Default: 30
name : The name of the object
flags: readable, writable
String. Default: "vpuenc_hevc0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
quant : set quant value: HEVC(0-51) (-1 for automatic)
flags: readable, writable
Integer. Range: -1 - 51 Default: -1
stream-multislice : the number of slices a picture contains
flags: readable, writable
Integer. Range: 1 - 2147483647 Default: 1
Test Pattern Stream¶
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:
Test pattern with specific parameters on the input element:
Camera Loopback¶
Stream video from a camera to a display:
Video Recording¶
The i.MX8M-PLUS can encode videos to AVC(H.264) & HEVC(H.265) at resolutions of up to 1080p60 Full HD (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the src pad capabilities in the [vpuenc_264]#the-imx8m-plus-vpu-h264-video-encoder-element) and vpuenc_265 elements details for the complete list of features supported by the i.MX8M-PLUS VPU encoder.
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi
Example of encoding and saving a short video stream from a camera to an H.265-encoded TS file:
# gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_hevc ! mpegtsmux ! filesink location=./test_h265.ts
Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi
Video Playback¶
The i.MX8M-PLUS can decode and playback AVC(H.264), HEVC(H.265), VP9, VP8, etc. encoded videos with at resolutions of up to 1080p60 (1920×1080p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M-PLUS VPU decoder.
Example of a video file playback using the playbin element (making use of the VPU decoder):
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux name=demux demux.video_0 ! vpudec ! autovideosink
RTP Video streaming¶
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! \
jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223
References¶
- The GStreamer website
- NXP BSP Linux Users Guide, Multimedia section download link