Streaming Video¶
Introduction to GStreamer¶
GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX8 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Major GStreamer commands¶
There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect¶
Lists all available elements:
Lists all available sink elements:
For example:
# gst-inspect-1.0 | grep sink
alsa: alsasink: Audio sink (ALSA)
waylandsink: waylandsink: wayland video sink
shm: shmsink: Shared Memory Sink
decklink: decklinkaudiosink: Decklink Audio Sink
decklink: decklinkvideosink: Decklink Video Sink
ximagesink: ximagesink: Video sink
soup: souphttpclientsink: HTTP client sink
autodetect: autovideosink: Auto video sink
autodetect: autoaudiosink: Auto audio sink
gdkpixbuf: gdkpixbufsink: GdkPixbuf sink
playback: playsink: Player Sink
video4linux2: v4l2sink: Video (video4linux2) Sink
fbdevsink: fbdevsink: fbdev video sink
tcp: tcpclientsink: TCP client sink
tcp: tcpserversink: TCP server sink
…
Detailed help information on a specific element:
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank primary + 2 (258)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <[email protected]>, George Kiagiadakis <[email protected]>
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.14.0
License LGPL
Source module gst-plugins-bad
Source release date 2018-03-19
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
GstWaylandVideo
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
(string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
(string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
(string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR,
(string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU,
(string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9,
(string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 20000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
window-width : Wayland sink preferred window width in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
window-height : Wayland sink preferred window height in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
display : Wayland display name to connect to, if not supplied via the GstContext
flags: readable, writable
String. Default: null
alpha : Wayland surface alpha value, apply custom alpha value to wayland surface
flags: readable, writable
Float. Range: 0 - 1 Default: 0
enable-tile : When enabled, the sink propose VSI tile modifier to VPU
flags: readable, writable
Boolean. Default: false
The i.MX8M VPU video decoder element¶
# gst-inspect-1.0 vpudec
Factory Details:
Rank primary + 1 (257)
Long-name VPU-based video decoder
Klass Codec/Decoder/Video
Description Decode compressed video to raw data
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu.imx
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.3.4
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoDecoder
+----GstVpuDec
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h265
video/x-vp9
video/x-vp8
video/x-vp6-flash
video/x-h264
video/mpeg
systemstream: false
mpegversion: { (int)1, (int)2 }
video/mpeg
mpegversion: 4
video/x-h263
video/x-flash-video
flvversion: 1
video/x-divx
divxversion: 3
video/x-divx
divxversion: 4
video/x-divx
divxversion: { (int)5, (int)6 }
video/x-xvid
video/x-cavs
video/x-wmv
wmvversion: 3
format: WMV3
video/x-wmv
wmvversion: 3
format: WVC1
video/x-pn-realvideo
image/jpeg
image/webp
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element Flags:
no flags set
Element Implementation:
Has change_state() function: gst_video_decoder_change_state
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "vpudec0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
output-format : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
flags: readable, writable
Enum "GstVpuDecOutputFormat" Default: 0, "auto"
(0): auto - enable chroma interleave. (default)
(1): NV12 - NV12 format
(2): I420 - I420 format
(3): YV12 - YV12 format
(4): Y42B - Y42B format
(5): NV16 - NV16 format
(6): Y444 - Y444 format
(7): NV24 - NV24 format
frame-drop : enable adaptive frame drop for smoothly playback
flags: readable, writable
Boolean. Default: true
frame-plus : set number of addtional frames for smoothly playback
flags: readable, writable
Unsigned Integer. Range: 0 - 16 Default: 3
use-vpu-memory : use vpu allocate video frame buffer
flags: readable, writable
Boolean. Default: true
Test Pattern Stream¶
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:
Test pattern with specific parameters on the input element:
Camera Loopback¶
Stream video from a camera to a display:
Video Recording¶
Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file:
# gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi
Video Playback¶
The i.MX8M can decode and playback HEVC(H.265), VP9, etc. encoded videos with HDR at resolutions of up to 4Kp60 UHD (3840×2160p @ 60fps) using its internal hardware Video Processing Unit (VPU).
See the sink pad capabilities in the vpudec element details for the complete list of codecs supported by the i.MX8M VPU decoder.
Example of a video file playback using the playbin element (making use of the VPU decoder):
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux name=demux demux.video_0 ! vpudec ! autovideosink
RTP Video streaming¶
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
# gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! \
jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223
References¶
- The GStreamer website
- NXP BSP Linux Users Guide, Multimedia section download link