Streaming Video¶
Introduction to GStreamer¶
GStreamer is a powerful pipeline-based multimedia framework.
It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing.
This is not a GStreamer reference manual but an overview for using it on the Variscite i.MX6 products.
This release uses the gstreamer-imx set of GStreamer 1.0 plugins for NXP's i.MX platform, which make use of the i.MX multimedia capabilities.
Major GStreamer commands¶
There are two major commands:
gst-inspect - allows you to to get documentation on available elements and detailed information on a specific element.
gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect¶
Lists all available elements:
Lists all available sink elements:
For example:
# gst-inspect-1.0 | grep sink
alsa: alsasink: Audio sink (ALSA)
waylandsink: waylandsink: wayland video sink
shm: shmsink: Shared Memory Sink
decklink: decklinkaudiosink: Decklink Audio Sink
decklink: decklinkvideosink: Decklink Video Sink
ximagesink: ximagesink: Video sink
soup: souphttpclientsink: HTTP client sink
autodetect: autovideosink: Auto video sink
autodetect: autoaudiosink: Auto audio sink
gdkpixbuf: gdkpixbufsink: GdkPixbuf sink
playback: playsink: Player Sink
video4linux2: v4l2sink: Video (video4linux2) Sink
fbdevsink: fbdevsink: fbdev video sink
tcp: tcpclientsink: TCP client sink
tcp: tcpserversink: TCP server sink
…
Detailed help information on a specific element:
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank marginal (64)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <[email protected]>, George Kiagiadakis <[email protected]>
Documentation https://gstreamer.freedesktop.org/documentation/waylandsink/#waylandsink-page
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.24.7
License LGPL
Source module gst-plugins-bad
Documentation https://gstreamer.freedesktop.org/documentation/waylandsink/
Source release date 2024-08-21
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)AYUV, (string)RGBA, (string)ARGB, (string)BGRA, (string)ABGR, (string)P010_10LE, (string)v308, (string)RGBx, (string)xRGB, (string)BGRx, (string)xBGR,Y}
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: DMA_DRM
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
alpha : Wayland surface alpha value, apply custom alpha value to wayland surface
flags: readable, writable
Float. Range: 0 - 1 Default: 0
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
display : Wayland display name to connect to, if not supplied via the GstContext
flags: readable, writable
String. Default: null
drm-device : Path of the DRM device to use for dumb buffer allocation
flags: readable, writable, can be set only at object construction time
String. Default: null
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
enable-tile : When enabled, the sink propose VSI tile modifier to VPU
flags: readable, writable
Boolean. Default: false
fullscreen : Whether the surface should be made fullscreen
flags: readable, writable
Boolean. Default: false
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 5000000
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
processing-deadline : Maximum processing time for a buffer in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 15000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
render-rectangle : The render rectangle ('<x, y, width, height>')
flags: writable
Default: "< >"
GstValueArray of GValues of type "gint" Write only
rotate-method : rotate method
flags: readable, writable
Enum "GstVideoOrientationMethod" Default: 0, "identity"
(0): identity - GST_VIDEO_ORIENTATION_IDENTITY
(1): 90r - GST_VIDEO_ORIENTATION_90R
(2): 180 - GST_VIDEO_ORIENTATION_180
(3): 90l - GST_VIDEO_ORIENTATION_90L
(4): horiz - GST_VIDEO_ORIENTATION_HORIZ
(5): vert - GST_VIDEO_ORIENTATION_VERT
(6): ul-lr - GST_VIDEO_ORIENTATION_UL_LR
(7): ur-ll - GST_VIDEO_ORIENTATION_UR_LL
(8): auto - GST_VIDEO_ORIENTATION_AUTO
(9): custom - GST_VIDEO_ORIENTATION_CUSTOM
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
stats : Sink Statistics
flags: readable
Boxed pointer of type "GstStructure"
average-rate: 0
dropped: 0
rendered: 0
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
window-height : Wayland sink preferred window height in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
window-width : Wayland sink preferred window width in pixel
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
The i.MX6 VPU video decoder element¶
# gst-inspect-1.0 vpudec
[INFO] Product Info: i.MX6Q/D/S
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based video decoder
Klass Codec/Decoder/Video
Description Decode compressed video to raw data
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.9.2
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoDecoder
+----GstVpuDec
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-vp8
video/x-h264
video/mpeg
systemstream: false
mpegversion: { (int)1, (int)2 }
video/mpeg
mpegversion: 4
video/x-h263
video/x-xvid
video/x-wmv
wmvversion: 3
format: WMV3
video/x-wmv
wmvversion: 3
format: WVC1
image/jpeg
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YV12, (string)Y42B, (string)NV16, (string)Y444, (string)NV24, (string)NV12_10LE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
automatic-request-sync-point-flags: Flags to use when automatically requesting sync points
flags: readable, writable
Flags "GstVideoDecoderRequestSyncPointFlags" Default: 0x00000003, "corrupt-output+discard-input"
(0x00000001): discard-input - GST_VIDEO_DECODER_REQUEST_SYNC_POINT_DISCARD_INPUT
(0x00000002): corrupt-output - GST_VIDEO_DECODER_REQUEST_SYNC_POINT_CORRUPT_OUTPUT
automatic-request-sync-points: Automatically request sync points when it would be useful
flags: readable, writable
Boolean. Default: false
disable-reorder : disable vpu reorder when end to end streaming
flags: readable, writable
Boolean. Default: false
discard-corrupted-frames: Discard frames marked as corrupted instead of outputting them
flags: readable, writable
Boolean. Default: false
frame-drop : enable adaptive frame drop for smoothly playback
flags: readable, writable
Boolean. Default: true
frame-plus : set number of addtional frames for smoothly playback
flags: readable, writable
Unsigned Integer. Range: 0 - 16 Default: 3
max-errors : Max consecutive decoder errors before returning flow error
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
name : The name of the object
flags: readable, writable
String. Default: "vpudec0"
output-format : set raw video format for output (Y42B NV16 Y444 NV24 only for MJPEG)
flags: readable, writable
Enum "GstVpuDecOutputFormat" Default: 0, "auto"
(0): auto - enable chroma interleave. (default)
(1): NV12 - NV12 format
(2): I420 - I420 format
(3): YV12 - YV12 format
(4): Y42B - Y42B format
(5): NV16 - NV16 format
(6): Y444 - Y444 format
(7): NV24 - NV24 format
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: true
use-vpu-memory : use vpu allocate video frame buffer
flags: readable, writable
Boolean. Default: true
The i.MX6 VPU H.264 video encoder element¶
# gst-inspect-1.0 vpuenc_h264
Factory Details:
Rank primary + 1 (257)
Long-name IMX VPU-based AVC/H264 video encoder
Klass Codec/Encoder/Video
Description Encode raw data to compressed video
Author Multimedia Team <[email protected]>
Plugin Details:
Name vpu
Description VPU video codec
Filename /usr/lib/gstreamer-1.0/libgstvpu.so
Version 4.9.2
License LGPL
Source module imx-gst1.0-plugin
Binary package Freescle Gstreamer Multimedia Plugins
Origin URL http://www.freescale.com
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----vpuenc_h264
Implemented Interfaces:
GstPreset
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)NV12, (string)I420, (string)YV12 }
width: [ 64, 1920, 8 ]
height: [ 64, 1088, 8 ]
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
stream-format: { (string)avc, (string)byte-stream }
alignment: { (string)au, (string)nal }
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
bitrate : set bit rate in kbps (0 for automatic)
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 0
gop-size : How many frames a group-of-picture shall contain
flags: readable, writable
Unsigned Integer. Range: 0 - 32767 Default: 15
min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
name : The name of the object
flags: readable, writable
String. Default: "vpuenc_h264-0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
quant : set quant value: H.264(0-51) (-1 for automatic)
flags: readable, writable
Integer. Range: -1 - 51 Default: -1
Test Pattern Stream¶
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer.
Default test pattern:
Test pattern with specific parameters on the input element:
Camera Loopback¶
Stream video from a camera to a display:
Video Recording¶
The i.MX6 family can encode AVC (H.264) video streams at resolutions of up to 1080p using its internal hardware Video Processing Unit (VPU). Supported frame rates and features depend on the specific i.MX6 variant and system configuration.
Example of encoding and saving a short video stream from a camera to an H.264-encoded AVI file:
# gst-launch-1.0 imxv4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! vpuenc_h264 ! avimux ! filesink location=./test_h264.avi
Example of encoding and saving a short video stream from a camera to an MJPEG-encoded AVI file (using the SW based jpegenc element - without using the VPU):
# gst-launch-1.0 imxv4l2src device=/dev/video0 num-buffers=100 ! video/x-raw,width=1920,height=1080 ! jpegenc ! avimux ! filesink location=./test_jpeg.avi
Video Playback¶
The i.MX6 family can decode and playback AVC (H.264), MPEG-2, MPEG-4, H.263, and MJPEG video streams at resolutions of up to 1080p using its internal hardware Video Processing Unit (VPU). Supported frame rates and codecs depend on the specific i.MX6 variant and system configuration.
Example of a video file playback using the playbin element (making use of the VPU decoder):
Example of a video file playback while explicitly specifying a pipeline (making use of the VPU decoder):
# gst-launch-1.0 filesrc location=/run/media/sda1/big_buck_bunny.mp4 ! qtdemux name=demux demux.video_0 ! vpudec ! autovideosink
RTP Video streaming¶
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
# gst-launch-1.0 imxv4l2src device=/dev/video0 ! video/x-raw,width=1280,height=720 ! \
jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223
References¶
- The GStreamer website
- NXP BSP Linux Users Guide, Multimedia section download link