Streaming Video¶
Introduction to GStreamer¶
GStreamer is a powerful pipeline-based multimedia framework. It allows programmers to create a variety of media-handling components, including simple audio playback, audio and video playback, recording, streaming and editing. This is not a GStreamer reference manual but an overview for using it on products based on the Variscite AM62P platform. This release uses some TI-optimized GStreamer plugins as an addition to the standard set of GStreamer 1.0 plugins.
The AM62P also has a H.264 / H.265 video decoder and encoder, allowing to playback or transcode videos using HW acceleration. See the examples below for details.
Major GStreamer commands¶
There are two major commands: gst-inspect - allows you to get documentation on available elements and detailed information on a specific element. gst-launch - allows you to create and execute GStreamer pipelines.
gst-inspect¶
Lists all available elements:
Lists all available sink elements:
For example:
# gst-inspect-1.0 | grep sink
coreelements: fakesink: Fake Sink
coreelements: fdsink: Filedescriptor Sink
coreelements: filesink: File Sink
fbdevsink: fbdevsink: fbdev video sink
tcp: tcpclientsink: TCP client sink
tcp: tcpserversink: TCP server sink
tcp: multifdsink: Multi filedescriptor sink
tcp: multisocketsink: Multi socket sink
multifile: multifilesink: Multi-File Sink
multifile: splitmuxsink: Split Muxing Bin
bluez: a2dpsink: Bluetooth A2DP sink
bluez: avdtpsink: Bluetooth AVDTP sink
video4linux2: v4l2sink: Video (video4linux2) Sink
...
Detailed help information on a specific element:
For example, the output for waylandsink:
# gst-inspect-1.0 waylandsink
Factory Details:
Rank marginal (64)
Long-name wayland video sink
Klass Sink/Video
Description Output to wayland surface
Author Sreerenj Balachandran <[email protected]>, George Kiagiadakis <[email protected]>
Plugin Details:
Name waylandsink
Description Wayland Video Sink
Filename /usr/lib/gstreamer-1.0/libgstwaylandsink.so
Version 1.16.3
License LGPL
Source module gst-plugins-bad
Binary package GStreamer Bad Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstVideoSink
+----GstWaylandSink
Implemented Interfaces:
GstVideoOverlay
GstWaylandVideo
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR, (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU, (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9, (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: { (string)BGRx, (string)BGRA, (string)RGBx, (string)xBGR, (string)xRGB, (string)RGBA, (string)ABGR, (string)ARGB, (string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)YUY2, (string)YVYU, (string)UYVY, (string)AYUV, (string)NV12, (string)NV21, (string)NV16, (string)YUV9, (string)YVU9, (string)Y41B, (string)I420, (string)YV12, (string)Y42B, (string)v308 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
display : Wayland display name to connect to, if not supplied via the GstContext
flags: readable, writable
String. Default: null
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
fullscreen : Whether the surface should be made fullscreen
flags: readable, writable
Boolean. Default: false
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: 5000000
name : The name of the object
flags: readable, writable
String. Default: "waylandsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
processing-deadline : Maximum processing deadline in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 15000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
show-preroll-frame : Whether to render video frames during preroll
flags: readable, writable
Boolean. Default: true
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
Test Pattern Stream¶
Test pattern allows you to display pre-built image/video patterns on the display. It is very useful when you don't have a camera but still want to test GStreamer. Default test pattern:
Test pattern with specific parameters on the input element:
Camera Loopback¶
To stream video from a camera to a display, also see Testing Camera Preview on Display
Video Playback¶
gst-launch-1.0 filesrc location=bigbuckbunny.mp4 ! \
qtdemux name=demux demux.video_0 ! queue ! h264parse ! v4l2h264dec capture-io-mode=dmabuf ! waylandsink \
demux.audio_0 ! queue ! decodebin ! audioconvert ! audioresample ! audio/x-raw,channels=2,rate=48000 ! autoaudiosink
The pipeline above makes use of the v4l2h264dec plugin which will use the VPU to decode the video.
This GStreamer pipeline performs the following tasks:
- filesrc element reads the file big_buck_bunny.mp4.
- qtdemux element demultiplexes the file into audio and video streams. The name demux is assigned to the qtdemux element for later reference.
- The video stream is selected and passed to a queue element that helps to buffer and smooth the data flow.
- The h264parse element parses the H.264 encoded video stream.
- The v4l2h264dec element decodes the H.264 encoded video stream using the VPU
- The waylandsink element is used to output the video to a Wayland display server.
- The audio stream is selected and passed to a queue element that helps to buffer and smooth the data flow.
- The decodebin plugin is used to determine the audio format. It is capable of detecting an selecting the proper plugin depending on the incoming audio data. An alternative can be to explicitly select the audio stream decoder.
- The audioconvert element is used to convert the audio data from one format to another if required.
- The audioresample element is used to change the audio sample rate if required.
- The alsasink element is used to output the audio to the default ALSA sound card.
Info
The SD Recovery SD card does not contain the plugin to decode AAC audio data. When trying to playback a video that uses AAC audio, an error is shown similar to
[..]
Missing element: MPEG-4 AAC decoder
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0: Your GStreamer installation is missing a plug-in.
[..]
Either convert the video to use a different audio format or build your own
custom image with the faad2 package enabled which is commercially licensed.
Video Output Sinks¶
In the above example, the waylandsink was used, which shows the video in an
active Wayland session, e.g on the Weston Desktop.
If Wayland is not running, replace the wayland sink with
kmssink driver-name=tidss to directly use the TI DSS Graphics Driver. This
will automatically resize the video to match the resolution of your display.
A third option is to let GStreamer chose the best option, simply replace
waylandsink with autovideosink.
RTP Video streaming¶
RTP JPEG Video Streaming¶
Encode video stream from camera using a software based JPEG encoder and stream it over RTP:
cam -c1 --stream width=1280,height=720,pixelformat=UYVY -C1
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=1280,height=720 ! \
jpegenc ! rtpjpegpay ! udpsink host=192.168.20.60 port=1223
RTP H.264 Video Streaming to Display¶
Encode video stream via HW acceleration and stream it over the network via UDP.
The receiver needs to start the GStreamer pipeline first.
gst-launch-1.0 udpsrc port=5000 \
caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! \
rtpjitterbuffer latency=50 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=dmabuf ! queue ! \
fpsdisplaysink text-overlay=false name=fpssink video-sink="kmssink driver-name=tidss sync=true show-preroll-frame=false" \
sync=true
Once the receiver GStreamer pipeline is started, the sender pipeline can also be started. The first call sets the video parameters.
cam -c1 --stream width=1280,height=720,pixelformat=UYVY -C1
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=1280,height=720,format=UYVY ! \
v4l2h264enc output-io-mode=dmabuf-import extra-controls="controls,h264_i_frame_period=60" ! \
rtph264pay ! udpsink port=5000 host=<receiver IP address>
As UDP is used, only the receiver IP address is needed, not the sender IP address.
RTP H.264 Video Streaming to File¶
Instead of showing the video stream on the display, it can also be written to the file system.
Once again, the receiver stream needs to be started prior to the sender pipeline.
gst-launch-1.0 -e udpsrc port=5000 \
caps="application/x-rtp,media=video,clock-rate=90000,encoding-name=H264,payload=96" ! \
rtpjitterbuffer latency=50 ! rtph264depay ! h264parse ! mp4mux ! \
filesink location=received.mp4
Afterwards, start the sender pipeline, using the identical call as before.
RTP H.265 Video Streaming to Display¶
The AM62P can also decode and encode H.265, allowing for smaller bitrates at identical quality compared to H.264.
First start the receiver:
gst-launch-1.0 udpsrc port=5000 \
caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265, payload=(int)96" ! \
rtpjitterbuffer latency=50 ! rtph265depay ! h265parse ! v4l2h265dec capture-io-mode=dmabuf ! queue ! \
fpsdisplaysink text-overlay=false name=fpssink video-sink="kmssink driver-name=tidss sync=true show-preroll-frame=false" \
sync=true
Then start the sender:
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=1280,height=720,format=UYVY ! \
v4l2h265enc output-io-mode=dmabuf-import extra-controls="controls,h265_i_frame_period=60" ! \
rtph265pay ! udpsink port=5000 host=<receiver IP address>