Tag: GStreamer

HTML5 + JavaScript + CSS3 RGBA video overlays on top of live GStreamer video pipelines

GStreamer comes with a number of plugins that allow rendering of text and/or graphics overlays on top of video: rsvgoverlay, subtitleoverlay, textoverlay, cairooverlay, gdkpixbufoverlay, opencvtextoverlay, etc. However, some of these plugins often allow only static graphics and text, and often do not approach the flexibility and power of dedicated video post-processing software products.

“noweffects” (a play on the name of a popular video post-processing software) is a proof-of-concept of leveraging the power of a modern HTML5 + JavaScript + CSS3 web browser engine to render high-quality, programmable, alpha-aware, animated, vector- and bitmap based content, which is then rendered into an RGBA raw video stream, which can then be transferred via some kind of IPC method to separate GStreamer processes, where it can be composited with other content via GStreamers regular compositor or videomixer plugins.

Qt was chosen for its ease of integration of modern WebKit (QtWebKit) and GStreamer (qt-gstreamer), and its ability to render widgets to RGBA images. The QMainWindow widget is rendered in regular intervals to QImages in RGBA format, then inserted into a GStreamer pipeline via the appsrc plugin. This pipeline simply uses udpsink to multicast the raw video RTP packets on localhost to allow for multiple ‘subscribers’. A second GStreamer pipleline can then use udpsrc and apply the overlay.

Proof-of-concept code is available at: https://github.com/michaelfranzl/noweffects

The following demonstration video was generated with “noweffects”: A website (showing CSS3 animations), rendered to an RGBA video via QtWebKit, then overlaid on top of a video test pattern in a separate GStreamer process.

GStreamer pipeline recipe: Stream file via RTP using rtpbin

Given an audio/video file encoded with

ffmpeg -i in.webm -vcodec vp9 -acodec opus -b:v 200k -b:a 80k out.mkv

then the following GStreamer pipeline (I’m using version 1.13.1) will stream it via RTP using rtpbin to localhost ports 50000-50003:

gst-launch-1.0 -v \
rtpbin name=rtpbin \
filesrc location=out.mkv ! matroskademux name=demux \
demux.audio_0 ! rtpopuspay ! rtpbin.send_rtp_sink_0 \
demux.video_0 ! rtpvp9pay ! rtpbin.send_rtp_sink_1 \
rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=50000 sync=true async=false \
rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=50001 sync=false async=false \
rtpbin.send_rtp_src_1 ! udpsink host=127.0.0.1 port=50002 sync=true async=false \
rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=50003 sync=false async=false

The receiver outputting the media to screen and speakers:

/home/michael/progs/gstreamer/bin/gst-launch-1.0 -v \
rtpbin name=rtpbin \
udpsrc address=127.0.0.1 port=50000 caps="application/x-rtp, media=audio, encoding-name=OPUS, clock-rate=48000" ! rtpbin.recv_rtp_sink_0 \
udpsrc address=127.0.0.1 port=50001 caps="application/x-rtcp" ! rtpbin.recv_rtcp_sink_0 \
udpsrc address=127.0.0.1 port=50002 caps="application/x-rtp, media=video, encoding-name=VP9, clock-rate=90000" ! rtpbin.recv_rtp_sink_1 \
udpsrc address=127.0.0.1 port=50003 caps="application/x-rtcp" ! rtpbin.recv_rtcp_sink_1 \
rtpbin. ! rtpopusdepay ! queue ! opusdec ! autoaudiosink sync=true \
rtpbin. ! rtpvp9depay ! queue ! avdec_vp9 ! autovideosink sync=true

Notes/Gotchas

  • The sender uses almost no CPU because the media is not transcoded.
  • Instead of vp9dec you can use avdec_vp9.
  • The sync attributes must be specified exactly as given.
  • When the sender is restarted while the client is running, the client terminates with the error streaming stopped, reason not-linked.
  • media, encoding-name and clock-rate attributes are required.
  • The encoding names specified in the receiver must match those of the sender.
  • It is not necessary to specify RTP payload numbers.
  • If on the receiver sync=false, audio and video are not in sync.
  • Above example only supports one receiver. To support multiple receivers, you can multicast the UDP packets to the loopback network device with the following modifications:
    • udpsink options: host=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 bind-address=127.0.0.1
    • udpsrc options: address=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 bind-address=127.0.0.1
    • Note: 225.0.0.37 is just an example multicast address. ttl-mc=0 is important, otherwise the packets will be forwarded across network boundaries. You should be careful with multicasting, and educate yourself before you try it.

Receiver without rtpbin

To receive without using rtpbin:

gst-launch-1.0 -v \
udpsrc address=127.0.0.1 port=50000 caps="application/x-rtp" ! queue ! rtpopusdepay ! queue ! opusdec ! autoaudiosink sync=true \
udpsrc address=127.0.0.1 port=50002 caps="application/x-rtp" ! queue ! rtpvp9depay ! queue ! vp9dec ! autovideosink sync=true

Here, the sender can be restarted without bringing the receiver down.