GStreamer pipeline recipe: Stream file via RTP using rtpbin

Given an audio/video file encoded with

ffmpeg -i in.webm -vcodec vp9 -acodec opus -b:v 200k -b:a 80k out.mkv

then the following GStreamer pipeline (I’m using version 1.13.1) will stream it via RTP using rtpbin to localhost ports 50000-50003:

gst-launch-1.0 -v \
rtpbin name=rtpbin \
filesrc location=out.mkv ! matroskademux name=demux \
demux.audio_0 ! rtpopuspay ! rtpbin.send_rtp_sink_0 \
demux.video_0 ! rtpvp9pay ! rtpbin.send_rtp_sink_1 \
rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=50000 sync=true async=false \
rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=50001 sync=false async=false \
rtpbin.send_rtp_src_1 ! udpsink host=127.0.0.1 port=50002 sync=true async=false \
rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=50003 sync=false async=false

The receiver outputting the media to screen and speakers:

/home/michael/progs/gstreamer/bin/gst-launch-1.0 -v \
rtpbin name=rtpbin \
udpsrc address=127.0.0.1 port=50000 caps="application/x-rtp, media=audio, encoding-name=OPUS, clock-rate=48000" ! rtpbin.recv_rtp_sink_0 \
udpsrc address=127.0.0.1 port=50001 caps="application/x-rtcp" ! rtpbin.recv_rtcp_sink_0 \
udpsrc address=127.0.0.1 port=50002 caps="application/x-rtp, media=video, encoding-name=VP9, clock-rate=90000" ! rtpbin.recv_rtp_sink_1 \
udpsrc address=127.0.0.1 port=50003 caps="application/x-rtcp" ! rtpbin.recv_rtcp_sink_1 \
rtpbin. ! rtpopusdepay ! queue ! opusdec ! autoaudiosink sync=true \
rtpbin. ! rtpvp9depay ! queue ! avdec_vp9 ! autovideosink sync=true

Notes/Gotchas

  • The sender uses almost no CPU because the media is not transcoded.
  • Instead of vp9dec you can use avdec_vp9.
  • The sync attributes must be specified exactly as given.
  • When the sender is restarted while the client is running, the client terminates with the error streaming stopped, reason not-linked.
  • media, encoding-name and clock-rate attributes are required.
  • The encoding names specified in the receiver must match those of the sender.
  • It is not necessary to specify RTP payload numbers.
  • If on the receiver sync=false, audio and video are not in sync.
  • Above example only supports one receiver. To support multiple receivers, you can multicast the UDP packets to the loopback network device with the following modifications:
    • udpsink options: host=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 bind-address=127.0.0.1
    • udpsrc options: address=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 bind-address=127.0.0.1
    • Note: 225.0.0.37 is just an example multicast address. ttl-mc=0 is important, otherwise the packets will be forwarded across network boundaries. You should be careful with multicasting, and educate yourself before you try it.

Receiver without rtpbin

To receive without using rtpbin:

gst-launch-1.0 -v \
udpsrc address=127.0.0.1 port=50000 caps="application/x-rtp" ! queue ! rtpopusdepay ! queue ! opusdec ! autoaudiosink sync=true \
udpsrc address=127.0.0.1 port=50002 caps="application/x-rtp" ! queue ! rtpvp9depay ! queue ! vp9dec ! autovideosink sync=true

Here, the sender can be restarted without bringing the receiver down.

How to digitize old VHS videos with an EasyCAP UTV007 USB converter on Linux

2018: VHS is dead! If you don’t have a functioning VHS player any more, your only option is to buy second-hand devices. But if you still have old, valuable VHS videos (e.g. family videos) you should digitize them today, as long as there are still working VHS players around.

Our goal is to feed the audio/video (AV) signals coming out of an old VHS player into an EasyCAP UTV007 USB video grabber, which can receive 3 RCA cables (yellow for Composite Video, white for left channel audio, red for right channel audio).

EasyCAP UTV007 USB video grabber

VHS players usually have a SCART output which lucklily carries all the needed signals.

SCART connector

Via a Multi AV SCART adapter you can output the AV signals into three separate RCA cables (male-to-male), and from there into the EasyCap video grabber. If your adapter should have an input/output switch, set it to “output”.

Multi AV Adapter outputting 3 RCA connectors (yellow for Composite Video, white for left channel audio, red for right channel audio)

The EasyCAP USB converter uses a UTV007 chip, which is supported by Linux out-of-the-box. (Who said that installing drivers is a pain in Linux???) After plugging the converter into an USB slot, you should get two additional devices:

  1. A video device called “usbtv”
  2. A sound card called “USBTV007 Video Grabber [EasyCAP] Analog Stereo”

Too see if you have the video device, run v4l2-ctl –list-devices . It will output something like:

usbtv (usb-0000:00:14.0-7): 
       /dev/video0

To see if you have the audio device, run

pactl list | grep -C 50 'Description: USBTV007'

It will output something like:

Source #1
        State: SUSPENDED
        Name: alsa_input.pci-0000_00_14.0-usb-0_7.analog-stereo
        Description: USBTV007 Video Grabber [EasyCAP] Analog Stereo
        Driver: module-alsa-card.c
        Sample Specification: s16le 2ch 48000Hz
        Channel Map: front-left,front-right
        Owner Module: 7
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Base Volume: 65536 / 100% / 0.00 dB
        Monitor of Sink: n/a
        Latency: 0 usec, configured 0 usec
        Flags: HARDWARE DECIBEL_VOLUME LATENCY 
        Properties:
                alsa.resolution_bits = "16"
                device.api = "alsa"
                device.class = "sound"
                alsa.class = "generic"
                alsa.subclass = "generic-mix"
                alsa.name = "USBTV Audio Input"
                alsa.id = "USBTV Audio"
                alsa.subdevice = "0"
                alsa.subdevice_name = "subdevice #0"
                alsa.device = "0"
                alsa.card = "3"
                alsa.card_name = "usbtv"
                alsa.long_card_name = "USBTV Audio at bus 3 device 3"
                alsa.driver_name = "usbcore"
                device.bus_path = "pci-0000:00:14.0-usb-0:7"
                sysfs.path = "/devices/pci0000:00/0000:00:14.0/usb3/3-7/sound/card3"
                device.vendor.name = "Fushicai"
                device.product.name = "USBTV007 Video Grabber [EasyCAP]"
                device.string = "hw:3"
                device.buffering.buffer_size = "22120"
                device.buffering.fragment_size = "11060"
                device.access_mode = "mmap"
                device.profile.name = "analog-stereo"
                device.profile.description = "Analog Stereo"
                device.description = "USBTV007 Video Grabber [EasyCAP] Analog Stereo"
                module-udev-detect.discovered = "1"
                device.icon_name = "audio-card"
        Ports:
                analog-input: Analog Input (priority: 10000)
        Active Port: analog-input
        Formats:
                pcm

To quickly test if you are getting any video, use a webcam application of your choice (e.g. “cheese“) and select “usbtv” as video source under “Preferences”. Note that this will only get video, but no audio.

We will use GStreamer to grab video and audio separately, and mux them together into a container format.

Install GStreamer

To install GStreamer on Debian-based distributions (like Ubuntu), run

apt-get installgstreamer1.0-tools gstreamer1.0-alsa gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly

Test video with GStreamer

Now, test if you can grab the video with GStreamer. This will read the video from /dev/video0 (device name from v4l2-ctl –list-devices above) and directly output in a window:

gst-launch-1.0 v4l2src device=/dev/video0 ! autovideosink

Test audio with GStreamer

Now, test if you can grab the audio with GStreamer. This will read the audio from the ALSA soundcard ID hw:3 (this ID comes from the output of pactl list above) and output it to PulseAudio (should go to your currently selected speakers/headphones):

gst-launch-1.0 alsasrc device="hw:3" ! pulsesink

Convert audio and video into a file

If both audio and video tested OK separately, we now can grab them both at the same time, mux them into a container format, and output it to a file /tmp/vhs.mkv. I’m choosing Matroska .mkv containing H264 video and Ogg Vorbis audio:

gst-launch-1.0 -e \
matroskamux name="muxer" ! queue ! filesink location=/tmp/vhs.mkv \
v4l2src ! queue ! x264enc ! queue ! muxer. \
alsasrc device="hw:3" ! queue ! audioconvert ! queue ! vorbisenc ! queue ! muxer.

Record some video and then press Ctrl+C. The file /tmp/vhs.mkv should now have audio and video.

It would be nice if we could see the video as we are recording it, so that we know when it ends. The command below will do this:

gst-launch-1.0 -e \
matroskamux name="muxer" ! queue ! filesink location=/tmp/vhs.mkv async=false \
v4l2src ! tee name=mytee \
mytee. ! queue ! x264enc ! queue ! muxer. \
mytee. ! queue ! autovideosink \
alsasrc device="hw:3" ! queue ! audioconvert ! queue ! vorbisenc ! queue ! muxer.

You also can re-encode the video by running it through ffmpeg:

ffmpeg -i vhs.mkv -vb 700k -ab 100k seekable-vhs.mkv

You can adjust the video and audio bitrate depending on the type and length of video so that your file will not be too large. The nice side-effect is that the coarser the video encoding, the more of the fine-grained noise in the VHS video is smoothed out.

Voila! You now should be able to record and archive all your old family videos for posterity!

Digitization of VHS video with Gstreamer.

PhantomJS alternative: Write short PyQt scripts instead (phantom.py)

For a project of mine, I needed a ‘headless’ script that renders dynamically generated HTML (via JavaScript) to a PDF file. In looking for existing headless browsers, I found PhantomJS, CasperJS and similar projects.

PhantomJS looked most promising, but it had bugs related to the CSS @media type ‘print’ which made the project useless for generating properly formatted PDFs.[1][2] These issues are still open after three years. As of 2017, the official PhantomJS binary download is still at version 2.1 which includes an older downstream QtWebKit version 5.5.1 [3].

In trying to fix the bugs myself, or upgrade QtWebKit, I spent two full days trying to build PhantomJS, but eventually had to give up because the build system seems to be reworked right now. It seems that instead of patching the custom QtWebKit, the development team is moving towards making “PhantomJS a normal Qt application with system-installed Qt and QtWebKit”. [4]

So, I started questioning PhantomJS. What magic does it really do? By inspecting the source code, I found out that its entire concept hinges on just two Qt functions: QWebFrame::addToJavaScriptWindowObject() and QWebFrame::evaluateJavascript(). The former allows exposing Qt objects to the JavaScript space of WebKit. For example, PhantomJS’s JavaScript API method injectJs is simply a wrapper for an underlying evaluateJavascript call inside of a C++ function of a C++ object which has been previously attached via addToJavaScriptWindowObject.

In short, PhantomJS’s custom (and in my opinion rather sparse) JavaScript API simply calls Qt’s C++ functions.

The obvious disadvantage of this concept is that an extension or fixing of the API requires a recompilation of the C++ code.

It is obviously much simpler to directly script Qt, for example via PyQt.

So, I started writing a short PyQt application, and after just 90 lines of Python code, I had what I needed: a headless browser using an up-to-date version of WebKit, which did not have the shortcomings of the version in PhantomJS.

The script is published on my blog and as a Github gist. I don’t expect getting 20,000 Github stars like others, but honestly, following (perhaps too simple) logic, my phantom.py script should get more. 😉

References:

[1]: https://github.com/ariya/phantomjs/issues/12685

[2]: https://github.com/ariya/phantomjs/issues/13524

[3]: https://github.com/ariya/phantomjs/tree/2.1.1/src/qt

[4]: https://github.com/ariya/phantomjs/issues/14458#issuecomment-242391757