GStreamer to Gain the First RTSP 2.0 Implementation!

Real Time Stream Protocol 2.0

The RTSP 2.0 was proposed in December 2016 to replace the 1.0 version of the standard; this new version is not backward compatible with the previous one. RTSP 1.0 is almost 20 years old, and it’s done a good job at defining a standard protocol for real time media streaming signaling, but it has not evolved much since then and had some issues that were worth fixing. The new version of this standard aims to resolve inconsistencies, clean up the RFC, add missing definitions, and restructure the document to reach better interoperability between implementers. RTSP 2.0 also comes with a bunch of new features (such as pipelined setup request, to avoid round trip time on initialization) and removes some features that were considered not useful enough (such as the RECORD command).

GStreamer’s RSTP 2.0 Implementation

At Samsung, we decided it was the right time to introduce the very first implementation, and we’ve now readied support for the new RSTP 2.0 protocol. This work has led to a set of patches that have been proposed in the upstream GStreamer project!

How to Test GStreamer Pipelines with gst-validate Scenarios

The gst-validate utility allows for the detection of known issues in GStreamer pipelines, and it’s part of the developer tools the GStreamer community offers through the gst-devtols modules. In this guide, I will demonstrate how to create playback scenarios to test a pipeline’s reaction to a new set of controlling actions.

There are a couple of common and not-so-common scenarios that are already included with the GStreamer validate suite. This set allows for the identification of known error conditions on a running pipeline subjected to the actions the provided scenarios express. Now, what if you want to inspect the reaction of your pipeline to a new set of actions? Then, you need a new scenario that describes it.

What’s in a Scenario?

Scenarios are built from serialized actions on a .scenario file. These actions and their parameters are expressed using a GstStructure-based text format, and includes the following core actions: seek, pause, switch-track, wait, dot-pipeline, set-feature-rank, set-state, set-property, set-debug-threshold, emit-signal, disable-plugin. It also includes a few others implemented in gst validate plugins, like validatefaultinjection, corrupt-socket-recv, validategtk, gtk-put-event, validate-launcher, set-subtitle. This introductory article will deal with only a few of them to provide an idea of how everything fits together.

Let’s say we want to test the resilience of a pipeline to a flush seek to 10s after the pipeline is paused at playback-time 1s for 5s. Lets also say that we want to stop playback at 20s. A preliminary version of our scenario file will look like the following:

description, seek=true, need-clock-sync=true
 pause, name=Our-pause, playback-time=1.0, duration=5.0
 seek, name=Our-forward-seek, playback-time=6.0, start=10.0, flags=accurate+flush
 stop, playback-time=20.0

Whenever you run into a new action name (first member on each line), use the following command to get a full description of what the action does and what parameters you can use to control it:

gst-validate-1.0 -t <action name>

For example, the description line describes two important aspects of the scenario:

  • seek=true – This scenario performs seek operations
  • need-clock-sync=true – The scenario execution needs to be synchronized to the pipeline’s clock

Pretty straightforward isn’t it? Now, it’s necessary to add some boilerplate for this scenario to be fully operational. For example, if the media has a duration that doesn’t allow a proper forward seek from 5s to 10s, it’s likely this scenario won’t provide the desired results. For the sake of this example, let’s inform the scenario runner that the test media should have a minimum duration of 30 seconds. This can be done with the following description line:

description, seek=true, need-clock-sync=true, min-media-duration=30s

The scenario parsing system assumes each line of the scenario file represents an action, but it’s also possible to use backslashes to enter multi-line actions. Here’s how the scenario file looks with a multi-line description (sic) action and a long summary:

description, seek=true, need-clock-sync=true, min-media-duration=30s, \
 summary="This example scenario seeks forward to 10s after pausing playback for 5. \
 Then, it stops the execution at 20s"
 pause, name=Our-pause, playback-time=1.0, duration=5.0
 seek, name=Our-forward-seek, playback-time=6.0, start=10.0, flags=accurate+flush
 stop, playback-time=20.0

By the way, I’ve been carrying this one for the entire article so far, but description is not really an action. Sorry to break it to you, but please consider this an implementation detail you’ll have to live with. It’s referred to as an action in the documentation, so I’m following suit.

Test a GStreamer Pipeline

Now that we have our scenario script, it needs to be saved to a file named after the scenario. In this particular example, we’ll save it as test-seek.scenario and pass this file name (without extension) directly to gst-validate for testing a pipeline:

gst-validate-1.0 --set-scenario test-seek filesrc location=test20sVP9.mkv ! matroskademux ! vp9dec ! autovideosink

Beyond passing the full PATH to the scenario file with –set-scenario, it’s also possible to put this file somewhere else and set the GST_VALIDATE_SCENARIOS_PATH environment variable to point to that location. These are not the only options, but they should be enough to get going with the example.

You can try this procedure out on your own pipeline, but if you want to follow this guide to the letter you will need to download the test20sVP9.mkv sample file. Additionally, if the syntax used to express the pipeline is not something you are familiar with, you can find our best attempt at an explanation on the gst-launch documentation page.

And that’s it! Gst-validate will subject the pipeline to the actions that were scripted via the scenario file, and will provide a report on whether any of the built-on gst-validate tests failed. Beyond a simple statement of problem, sometimes gst-validate will even provide hints towards a full diagnostic.

In this article, we briefly explored the scenario-writing aspect of gst-validate. While the suite of tools goes far beyond, a good understanding of the steps required to express action scenarios provides a great foray into this GStreamer testing system. Moving forward, If you want additional examples, take a look at the real-life examples included with gst-validate, and consider contributing your own by submitting a patch to the gst-devtools component on the project’s Bugzilla.

Good luck!

How to Setup a Modern GStreamer Development Environment Using Meson and gst-build

As you might already know, many open source projects are moving away from autotools as a build system and are embracing Meson. GStreamer was one of the first projects to initiate this move as the community pushed for it to happen.

Meson has many advantages over autotools, but one I would like to talk about in this post is the notion of subprojects, which Meson introduces. Basically, thanks to this it’s easy to build several projects as if it was one; GStreamer has many components that were formerly independent in the build system, meaning that if you wanted to build the latest version of, say, gst-plugins-bad, you also needed to build GStreamer core and GStreamer base one way or another. Previously, we had some scripts to help with this process, but it was still necessary to clone and build everything separately and handle interdependency between things manually.

Today, things are different when using meson; we developed a new module called gst-build that basically sets up all GStreamer modules as subprojects and allows you to build everything in one go! It exposes options to define what gets built using the meson infrastructure. The final impression is that GStreamer becomes one single project to work with when developing with any of its components. For example, tests from all components can be run in one go using ninja test. Also, updating all of GStreamer becomes as easy as running ninja update to update all subprojects and rebuild. In the long run, we would like to have all dependencies built with meson, but for now external dependencies still need to be available using another method (for example, with a package manager on Linux).

Note: gst-build is continuously tested on Linux and Windows (using the msys2_setup.py script from gst-build).

How to Get Started with gst-build

First, get build dependencies for GStreamer core and plugins on your distribution. Commands to do this on some popular distributions follow. This will install a lot of packages, but should mean that you won’t have to play find-the-plugin-dependency for your local build.

  • Fedora: $ sudo yum-builddep gstreamer1-*
  • Debian/Ubuntu: $ sudo apt-get build-dep gstreamer1.0-plugins-{base,good,bad,ugly}
  • Gentoo: having the GStreamer core and plugin packages should suffice
  • Others: drop me a note with the command for your favourite distro, and I’ll add it here

Assuming you have meson, ninja installed, using gst-build is as easy as running the following command.

git clone git://anongit.freedesktop.org/gstreamer/gst-build && mkdir build && meson build/ && ninja -C build/

Now, it’s possible to get directly into an uninstalled environment with

ninja -C build/ uninstalled

All GStreamer modules are cloned when running meson, and the repository can be accessed in the root folder of gst-build. Development and hacking can be done from this directory and rebuilding is a matter of running ninja -C build/ which will rebuild each component that needs it after changes have been made.

Happy hacking!

How to Watch Digital Television in Two Easy Steps with GStreamer and V4L

This article will explain how to use GStreamer to capture Digital Television (DTV) streams; it will focus on terrestrial DTV and ATSC broadcasts in the Silicon Valley area to provide examples, but the principles are the same for every other DTV standard or supported location.

If you want to follow the examples, you will at the very least need a Linux machine with GStreamer and v4l-utils, and a DTV capture device. For my ATSC testing setup I use a WinTV-HVR 950Q USB stick (Hauppauge), connected to a Debian desktop computer that runs the latest code for GStreamer and dvbv5-scan from their respective git repositories, both uninstalled. This setup works well for me as a developer but if you simply want to play DTV streams on your machine, the version from your distribution’s binary packages should suffice.

A Few Notes on Receiver Setup

You’ll need to have your receiver setup properly to capture multi-media content from a radio signal. For this to happen, the capture device needs to have an antenna attached and be positioned in an area with coverage. There are several online sources for information on the latter, for the antenna itself my advice is to get a cheap one like the AmazonBasics Ultra Thin Indoor TV Antenna – 35 Mile Range and go from there. I have used this antenna and other slim ones from Mohu and Winnegard with similar results (~40 channels), both in south San Jose and north Mountain View. Do not blindly buy a cheap amplified antenna without first making sure there’s no passive way around your reception problems, such as ensuring proper antenna positioning, using a better cable for the feed, or investing in another antenna with better gain. The latter usually means you’ll get a more directional antenna, and in turn, only be able to get a signal from a single tower. Keep this in mind!

The Process

This two-step process is quite simple after the preliminary requirements have been dealt with. All you have to do is to perform a channel scan, save its results to a file, and have GStreamer use its contents to figure out each channel’s configuration parameters to tune, capture, demux, decode, and render the channel’s digital content on the screen.

Step 1: Perform a Channel Scan

Depending on how you want to look at this, a DTV broadcast – let’s call it a channel for short – is nothing like an analog TV broadcast, but they do share some compatible aspects. For example, a basic set of RF parameters define both of them, including frequency and modulation. However, there are quite a few newer, analog-incompatible properties that show up in the digital domain, like MPEG Program IDs, Error Correction Codes, etc. All of these parameters, both the basic analog and their digital counterparts, are needed to tune into the channels and ultimately play the broadcasted content. For example, here’s the set of parameters defining KQED in the bay area, an ATSC broadcast:

VCHANNEL = 54.2
 SERVICE_ID = 2
 VIDEO_PID = 65
 AUDIO_PID = 67 68 69
 FREQUENCY = 689000000
 MODULATION = VSB/8
 DELIVERY_SYSTEM = ATSC

The lists grows quite a bit for other broadcast standards, like my home-country’s ISDB-T (a variation of the Brazilian standard, but that’s another story) which is why people seldom remember a given digital channel’s parameters like they used to back in the days when “Channel 5” meant something other than a bootleg version of expensive perfume. We now use hardware and software to identify and store these parameters; the process is known in the tech vernacular as “scanning,” and the next few paragraphs will explain how it’s done.

First, you’ll need to get a “frequency file” for your specific geographic location; this is used to feed a program, like dvbv5-scan, that will output a set of discovered channels (and their associated parameters) stored as a DVB configuration file that the playback software will use.

In Debian, the dtv-scan-tables package provides frequency-table files for pretty much all countries in the known universe :-) These files are the result of the Linux TV community’s efforts to provide open access to this information. If your distribution doesn’t provide this package, or you want to have access to its latest information for your area, you can get the corresponding file directly from their upstream repository.

In the following command-line example you can see the corresponding file for my area being passed as the last argument to dvbv5-scan:

dvbv5-scan -O DVBV5 -o channels-dvbv5.conf /usr/share/dvb/atsc/us-CA-SF-Bay-Area

Obviously, for this to work you need to have your DTV capture dongle (and antenna) attached to your computer. The number of channels added to the resulting channels-dvbv5.conf file will depend mostly on how well your antenna is positioned and your area is covered by broadcast towers. You can get this kind of coverage information from official sources like the FCC in the USA, or community/commercial ones like nocable.org, but there are countless other similar sites you can use so feel free to look around as necessary.

Step 2: Use GStreamer to Capture and Play Digital Television Broadcasts

This part can be incredibly complex and complete or extremely easy and incomplete depending on who you ask. Since this is a two step guide, you shouldn’t expect the former.

As long as dvbv5-scan was able to discover the parameters of at least one channel, you are ready for the final step: to actually watch DTV. For the task we’ll use GStreamer’s magical gst-play, passing the channel name as a parameter (grep the generated channels-dvbv5.conf for your selection, I’m going to be using San Jose’s “KQED+”) and pointing to the parameters file by setting an environment variable. It looks like this

GST_DVB_CHANNELS_CONF=channels-dvbv5.conf gst-play-1.0 dvb://KQED+

Now, the underlying system spawned by this innocent looking gst-play line is, in reality, far more complex. For the capture part alone, it wraps the interaction of at least two dvb-related GStreamer elements: dvbsrc and dvbbasebin. These elements can be configured in a zillion, and I mean it, a zillion different ways. Dvbsrc has some fifty configurable properties that even allow it to tune from space! I plan to write a more in-deep guide that covers a more manual approach later on, so the intention of this first guide was only to get you started, not dizzy.

Good luck and have fun!

Playback Synchronization & Video Walls with GStreamer

Hello again, and I hope you’re having a pleasant end of the year (if you are, you might want to consider avoiding the news until next year).

In a previous post, I wrote about synchronized playback with GStreamer, and work on this has continued apace. Since I last wrote about it, a bunch of work has been completed:

  • Added support for sending a playlist to clients (instead of a single URI),
  • Added the ability to start/stop playback,
  • Cleaned up the API considerably to improve the potential for it to be included upstream,
  • Turned the control protocol implementation into an interface to remove the necessity to use the built-in TCP server (different use-cases might want different transports),
  • Improved overall robustness of code and documentation,
  • Introduced an API for clients to send the server information about themselves, and finally
  • Added an API for the server to send video transformations for each specific client to apply before rendering.

While the other bits are exciting in their own right, in this post I’m going to talk about the last two items.

Video Walls

For those of you who aren’t familiar with the term, a video wall is an array of displays that are aligned to make a larger display; these are often used in public installations. One way to set up a video wall is to have each display connected to a small computer (such as the Raspberry Pi), and have them play a part of a video that’s cropped and scaled for the display it’s connected to.

Playback Synchronization Video Walls with GStreamer - video-wall

The tricky part, of course, is synchronization; this is where gst-sync-server comes in. Since it’s already possible to play a given stream in sync across devices on a network the only missing piece was the ability to distribute a set of per-client transformations that the clients could apply; support for this is now complete.

In order to keep things clean from an API perspective, I took the following approach

  • Clients now have the ability to send a client ID and configuration (which is just a dictionary) when they first connect to the server.
  • The server API emits a signal with the client ID and configuration, this provides the ability to know when a client connects, what kind of display it’s running, and where it’s positioned.
  • The server now has additional fields to send a map of client IDs to a set of video transformations.

This allows us to do fancy things, like having each client manage its own information with the server while dynamically adapting the set of transformations based on what’s connected. Of course, the simpler case of having a static configuration on the server also works.

Demo

Since seeing is believing, here’s a demo of the synchronised playback in action:

The setup is my laptop, which has an Intel GPU, and my desktop, which has an NVidia GPU. These are connected to two monitors.

The video resolution is 1920×800, and I’ve adjusted the crop parameters to account for the bezels, so the video looks continuous. I’ve uploaded the text configuration if you’re curious about what that looks like.

As I mention in the video, the synchronization is not as tight than I would like it to be. This is most likely because of the differing device configurations. I’ve been working with Nicolas Dufresne to address this shortcoming by using some timing extensions the Wayland protocol allows; more news on this as it breaks. More generally, I’ve also worked to quantify the degree of sync, but I’m going to leave that for another day.

p.s. the reason I used kmssink in the demo was that it was the quickest way I know of to get a full-screen video going. I’m happy to hear about alternatives, though

Future Work

Make it Real

I implemented my demo quite quickly by having the example server code use a static configuration. What I would like is to have a proper application that people can easily package and deploy on the embedded systems used in real video walls. If you’re interested in taking this up, I’d be happy to help out. Bonus points if we can dynamically calculate transformations based on client configuration (position, display size, bezel size, etc.)

Hardware Acceleration

One thing that’s bothering me is that the video transformations are applied using GStreamer software elements. This works fine(ish) for the hardware I’m developing on, but in real life OpenGL(ES) or platform specific elements should be used to have hardware-accelerated trasnformations. My initial thoughts are for this to be either an API for playbin or a GstBin that takes a set of transformations as parameters and sets up the best method to do this internally based on whatever sink is available downstream (some sinks provide cropping and other transformations).

Why not Audio?

I’ve only written about video transformations here, but we can do the same with audio transformations too. For example, multi-room audio systems allow you to configure the locations of wireless speakers — so you can set which one is left, right, center, etc. — and the speaker will automatically play the appropriate channel. It should be quite easy to implement this with the infrastructure that’s currently in place.

Here’s to the New Year!

I hope you enjoyed reading this post, I’ve had great responses from a lot of people about how they might be able to use this work. If there’s something you’d like to see, leave a comment or file an issue.

Happy end of the year, and all the best for 2017!

GStreamer Media Synchronization Made Easy

A lesser known, but particularly powerful feature of GStreamer is the ability to synchronize media playback across multiple devices with fairly good accuracy. However, with the way things stand right now, it requires some amount of fiddling and a reasonably thorough knowledge of GStreamer’s synchronization mechanisms. While we have had some excellent talks about these at previous GStreamer conferences, getting things to work is still a fair amount of effort for someone not well-versed with GStreamer.

As part of my work with the Samsung OSG, I’ve been working on addressing this problem by wrapping all the complexity associated with this in a library. The intention is for anyone who wants to implement synchronized streaming between devices on the same network to be able to do so with a few lines of code and basic know-how of writing applications based on GStreamer. I’ve started work on this already, and you can find the code in the creatively named gst-sync-server repo.

Design and API

Let’s make this easier by starting with a picture …

GStreamer Media Synchronization Made Easy - big-arch.svg_

Let’s say you’re writing a simple application that uses two or more devices to play the same video stream in sync. The system would consist of two entities:

  • A server – This is configures what needs to be played. It instantiates a GstSyncServer object on which it can set a URI to be played. There are other controls available, but I’ll get to those in a moment.
  • A client – Each device runs a copy of the client and gets information from the server about what to play and what clock to use to synchronize playback. In practical terms,  a GstSyncClient object is created and given a playbin element that has been configured appropriately (this usually involves setting at least the appropriate video sink that integrates with the UI).

That’s pretty much it. The application instantiates these two objects and starts them up; as long as the clients can access the media URI, the result is multiple streams that are magically synchronized on all devices.

Controlling the Streams

The keen observers among you might have noticed there is a control entity in the diagram above that deals with communicating information from the server to the clients over the network. While I’ve currently implemented a simple TCP protocol for this, my goal is to abstract the control transport interface so that it’s easy to drop in a custom transport (Websockets, REST API, etc.).

The actual sync information is merely a structure that’s marshaled into a JSON string and sent to the clients every time something happens. Once the application has media playing, the next thing is for the server to control playback. This can include

  • changing the media that’s playing, including after the current media ends,
  • pausing and resuming the media,
  • seeking, and
  • “trick modes” such as fast forward or reverse playback.

The first two of these work already, and seeking is on my short-term to-do list. Trick modes, as the name suggests, can be a bit more tricky so I’ll likely get to them after other things are done.

Getting Fancy

My hope is to see this library get used in a few other interesting use cases:

  • Video walls -A number of displays that are stacked together to provide one giant display; each device effectively plays different rectangles from the same video.
  • Multi-room audio – Play the same music across different speakers in a single room, multiple rooms, or even group sets of speakers and play different media on different groups.
  • Media sharing – Play music or videos on your phone while your friends listen and watch at the same time (a silent disco app, anyone?).

What’s Next

At this point, an outline of the API is completed; I still need to create the transport abstraction, but that’s basically a matter of extracting the properties and signals that are included in the existing TCP transport. What I would like is to hear from you, my dear readers who are interested in this library, is the following. Does the API look like it would work for you? Does the transport mechanism I describe above cover what you might need? I’ve included example code that should make it easier to understand how this library is meant to be used.

Depending on the feedback I get, my next steps will be to implement the transport interface, refine the API a bit, fix a bunch of FIXMEs, and then see if this is something we can include in gst-plugins-bad. Feel free to comment either on the Github repo, in the comment section of this post, or via the gstreamer-devel mailing list. Don’t forget to watch this blog for videos and measurements of how GStreamer synchronized fares in real life!

The 2016 GStreamer Hackfest & Conference in Berlin

A few days ago, while on my way back from the GStreamer hackfest and conference combo, I tried to come up with a list of pearls to share here and arrived at a pretty troubling conclusion: there was no easy way to do so.

The Hackfest

I met Luis (De Bethencourt) a kilometer away from my hotel on a cold Saturday morning on October 8th. By then, I had already figured the only exercise I’d be able to get during my short 4-day stay in Berlin was going to be related to my commute, so I decided to embrace walking. We headed for C-Base as soon as we meet at his hotel’s lobby and arrived 10 minutes later to meet the typical familiar faces from the GStreamer community. So did everything start.

The 2016 GStreamer Hackfest Conference in Berlin - cbase2
First day of hackfest at C-Base

The GStreamer developer community, while quite compact, has a surprisingly large number of regulars; people tend to stay around. This is something you hardly notice from outside the project – not even as a user if you aren’t paying attention – but it’s quite evident if you have been around for a while because you keep running into the same people over and over. Granted, you might not know everyone, but chances are good you will eventually get to meet the new ones. They’re likely going to be around for the next gathering anyways.

The OSG GStreamer team, Luis, Arun, Thibault and myself, had some sort of plan. There were areas we had agreed to work on and for the most part, we did. Everyone had a talk to deliver too and we even had a sprint arranged for Sunday, so it looked like we were off to a busy weekend. The way we work closely resembles the community we work with and there’s no single pair of OSG multimedia engineers living in the same part of the globe. A properly timed face to face can do wonders, and it did. For example, Luis had recently started playing in the management big leagues with his promotion to European team lead and it was the first time we saw him exercising his new role. We can spend 8 hours a day on IRC, but there’s nothing like seeing wildlife in it’s native habitat, and you can’t exactly get that from ASCII alone. Images are necessary.

So what were we working on for the hackfest? Well… Here’s the silver-lining run down.

The 2016 GStreamer Hackfest Conference in Berlin - team
OSG team at a break during the GStreamer Hackfest 10/08/2016

Arun spent most of his time presenting and gathering feedback on his synchronized/coordinated playback ideas. We have been thinking about quite a few use cases that might benefit greatly from a more established infrastructure in that regard. Additionally, he helped with the changelog wrap-up for this period and an assorted set of other release-related tasks.

Thibault has been putting a lot of thought into enhancing deep run-time inspection, notification and logging. It shouldn’t come as a surprise then, that besides fixing quite a few GStreamer Editing Services bugs, he spent most of the hackfest coordinating with some of the elders on bringing CTF (Common Trace Format) to GStreamer and continued working on his implementation while considering ideas that may ultimately lead to a tighter integration of traces and logs. More on this as HE develops :-)

Luis entertained himself with a set of tasks better presented as a full list:

  • prep work to write a GStreamer parser element in Rust,
  • write documentation for the AudioVisualizer Base Class,
  • port the puzzle (gst-plugins-bad) element to 1.x,
  • play with the Meson build system and its use for GStreamer,
  • rebase some Linux kernel work (with mainline) to get it ready to submit for 4.9-rc1,
  • research the latest work done in v4l2 and,
  • prepare the OSG Multimedia team sprint.

I worked mostly on digital terrestrial television. Primarily to make sure whatever I was presenting on my Monday talk actually worked in Berlin; a few fixes to the DVB-T supporting code included. I have a signal generator that allows me to test multiple broadcast standards, but there’s nothing like a real-life setup. Relatedly, here’s the channels.conf file for Berlin mitte I came up with while testing. You’re welcome.

And all of a sudden, it was Monday morning.

The Conference

The 2016 GStreamer Hackfest Conference in Berlin - betahaus
Betahaus, the conference Venue as it looked at the end of the event on Tue 10/12/2016

The conference took place at a venue with a completely different vibe this time around; Betahaus defines itself as a place for idea sharing and it looks the part. There were two main conference rooms, the Arena (where I had to fight Monday after lunch) and the Innospace, which was a bit larger. For the most part, the two parallel tracks had a full attendance, with everyone of the hundred-something GStreamer lovers, haters, or both having a proper place to sit. Drinks and lunch were included and complementary, the latter sponsored by Facebook. The conference had a comparatively large set of corporate sponsors but that’s not unlike every single GStreamer Conference I’ve attended. Did I mention people tend to stay around? Well, companies do too.

The big news came from tpm, aka __tim, and officially Tim-Philipp Müller (everyone’s favorite reviewer), right after some slides with the latest development highlights: the project is going to be backed up with a foundation; some really brief notes on the reasoning behind it and spare details followed. This is a somewhat popular idea, and it makes sense. I think everyone who didn’t know about it was in agreement. Way to go!

Going through each one of the talks would be kind of boring. Not to mention redundant, as they were all recorded by Ubicast (long time GStreamer user and conference partner) and are now available online. For the most part, we got the usual mix of incredibly useful light-training sessions and exposure to new and interesting use-cases and novelties. The topics were well balanced and rounded, and the schedule was followed to the letter, always a plus. Our team had a few appearances in it:

The 2016 GStreamer Hackfest Conference in Berlin - talk
Conference talk at the “Innospace” room during the first day of conference
The 2016 GStreamer Hackfest Conference in Berlin - panel
OSG’s Europe lead, Luis de Bethencourt, moderating a panel of GStreamer developers at the end of the conference 12/10/2016

Additionally, Julien Isorce, a Samsung colleague and long-time GStreamer developer, briefly gifted the audience with “Code review in Chromium project + quick update on the Media Process based GStreamer backend”. It’s always great to see Julien and even better to passively pick his brain. Thank you!

The Pearl

Remember I told you right at the beginning how hard was to select what to share from the GST Conference/Hackfest combo-like experience? There’s just too much to process, let alone the overwhelming sense of respect for everyone presenting talks and helping around. It seems almost like any kind of selective process would end up being unfair. I pondered about this for a few minutes while my plane approached home and made a bold choice. If being fair was impossible, I would aim for absolute unfairness and present you with my absolutely arbitrary single pearl of the conference: Lubosz Sarnecki’s Holographic Telecommunication in the Age of Free Software. This VR-focused talk had my attention right from its title and Lubosz delivered to the promise; enthusiastic, complete and even loquacious would be the words I’d use to describe the lecture, but you’d have to see it to understand, and you should.

The End

That was pretty much it. The hackfest went for a weekend, the conference for 2 days and most of the attendees for four to five if you catch my drift. The rest of the world just continued living without paying much attention. Considering we’re a group of developers who enable distraction as part of our work, this might be our very own fault.

Improving Debugging in GStreamer Validate

Debugging GStreamer is often a hard and time-consuming task; because of this, the community has been working to enhance debugging tools and make it simpler in the gst-devtools official module and its gst-validate component (you can find more information about this in my previous post).

Lately, we’ve decided to take a step forward and enhance the GStreamer validate reports for specific and very common GStreamer issues. We started with the classic Not Negotiated Error which basically happens when the elements in the pipeline are not able to agree on a data format (Caps) in which to do the processing. This can happen for many reasons, and until recently, the only way to figure out what went wrong was to read verbose and sometimes hard to read GStreamer debug logs; this is time consuming, particularly for people who are not very familiar with GStreamer. Starting from the next 1.10 release, GstValidate will attempt to explain the precise reason and place in the pipeline where the negotiation could not properly occur. Basically, we started caching information about negotiation events in the pipeline (query caps queries and accept caps events) in the GstValidateMonitor-s so that we can figure out what led to the error and report it.

Examples of New Debugging Helpers

For example, for the following pipeline that can’t be negotiated:

gst-validate-1.0 audiotestsrc ! input-selector ! capsfilter name=capsfilter caps=video/x-raw ! fakesink

GstValidate will output the following report:

critical : a NOT NEGOTIATED message has been posted on the bus.
 Detected on pipeline0
 Details : Error message posted by: audiotestsrc0
 Caps negotiation failed starting from pad 'capsfilter:sink' as the QUERY_CAPS returned EMPTY caps for the following possible reasons:
 - Downstream caps struct 0 name 'video/x-raw' differs from filter caps struct 0 name 'audio/x-raw'

Which explains in ‘plain English’ that the data coming inside capsfilter is an audio stream, but we’re trying to force a video stream using the capsfilter, which is simply not possible and thus, produces the Not Negotiated Error.

Here’s another example where the problem happens later in the negotiation (during the ACCEPT_CAPS event):

gst-validate-1.0 audiotestsrc ! audio/x-raw,channels=2 ! audioconvert ! capsfilter caps=audio/x-raw,channels=6,channel-mask='(bitmask)0x32'  name=capsfilter ! fakesink

GStreamer validate will report the following issue:

 critical : a NOT NEGOTIATED message has been posted on the bus.
 Detected on pipeline0
 Details : Error message posted by: audiotestsrc0
 Caps negotiation failed at pad 'capsfilter:sink' as it refused caps: audio/x-raw, layout=(string)interleaved, rate=(int)44100, format=(string)S16LE, channels=(int)6, channel-mask=(bitmask)0x000000000000003f
 - Field 'channel-mask' downstream value from structure 0 '(GstBitmask)0x000000000000003f' can't intersect with filter value from structure number 0 '(GstBitmask)0x0000000000000032'

This informs us there is incompatibility in the ‘channel-mask’ field of the caps proposed by the audioconvert element and the caps we are forcing to be used in the capsfilter (this seems to be a bug in audioconvert as it does not take into account the value returned during QUERY_CAPS).

GstValidate is now also able to add backtraces to its reports if the user explicitly asks for them by setting the GST_VALIDATE_REPORTING_DETAILS variable to all for the wanted messages. The backtraces will contain the stack leading to the message report with the source files names and line numbers.

We are planning to add more of these detailed reports as we have a good idea of  how to do it. Finally, if you have any ideas to contribute to this, please let us know either in the comments here or on the gstreamer-devel mailing list.

Wayland Uninstalled, the Easy Way

I recently had to start looking at some GStreamer & Wayland integration issues and, as everyone would, commenced by trying to setup a Wayland development environment. Before getting my feet wet though, I decided to have a chat about this with Derek Foreman: our resident Wayland expert. This isn’t surprising because on our team, pretty much every task starts by having a conversation (virtual or not) with one of the field specialists in the group. The idea is to save time, as you might have guessed.

This time around I was looking for a fairly trivial piece of info:

Me – “Hey Derek, I have Wayland installed on my distro for some reason – I don’t really want to take a look at now – and I would like to setup an upstream (development) Wayland environment without messing it up. Do you have some script like GStreamer’s gst-uninstalled so I can perform this feat without messing with my local install at all?

Derek – “Hey Reynaldo, No.”

After that, and taking care of the usual I-forgot-to-ask-how-you-were-doings, I’m not sure who’s idea it was but we decided to try to upstream a wl_uninstalled script (the name had to match existing infrastructure, but this is a long and boring story you don’t want to know about) that, like gst-uninstalled, enables developers to quickly setup and use a build and run-time environment consisting of an uninstalled set of interdependent Wayland repositories. The idea is not new, but is still extremely useful. Sure, you can perform similar feats by littering your local disk (yes, I’m that old) with custom-prefix installs of pretty much anything and hope to hit the right buttons to build and exec stuff on your neat patchwork piece, but hey, it is bad taste OK? and you will waste a lot of time, especially the first time.

As if the foreword wasn’t boring enough and just to try your resilience, here are the instructions to use this beauty:

How Wayland Uninstalled Works

Essentially, wl_uninstalled is a helper script that automates the setup needed to build and work with an uninstalled Wayland/Weston environment comprised at least of the wayland, wayland-protocols, libinput, and weston repositories. The wl_uninstalled script provides a shell environment where all build and run-time dependencies are resolved in such a way that the uninstalled versions of these projects take precedence.

A Quick Guide to Using Wayland Uninstalled

I’ll use Weston as an example although other Wayland-based projects should work as well.

Edit a local copy of the script to make $WLD point to the base directory where the repositories are located, make sure to use the absolute path. Then, after executing the script, issue the following commands to get everything built and weston running from the uninstalled environment:

cd $WLD
for i in wayland wayland-protocols libinput weston; do cd $i && ./autogen.sh && make && cd ..; done
weston &

Here’s a short console-cast that shows this same procedure live.

This work is now upstream, you can see the discussions by searching for ‘uninstalled’ in the Wayland mailing lists discussions for August. As an added bonus, it turned out that fiddling with this uninstalled environment mambo-jambo lead to the uncovering of a few additional broken pieces of buildsystem/pkg-config cruft that we fixed too. ‘Cause we are good as that see ;)

Back to my own use case – GStreamer / Wayland integration work -, now I can simply exec the wl_uninstalled script before gst-uninstalled and I’m all set. My uninstalled GStreamer environment will use my (in-turn) uninstalled Wayland/Weston environment. This makes debugging, fixing and integrating against the upstream version of both projects a breeze, and my filesystem more FHS-compliant. But then again, who cares about that; you probably just want all this to be quick and painless. I’m in the same boat.

Note:

While all patches for this are on Wayland’s public mailing list, not all have been merged yet.If you want to try the procedure described here you will need to manually apply the missing ones till they find their way into the official repository:

  • wayland
  • weston
  • wayland-build-tools

The official version of the script lives here.

Video Decoding with the Exynos Multi-Format Codec & GStreamer

Exynos SoCs have an IP block known as the Multi-Format Codec (MFC) that allows them to do hardware accelerated video encoding/decoding, and the mainline kernel has a s5p-mfc Video for Linux2 (V4L2) driver that supports the MFC. The s5p-mfc driver is a Memory-to-Memory (M2M) V4L2 driver, it’s called M2M because the kernel moves video buffers from an output queue to a capture queue. The user-space enqueues buffers into the output queue, then the kernel passes these buffers to the MFC where they are converted and put it in the capture queue so the user-space can dequeue them.

The GStreamer (gst) multimedia framework supports V4L2 M2M devices, but only for decoders the v4l2videodec element supports. Randy Li is working to also support M2M encoders in GStreamer (v4l2videoenc), but this hasn’t landed in upstream GStreamer yet.

This post will explain how to use GStreamer and the Linux mainline kernel to do hardware video decoding on an Exynos based machine.

Install a Mainline Kernel on an Exynos Machine

The first step is to install a mainline kernel in the Exynos board. We have been working lately to make the s5p-mfc driver more stable and to have everything enabled in by default, so the Linux kernel version used should be at least v4.8-rc1 and the defconfig used should be exynos_defconfig. I’ve made previous blog posts that explain how to cross compile and install a mainline kernel on different Exynos-based machines, including Exynos5 Odroid boards and Exynos5 Chromebooks, so refer to these to learn how to do this.

GStreamer Uninstalled Setup

The GStreamer package in most Linux distributions doesn’t come with support for the v4l2videodec gst element enabled, so GStreamer should be built from source to do hardware accelerated video decoding. Fortunately, GStreamer has the gst-uninstalled script to setup a development environment that allows the use of gst elements that were built from source and not installed in the system. There are many guides on how to use gst-uninstalled, including this great tutorial from Arun Raghavan.

When building the gst-plugins-good module, the –enable-v4l2-probe configure option must be enabled to allow GStreamer to probe v4l2 devices. This is needed because v4l2 device drivers can be probed in different orders, meaning the video device nodes can change. This option needs to be  enabled explicitly before building gst-plugins-good:

$ cd gst-plugins-good
$ ./configure --enable-v4l2-probe
$ make

Find the Video Device Node for MFC Decoding

At this point, the s5p-mfc driver can be used to do hardware accelerated video decoding. You first need to know which video device nodes are for the s5p-mfc video decoder; gst v4l2videodec expects to use a gst element named v4l2videoNdec, where N is the video device node number.

First, get the video device nodes the s5p-mfc driver has registered; the v4l2-ctl command from the v4l-utils can be use for this:

$ v4l2-ctl --list-devices
s5p-mfc-dec (platform:11000000.codec):
	/dev/video5
	/dev/video6

There are two device nodes the driver has registered, v4l2-ctl can also be used to know more about these nodes:

$ v4l2-ctl --info -d /dev/video5 
Driver Info (not using libv4l2):
	Driver name   : s5p-mfc
	Card type     : s5p-mfc-dec
	Bus info      : platform:11000000.codec
	Driver version: 4.8.0
	Capabilities  : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
$ v4l2-ctl --info -d /dev/video6
Driver Info (not using libv4l2):
	Driver name   : s5p-mfc
	Card type     : s5p-mfc-enc
	Bus info      : platform:11000000.codec
	Driver version: 4.8.0
	Capabilities  : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format

The video device node for the decoder is /dev/video5 in this case, there should be a v4l2video5dec gst element that should be used.

$ gst-inspect-1.0 v4l2video5dec
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                V4L2 Video Decoder
  Klass                    Codec/Decoder/Video
  Description              Decode video streams via V4L2 API
  Author                   Nicolas Dufresne 

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  ...

Video Decoding

MFC supports a variety of video formats for decoding and produces decoded NV12/NV21 YUV frames. The v4l2-ctl tool can be used with the option –list-formats-out to list all supported formats.

$ v4l2-ctl -d /dev/video5 --list-formats-out
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Output Multiplanar
	Pixel Format: 'H264' (compressed)
	Name        : H.264

	Index       : 1
	Type        : Video Output Multiplanar
	Pixel Format: 'M264' (compressed)
	Name        : H.264 MVC

	Index       : 2
	Type        : Video Output Multiplanar
	Pixel Format: 'H263' (compressed)
	Name        : H.263
	...

Now that the video device node for the decoder and its corresponding v4l2videodec gst element are known, a supported video (e.g. H.264) can be decoded using GStreamer such as with this example pipeline:

$ gst-launch-1.0 filesrc location=test.mov ! qtdemux ! h264parse ! v4l2video5dec ! videoconvert ! kmssink

Happy hacking!