The 2016 GStreamer Hackfest & Conference in Berlin

A few days ago, while on my way back from the GStreamer hackfest and conference combo, I tried to come up with a list of pearls to share here and arrived at a pretty troubling conclusion: there was no easy way to do so.

The Hackfest

I met Luis (De Bethencourt) a kilometer away from my hotel on a cold Saturday morning on October 8th. By then, I had already figured the only exercise I’d be able to get during my short 4-day stay in Berlin was going to be related to my commute, so I decided to embrace walking. We headed for C-Base as soon as we meet at his hotel’s lobby and arrived 10 minutes later to meet the typical familiar faces from the GStreamer community. So did everything start.

The 2016 GStreamer Hackfest Conference in Berlin - cbase2
First day of hackfest at C-Base

The GStreamer developer community, while quite compact, has a surprisingly large number of regulars; people tend to stay around. This is something you hardly notice from outside the project – not even as a user if you aren’t paying attention – but it’s quite evident if you have been around for a while because you keep running into the same people over and over. Granted, you might not know everyone, but chances are good you will eventually get to meet the new ones. They’re likely going to be around for the next gathering anyways.

The OSG GStreamer team, Luis, Arun, Thibault and myself, had some sort of plan. There were areas we had agreed to work on and for the most part, we did. Everyone had a talk to deliver too and we even had a sprint arranged for Sunday, so it looked like we were off to a busy weekend. The way we work closely resembles the community we work with and there’s no single pair of OSG multimedia engineers living in the same part of the globe. A properly timed face to face can do wonders, and it did. For example, Luis had recently started playing in the management big leagues with his promotion to European team lead and it was the first time we saw him exercising his new role. We can spend 8 hours a day on IRC, but there’s nothing like seeing wildlife in it’s native habitat, and you can’t exactly get that from ASCII alone. Images are necessary.

So what were we working on for the hackfest? Well… Here’s the silver-lining run down.

The 2016 GStreamer Hackfest Conference in Berlin - team
OSG team at a break during the GStreamer Hackfest 10/08/2016

Arun spent most of his time presenting and gathering feedback on his synchronized/coordinated playback ideas. We have been thinking about quite a few use cases that might benefit greatly from a more established infrastructure in that regard. Additionally, he helped with the changelog wrap-up for this period and an assorted set of other release-related tasks.

Thibault has been putting a lot of thought into enhancing deep run-time inspection, notification and logging. It shouldn’t come as a surprise then, that besides fixing quite a few GStreamer Editing Services bugs, he spent most of the hackfest coordinating with some of the elders on bringing CTF (Common Trace Format) to GStreamer and continued working on his implementation while considering ideas that may ultimately lead to a tighter integration of traces and logs. More on this as HE develops :-)

Luis entertained himself with a set of tasks better presented as a full list:

  • prep work to write a GStreamer parser element in Rust,
  • write documentation for the AudioVisualizer Base Class,
  • port the puzzle (gst-plugins-bad) element to 1.x,
  • play with the Meson build system and its use for GStreamer,
  • rebase some Linux kernel work (with mainline) to get it ready to submit for 4.9-rc1,
  • research the latest work done in v4l2 and,
  • prepare the OSG Multimedia team sprint.

I worked mostly on digital terrestrial television. Primarily to make sure whatever I was presenting on my Monday talk actually worked in Berlin; a few fixes to the DVB-T supporting code included. I have a signal generator that allows me to test multiple broadcast standards, but there’s nothing like a real-life setup. Relatedly, here’s the channels.conf file for Berlin mitte I came up with while testing. You’re welcome.

And all of a sudden, it was Monday morning.

The Conference

The 2016 GStreamer Hackfest Conference in Berlin - betahaus
Betahaus, the conference Venue as it looked at the end of the event on Tue 10/12/2016

The conference took place at a venue with a completely different vibe this time around; Betahaus defines itself as a place for idea sharing and it looks the part. There were two main conference rooms, the Arena (where I had to fight Monday after lunch) and the Innospace, which was a bit larger. For the most part, the two parallel tracks had a full attendance, with everyone of the hundred-something GStreamer lovers, haters, or both having a proper place to sit. Drinks and lunch were included and complementary, the latter sponsored by Facebook. The conference had a comparatively large set of corporate sponsors but that’s not unlike every single GStreamer Conference I’ve attended. Did I mention people tend to stay around? Well, companies do too.

The big news came from tpm, aka __tim, and officially Tim-Philipp Müller (everyone’s favorite reviewer), right after some slides with the latest development highlights: the project is going to be backed up with a foundation; some really brief notes on the reasoning behind it and spare details followed. This is a somewhat popular idea, and it makes sense. I think everyone who didn’t know about it was in agreement. Way to go!

Going through each one of the talks would be kind of boring. Not to mention redundant, as they were all recorded by Ubicast (long time GStreamer user and conference partner) and are now available online. For the most part, we got the usual mix of incredibly useful light-training sessions and exposure to new and interesting use-cases and novelties. The topics were well balanced and rounded, and the schedule was followed to the letter, always a plus. Our team had a few appearances in it:

The 2016 GStreamer Hackfest Conference in Berlin - talk
Conference talk at the “Innospace” room during the first day of conference
The 2016 GStreamer Hackfest Conference in Berlin - panel
OSG’s Europe lead, Luis de Bethencourt, moderating a panel of GStreamer developers at the end of the conference 12/10/2016

Additionally, Julien Isorce, a Samsung colleague and long-time GStreamer developer, briefly gifted the audience with “Code review in Chromium project + quick update on the Media Process based GStreamer backend”. It’s always great to see Julien and even better to passively pick his brain. Thank you!

The Pearl

Remember I told you right at the beginning how hard was to select what to share from the GST Conference/Hackfest combo-like experience? There’s just too much to process, let alone the overwhelming sense of respect for everyone presenting talks and helping around. It seems almost like any kind of selective process would end up being unfair. I pondered about this for a few minutes while my plane approached home and made a bold choice. If being fair was impossible, I would aim for absolute unfairness and present you with my absolutely arbitrary single pearl of the conference: Lubosz Sarnecki’s Holographic Telecommunication in the Age of Free Software. This VR-focused talk had my attention right from its title and Lubosz delivered to the promise; enthusiastic, complete and even loquacious would be the words I’d use to describe the lecture, but you’d have to see it to understand, and you should.

The End

That was pretty much it. The hackfest went for a weekend, the conference for 2 days and most of the attendees for four to five if you catch my drift. The rest of the world just continued living without paying much attention. Considering we’re a group of developers who enable distraction as part of our work, this might be our very own fault.

Faster GStreamer Pipeline Startup Through Improved Caps Negotiation

Media pipelines are important to virtually any modern device that displays video or audio. It’s important these pipelines are optimized in order to provide responsive video and audio playback on devices that might have tightly constrained hardware resources like smartphones, smartwatches, and IoT devices. During the GStreamer summer hackfest in Montpellier, France one of our areas of focus was on improving how GStreamer performs caps negotiation.

Brief review of caps negotiation

Before data can flow on a GStreamer pipeline, elements must agree on the data format they will use. The process of selecting those formats is called Caps Negotiation and it uses three different types of interactions: caps queries, accept-caps queries and caps events.

The Caps Query is used to ask elements what formats they can receive or produce, and it is recursive in that one element also queries others to avoid exposing formats that wouldn’t be accepted further ahead in the pipeline. Suppose the following pipeline:

Faster GStreamer Pipeline Startup Through Improved Caps Negotiation - capsnegotiation-audio
Pipeline Example

Before selecting the audio format to use, Audio Source (AS) needs to know what the Audio Enhancer (AE) accepts. For that, a caps query is used. Note that AE can deal with audio with any number of channels but the Audio Output (AO) only handles audio with 1 or 2 channels.  If AE claims to accept any number of channels and AS selects some format unsupported by the AO (more than two channels), the pipeline could fail. To avoid such situations, AE should, before replying to the caps query, find out what the next element is capable of handling. It will use a caps query itself to probe AO for its list of formats, and then it replies to AS with an intersection of what AO and itself could handle, asserting that AS will get a list of options that would work throughout the whole pipeline, and nothing more.

After getting the list of possible formats, AS can now select which one to use. This selection must be communicated to the next element before data flow, and this is done with the caps event. Upon receiving the caps event an element should configure itself to handle the specified format, and it should be ready to receive and process data. GStreamer core intercepts the caps event before it reaches an element and creates an accept-caps query out of it. The accept-caps query is used to validate that an element can accept the format before the caps event is received, this guarantees that the caps event format is something that the element can accept and makes handling it more straightforward. The accept-caps query can also be used independently, but it isn’t common.

Speeding up Negotiation

As accept-caps is automatically handled by the core and hasn’t much practical use for plugin writers or applications, it is mostly forgotten or overlooked. This year, during the GStreamer Summer Hackfest, it was noticed that very few elements implement a proper accept-caps handler, letting the default one act instead. The problem is that the default uses the safest approach possible: it does a caps query to find the possible caps to compare with the received caps. As said above, the caps query looks through the entire pipeline branch recursively while the accept-caps should be a simple, shallow verification that is much less expensive.

During the hackfest, developers went over most of the elements and implemented a proper accept-caps handler for them. Since caps negotiation is one of the steps in starting the pipeline it also helps reduce the time it takes to preroll.

To study the impact the default accept-caps handler has on the number of queries, we performed a benchmark to test the amount of queries and the time they consumed when prerolling a playback pipeline for a movie (MP4, H264, AAC). Using GstTracer, we could count the queries and the time they took to be processed.

Results for Accept-Caps Fixes on x86_64

Faster GStreamer Pipeline Startup Through Improved Caps Negotiation - ACF-chart-x86-11

Faster GStreamer Pipeline Startup Through Improved Caps Negotiation - ACF-chart-x86-21

On x86_64 architecture, these changes resulted in the following improvements:

  • Time Spent on Queries: 48% reduction
  • Preroll Time: 15% reduction
  • Total Number of Queries: 25% reduction

Results for Accept-Caps Fixes on Raspberry Pi 2 Running Tizen

Faster GStreamer Pipeline Startup Through Improved Caps Negotiation - ACF-chart-Rpi-11

Faster GStreamer Pipeline Startup Through Improved Caps Negotiation - ACF-chart-Rpi-21

On a Raspberry Pi 2 running Tizen, these fixes resulted in the following improvements:

  • Time Spent on Queries: 55% reduction
  • Preroll Time: 33% reduction
  • Total Number of Queries: 35% reduction

As you can see from these charts, accept-caps handling was a big part of the total of caps queries in these scenarios. GStreamer developers took care of most upstream elements but there are many other remaining issues out there. We recommend all developers double-check their applications for anything that still uses the default handler in order to speed up the application launch.

Do it Yourself

With that said, the process for doing this is very simple and I’ll cover that now. Usually, completing this fix falls in one of two categories. Both of these categories are outlined here.

1. The element accepts exactly what’s in its pad template

Simply use GST_PAD_SET_ACCEPT_TEMPLATE to enable the GST_PAD_FLAG_ACCEPT_TEMPLATE. This will switch the default accept-caps handler to use the pad template caps instead of doing a caps query. Also, verify that all elements and base classes don’t implement a custom handler that would override this.

Example: gst-plugins-good:1b27badcfd5c37da59ad1dfa76154358a2f21f4d

Some elements depend on probing their libraries to learn about possible caps; it’s also possible to use a correct template pad to generate the caps from code rather than from a string.

2. The element needs custom accept-caps handling

In this case, the best solution is to implement a query handler and complete any checks at that time. It’s possible for the base class of an element to provide a virtual method to add query handling.

Example: gst-plugins-bad:9e99102b48201abba60e6fee0c00518bcb2a3dba

If this isn’t possible, use the GstPad’s API to register query handling routines.

The Benefits are Immediate

This is a very simple change that can improve startup time considerably. If you want to read more about how caps negotiation works you can refer to the GStreamer design documentation for negotiation design. Meanwhile, we’ll keep on searching for other ways to optimize caps negotiation in order to continue to improve application startup times, especially for embedded applications.

Getting Things Done at the 2015 GStreamer Hackfest

Over the weekend of March 13-15th, the Samsung Open Source Group office in Staines-upon-Thames, UK, hosted 34 developers of the GStreamer  project for a hackfest. GStreamer is a library for constructing graphs of media-handling components, and its uses range from simple music file playback and audio/video streaming to complex audio mixing and video processing.

A lot of familiar faces showed up, as well as an unusual number of new people, and it was a very productive hackfest. While everybody hammered away on laptops, we worked on and discussed a variety of topics related to both the framework and applications.

Discussions to Be Had…

Some of the discussions that took place on the framework side included:

  • How to move forward with the DASH common encryption – Patches have been sitting in Bugzilla for this for a while. An agreement was reached on how to simplify things and make them more generic so its possible to add a minimal, new API rather than needing to add specialized libraries for each encryption scheme.
  • Moving bug tracking from Bugzilla to Phabricator – There seems to be a general consensus this is what we need to do; the main question is what needs to be done before this is possible. We still need to determine where to host it, developing scripts for porting the current bug information, and any potential new maintenance needs.
  • New design for how glimagesink outputs textures into applications – The first application for this is webkitgtk. Instead of having a custom webkitvideosink (defined and registered in WebKit code), you can now have glimagesink set up a GL context  for resource sharing. Glimagesink does not create an internal window, but rather creates a GL context which shares resources with WebKit’s GL context.
  • Plans for gap events supported by the aggregator base class – This is so it can be used with sparse streams, and the goal is to use this in a new SPU/subtitles base class.
  • How the RTP stack deals with various network conditions and how to improve this functionality – Currently, when there is a network burst (like when a stream starts) the jitterbuffer deals with it very poorly.
  • Customary discussion about documentation – The Pitivi people have implemented a more dynamic method for dealing with API docs, linking examples to API docs in a more modern way. The general consensus is that it could be a good framework to write more docs. These efforts have been postponed for a future Hackfest.

There were also a few questions and ideas related to PulseAudio development and some discussion about this topic. Finally, we had mandatory talks about ideas for GStreamer 2.0 and what we want to change in the future. Nothing was urgent, so 1.0 is here to stay for the foreseeable future.

…And Work to Be Done

Check out the event page to see all of the developers who contributed to this hackfest. Here are some of the highlights of our hands-on work:

  • Tim fixed some long-standing issues with the MPEG-TS muxer. He made the “alignment” property work for UDP streaming, and fixed a problem with pass through delta unit / keyframe flagging so that tcpserversink can keep a backlog from the last keyframe and do burst-on-connect when new clients connect.
  • Anton ported all of the 15 shader effects of GLeffects from GL to GLES2.
  • Arun prototyped his work on profiles in audio sinks/sources: work he revived from a long pending bug report. The idea is that we should allow clients to specify what kind of media they are playing and adapt our filtering and buffering appropriately. He wants to make setting up the sink easy for developers and save power where possible.
  • While fixing a bug, Arun also stumbled on a problem with the way PulseAudio sets up latency in capture devices for recording. He wrote a solution for this, and after some testing will submit it for review.
  • Thibault refactored the ges-launch command line interface branch which involved some internal cleanups and refactoring.
  • Wim reviewed and merged the patches from William that improved file descriptor passing between GStreamer processes. William has been toying with pulsevideo for his project which involves capturing video and passing it  between containers. The captured video is turned into a file descriptor which is passed between 2 GStreamer pipelines using sockets.
  • Jan and Nirbeek took a look at the alpha blending code in the compositor element and found many bugs. They fixed the alpha blending implementation in ORC for SSE and PPC versions and briefly looked at the ARM code. Nirbeek also wrote some unit tests for a video-converter and found some rounding errors that looked larger than expected. Wim investigated a little and experimented with different ways to make the rounding errors smaller. This presents a trade off between speed and accuracy, and no real solution has been found yet.
  • Nicolas, Thiago, and Victor were hacking on v4l2src and working to find methods that can be used to implement renegotiation of the format. The problem is that with the v4l2 API, its only possible to change the capture format after all buffers are returned to the driver, so it’s necessary to drain all the buffers from the pipeline.
  • Jan and Wim briefly looked at the current state of the stereoscopic video patches. The biggest challenge is to represent all possible layouts in the caps and negotiate them between elements. Jan made a new GstFlagSet type to GStreamer in order to simplify negotiation. There is some more work to do before this is ready to be merged, but it looks like they are making progress and we hope those patches will soon be ready.
  • Sebastian started merging and cleaning up an Android camera source element.
  • Thiago refactored camerabin to simplify its pipeline and be able to use bufferpools.
  • Finally, there were a lot of other bug fixes and patch reviews all around.

The Best Part

Since the second day of the Hackfest happened to be Pi Day (March 14th). We enjoyed some delicious pieces of pie.

 

Our next event, a GStreamer Conference, is scheduled to coincide with LinuxCon EU in Dublin, Ireland which is scheduled for October 7th – 9th. Looking forward to seeing you all there. Stay tuned for more details as this date approaches. In the meantime, check out this time lapse video of the event, and thank you everybody for coming!