Media pipelines are important to virtually any modern device that displays video or audio. It’s important these pipelines are optimized in order to provide responsive video and audio playback on devices that might have tightly constrained hardware resources like smartphones, smartwatches, and IoT devices. During the GStreamer summer hackfest in Montpellier, France one of our areas of focus was on improving how GStreamer performs caps negotiation.
Brief review of caps negotiation
Before data can flow on a GStreamer pipeline, elements must agree on the data format they will use. The process of selecting those formats is called Caps Negotiation and it uses three different types of interactions: caps queries, accept-caps queries and caps events.
The Caps Query is used to ask elements what formats they can receive or produce, and it is recursive in that one element also queries others to avoid exposing formats that wouldn’t be accepted further ahead in the pipeline. Suppose the following pipeline:
Before selecting the audio format to use, Audio Source (AS) needs to know what the Audio Enhancer (AE) accepts. For that, a caps query is used. Note that AE can deal with audio with any number of channels but the Audio Output (AO) only handles audio with 1 or 2 channels. If AE claims to accept any number of channels and AS selects some format unsupported by the AO (more than two channels), the pipeline could fail. To avoid such situations, AE should, before replying to the caps query, find out what the next element is capable of handling. It will use a caps query itself to probe AO for its list of formats, and then it replies to AS with an intersection of what AO and itself could handle, asserting that AS will get a list of options that would work throughout the whole pipeline, and nothing more.
After getting the list of possible formats, AS can now select which one to use. This selection must be communicated to the next element before data flow, and this is done with the caps event. Upon receiving the caps event an element should configure itself to handle the specified format, and it should be ready to receive and process data. GStreamer core intercepts the caps event before it reaches an element and creates an accept-caps query out of it. The accept-caps query is used to validate that an element can accept the format before the caps event is received, this guarantees that the caps event format is something that the element can accept and makes handling it more straightforward. The accept-caps query can also be used independently, but it isn’t common.
Speeding up Negotiation
As accept-caps is automatically handled by the core and hasn’t much practical use for plugin writers or applications, it is mostly forgotten or overlooked. This year, during the GStreamer Summer Hackfest, it was noticed that very few elements implement a proper accept-caps handler, letting the default one act instead. The problem is that the default uses the safest approach possible: it does a caps query to find the possible caps to compare with the received caps. As said above, the caps query looks through the entire pipeline branch recursively while the accept-caps should be a simple, shallow verification that is much less expensive.
During the hackfest, developers went over most of the elements and implemented a proper accept-caps handler for them. Since caps negotiation is one of the steps in starting the pipeline it also helps reduce the time it takes to preroll.
To study the impact the default accept-caps handler has on the number of queries, we performed a benchmark to test the amount of queries and the time they consumed when prerolling a playback pipeline for a movie (MP4, H264, AAC). Using GstTracer, we could count the queries and the time they took to be processed.
Results for Accept-Caps Fixes on x86_64
On x86_64 architecture, these changes resulted in the following improvements:
- Time Spent on Queries: 48% reduction
- Preroll Time: 15% reduction
- Total Number of Queries: 25% reduction
Results for Accept-Caps Fixes on Raspberry Pi 2 Running Tizen
On a Raspberry Pi 2 running Tizen, these fixes resulted in the following improvements:
- Time Spent on Queries: 55% reduction
- Preroll Time: 33% reduction
- Total Number of Queries: 35% reduction
As you can see from these charts, accept-caps handling was a big part of the total of caps queries in these scenarios. GStreamer developers took care of most upstream elements but there are many other remaining issues out there. We recommend all developers double-check their applications for anything that still uses the default handler in order to speed up the application launch.
Do it Yourself
With that said, the process for doing this is very simple and I’ll cover that now. Usually, completing this fix falls in one of two categories. Both of these categories are outlined here.
1. The element accepts exactly what’s in its pad template
Simply use GST_PAD_SET_ACCEPT_TEMPLATE to enable the GST_PAD_FLAG_ACCEPT_TEMPLATE. This will switch the default accept-caps handler to use the pad template caps instead of doing a caps query. Also, verify that all elements and base classes don’t implement a custom handler that would override this.
Some elements depend on probing their libraries to learn about possible caps; it’s also possible to use a correct template pad to generate the caps from code rather than from a string.
2. The element needs custom accept-caps handling
In this case, the best solution is to implement a query handler and complete any checks at that time. It’s possible for the base class of an element to provide a virtual method to add query handling.
If this isn’t possible, use the GstPad’s API to register query handling routines.
The Benefits are Immediate
This is a very simple change that can improve startup time considerably. If you want to read more about how caps negotiation works you can refer to the GStreamer design documentation for negotiation design. Meanwhile, we’ll keep on searching for other ways to optimize caps negotiation in order to continue to improve application startup times, especially for embedded applications.