Media Resource Sharing Through The Media Controller API

Media devices have hardware resources that are shared across several functions. However, media drivers have no knowledge of which resources are shared. At the root of this problem is a lack of a common locking mechanism that can provide access control.

I have been working on solving the sharing problem for a year or so. At the Media Summit in San Jose back in March of this year, we reviewed my Media Token API patch series. It solved the problem with minimal changes to drivers, however, it introduced a new framework, in addition to the existing ones. Every new framework adds maintenance cost. This led to a discussion that identified the existing Media Controller API as a better alternative in the interest of avoiding adding a new framework. Since then, I’ve worked on implementing the solution using the Media Controller API, and I’ve ported it over to Media Controller Next Gen API.

In this article, I will cover how ALSA and au0828 drivers share media resources using the Media Controller Next Gen API on Win-TV HVR 950Q Hybrid USB TV stick. In addition, I will share how media-ctl/mc_nextgen_test tool can be used to generate media graphs for a media device. Let’s first start looking at the differences between Multi-function Device (MFD) and Media devices.

How do Media Devices Differ from a Multi-function Device (MFD)?

Media devices can be very small, yet complex as they consist of a group of independent devices that share an attach point. Each device implements a unique function, and sharing isn’t limited to the attach point. Functions can be shared, and media drivers have to coordinate this sharing. In some cases, non-media drivers are leveraged as in the case of Win-TV HVR 950Q, snd-usb-audio for audio is leveraged to drive the audio chip on the device.

MFD presents itself to the kernel as a single device. The Linux Kernel MFD framework helps identify functions as discrete platform devices. Sharing is limited to the attach point with sub-devices attached to a shared bus. Each function is independent with no shared resources other than the attach point. MFD drivers don’t need to coordinate sharing resources.

Challenges with Sharing Media Resources

Drivers are unaware of shared resources on a media device and they don’t have common data structures to ensure exclusive access to shared resource/function. The fine grain locks that exist today don’t ensure exclusive access and are local to function drivers; there are no global locks or data structures that span all drivers. For example, starting a digital TV application disrupts video streaming. This is because the digital driver can access/change modes while a video (Analog) TV application is using the tuner resource.

Exclusive Access Use-Cases:

  1. Starting Digital application should not disrupt active Video and Audio streams.
  2. Starting Video application should not disrupt active Digital and Audio streams.
  3. Starting Audio application should not disrupt Video and Digital streams.
  4. Querying current configuration (read only access) should work even when the tuner resource is busy.

What is the Media Controller API?

The Media Controller API is a relational media graph framework. Media functions and resources can be represented as nodes on a graph. The media device sits at the top as the root node, and nodes can be connected using media links to represent data paths. These links can be marked active to indicate them as active data paths between two or more nodes; this allows data pipelines to be started.

For example, a tuner node can be linked to a decoder with the tuner operating as the source and the decoder operating as the sink. Then, an additional video sink node can be linked to decoder to display output. A data pipeline can be started between the video node and the tuner once the link between the tuner and decoder is marked as active. As long as this pipeline is active, no other node will be allowed to activate their link to the tuner, thereby reserving the tuner resource for exclusive use.

Using The Media Controller API to Share Resources on a Win-TV HVR 950Q

To illustrate this, take a look at the Win-TV HVR 950Q device/driver view:

Media Resource Sharing Through The Media Controller API - hvr950q_1

As you can see in this device/driver view, USB Device (struct usb_device) and USB Device Parent (struct device) are the only two common data structures across all the drivers for this device. If we were to create a media device at the USB Device Parent, then the media device will be common across all drivers. The Managed Media Controller API allows the media device to be created as a USB Device Parent device resource. This Media Device is created by either au0828 or snd-usb-audio as their probe routines run. The first probe routine that runs wins. This Managed Media Device serves as the root node for the media graph.

Managed Media Controller API:

  • media_device_get_devres()
  • media_device_find_devres()

Media Resource Sharing Through The Media Controller API - hvr950q_2

Drivers use probe routines to create media graph nodes for their functions and resources. Much like when drivers create new media nodes, other drivers might need to take some action. The Media Controller Entity Notify API are new interfaces that enable drivers to register callbacks to take action as entities get added by other drivers.

Media Controller Entity Notify API:

  • media_device_register_entity_notify()
  • media_device_unregister_entity_notify()

Drivers add add entities for their functions and the bridge driver (au0828) creates links between entities via the entity_notify API. Please note, the bridge driver takes ownership for creating links as it is the master driver that knows how resources are shared and the relationships between them.

HVR 950Q Media Graph

Media Resource Sharing Through The Media Controller API - graph_none

This has given us a media graph that is common and shared by all drivers, but the next step is to provide a means to acquire and release shared resources. A new Media Controller Enable source handler grants access to shared resources or returns -EBUSY if not possible.  Its counterpart, the Disable source marks shared resource as free. Bridge drivers (e.g au0828) implement and register enable/disable source handlers to manage access to shared resources on the device. Other drivers call the enable source handler from the media device to gain access to the resources and the disable source handler to release it.

snd-usb-audio and au0828 drivers have been changed to take advantage of the new Media Controller API. Now we look at a few media graphs when various applications own the tuner resource.

Audio Application Controls the Tuner

Media Resource Sharing Through The Media Controller API - graph_audio
When an audio application is running, active pipeline (solid line) is established that start at the XC5000 tuner node and pass through the decoder and audio mixer to provide access to the audio capture end point.

Audio Application Releases the Tuner

Media Resource Sharing Through The Media Controller API - graph_audio_done
The pipeline from the XC5000 tuner node to the decoder is set as inactive. Please note: the links between the decoder, audio mixer, and audio capture node are always enabled.

Video Application is Running

Media Resource Sharing Through The Media Controller API - graph_video
When a video application is running, active pipeline (solid line) is established that start at the XC5000 tuner node and pass through the decoder to provide access to the video node. Please note, the video application opens audio capture device while it is using the tuner. As a result, both video and audio are busy during video streaming and there is no difference between the active pipeline shown in the video and audio graphs.

 

Video Application Releases the Tuner

Media Resource Sharing Through The Media Controller API - graph_video_done
Please note: when a video application is running, both audio and video are held by the video driver.

Digital Application is Running

Media Resource Sharing Through The Media Controller API - graph_dvb
The pipeline between the XC5000 tuner and DTV demodulator is set to active when a digital application is running.

Digital Application Releases the Tuner

Media Resource Sharing Through The Media Controller API - graph_dvb_done
The active pipeline between XC5000 tuner and DTV Demodulator is set to inactive when the digital application releases the tuner.

Video Application is Started while an Audio Application is Running

Media Resource Sharing Through The Media Controller API - graph_audio_video_start_stop
The video application will find the audio device busy and exit gracefully; the active pipeline between the tuner and the Audio capture device continues with no interruptions.

Digital Application is Started while an Audio Application is Running

Media Resource Sharing Through The Media Controller API - graph_audio_dvb_start_stop
The digital application will find the tuner is busy and exit gracefully with no interruption to the active audio pipeline.

Audio Application is Started while a Digital Application is Running

Media Resource Sharing Through The Media Controller API - graph_dvb_audio_start_stop
The audio application will find the tuner busy and exit, leaving the active pipeline between the tuner and DVB demux active.

Video Application is Started while a Digital Application is Running

Media Resource Sharing Through The Media Controller API - graph_dvb_video_start_stop
The video application will find the tuner busy and exit leaving the active pipeline between the tuner and DVB Demux active.

Digital Application is Started while a Video Application is Running

Media Resource Sharing Through The Media Controller API - graph_video_dvb_start_stop

 

MC Next Gen Test Tool

Mauro Carvalho Chehab has developed a new mc_nextgen_test tool to generate the media graphs illustrated above.

Usage: mc_nextgen_test [OPTION…]
A testing tool for the MC next geneneration API
-d, –device=DEVICE media controller device (default: /dev/media0
-D, –dot show in Graphviz format
-e, –entities show entities
-i, –interfaces show pads
-I, –intf-links show interface links
-l, –data-links show data links
-?, –help Give this help list
–usage Give a short usage message
-V, –version Print program version

Demystifying Media Devices

Media devices can be small, yet very complex. I hope this post helps understand the complexity of these devices and how the Media Controller API helps solve the resource sharing problem across media and non-media drivers that control the device.

Author: Shuah Khan

Shuah contributes to multiple aspects of the Linux Kernel, and she maintains the Kernel Selftest framework.

5 thoughts on “Media Resource Sharing Through The Media Controller API”

  1. In the title “How do Media Devices Defer from a Multi-function Device (MFD)?” maybe it’s “differ” instead of “defer”?

    And it was not immediately clear to me what a “digital application” was in this context. I had to guess from the graphs that it was referring to digital TV as opposed to the Analog TV functionality provided by that particular device.

    Besides these little details, it was an interesting reading.

  2. Had a question, if at all is communication possible between the entities? Lets say, entity A, B & C forms an topology. In that topology entity A wants to pass an even to entity B and entity C and vice versa. Is it possible without routing it to user space and just have the communication with in Kernel?

    1. These entities are passive nodes in the graph. Drivers can change the topology by adding or removing entities during run-time, however there is no communication between the entities themselves.

Comments are closed.