Video Decoding with the Exynos Multi-Format Codec & GStreamer

Exynos SoCs have an IP block known as the Multi-Format Codec (MFC) that allows them to do hardware accelerated video encoding/decoding, and the mainline kernel has a s5p-mfc Video for Linux2 (V4L2) driver that supports the MFC. The s5p-mfc driver is a Memory-to-Memory (M2M) V4L2 driver, it’s called M2M because the kernel moves video buffers from an output queue to a capture queue. The user-space enqueues buffers into the output queue, then the kernel passes these buffers to the MFC where they are converted and put it in the capture queue so the user-space can dequeue them.

The GStreamer (gst) multimedia framework supports V4L2 M2M devices, but only for decoders the v4l2videodec element supports. Randy Li is working to also support M2M encoders in GStreamer (v4l2videoenc), but this hasn’t landed in upstream GStreamer yet.

This post will explain how to use GStreamer and the Linux mainline kernel to do hardware video decoding on an Exynos based machine.

Install a Mainline Kernel on an Exynos Machine

The first step is to install a mainline kernel in the Exynos board. We have been working lately to make the s5p-mfc driver more stable and to have everything enabled in by default, so the Linux kernel version used should be at least v4.8-rc1 and the defconfig used should be exynos_defconfig. I’ve made previous blog posts that explain how to cross compile and install a mainline kernel on different Exynos-based machines, including Exynos5 Odroid boards and Exynos5 Chromebooks, so refer to these to learn how to do this.

GStreamer Uninstalled Setup

The GStreamer package in most Linux distributions doesn’t come with support for the v4l2videodec gst element enabled, so GStreamer should be built from source to do hardware accelerated video decoding. Fortunately, GStreamer has the gst-uninstalled script to setup a development environment that allows the use of gst elements that were built from source and not installed in the system. There are many guides on how to use gst-uninstalled, including this great tutorial from Arun Raghavan.

When building the gst-plugins-good module, the –enable-v4l2-probe configure option must be enabled to allow GStreamer to probe v4l2 devices. This is needed because v4l2 device drivers can be probed in different orders, meaning the video device nodes can change. This option needs to be  enabled explicitly before building gst-plugins-good:

$ cd gst-plugins-good
$ ./configure --enable-v4l2-probe
$ make

Find the Video Device Node for MFC Decoding

At this point, the s5p-mfc driver can be used to do hardware accelerated video decoding. You first need to know which video device nodes are for the s5p-mfc video decoder; gst v4l2videodec expects to use a gst element named v4l2videoNdec, where N is the video device node number.

First, get the video device nodes the s5p-mfc driver has registered; the v4l2-ctl command from the v4l-utils can be use for this:

$ v4l2-ctl --list-devices
s5p-mfc-dec (platform:11000000.codec):
	/dev/video5
	/dev/video6

There are two device nodes the driver has registered, v4l2-ctl can also be used to know more about these nodes:

$ v4l2-ctl --info -d /dev/video5 
Driver Info (not using libv4l2):
	Driver name   : s5p-mfc
	Card type     : s5p-mfc-dec
	Bus info      : platform:11000000.codec
	Driver version: 4.8.0
	Capabilities  : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
$ v4l2-ctl --info -d /dev/video6
Driver Info (not using libv4l2):
	Driver name   : s5p-mfc
	Card type     : s5p-mfc-enc
	Bus info      : platform:11000000.codec
	Driver version: 4.8.0
	Capabilities  : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format

The video device node for the decoder is /dev/video5 in this case, there should be a v4l2video5dec gst element that should be used.

$ gst-inspect-1.0 v4l2video5dec
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                V4L2 Video Decoder
  Klass                    Codec/Decoder/Video
  Description              Decode video streams via V4L2 API
  Author                   Nicolas Dufresne 

Plugin Details:
  Name                     video4linux2
  Description              elements for Video 4 Linux
  ...

Video Decoding

MFC supports a variety of video formats for decoding and produces decoded NV12/NV21 YUV frames. The v4l2-ctl tool can be used with the option –list-formats-out to list all supported formats.

$ v4l2-ctl -d /dev/video5 --list-formats-out
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Output Multiplanar
	Pixel Format: 'H264' (compressed)
	Name        : H.264

	Index       : 1
	Type        : Video Output Multiplanar
	Pixel Format: 'M264' (compressed)
	Name        : H.264 MVC

	Index       : 2
	Type        : Video Output Multiplanar
	Pixel Format: 'H263' (compressed)
	Name        : H.263
	...

Now that the video device node for the decoder and its corresponding v4l2videodec gst element are known, a supported video (e.g. H.264) can be decoded using GStreamer such as with this example pipeline:

$ gst-launch-1.0 filesrc location=test.mov ! qtdemux ! h264parse ! v4l2video5dec ! videoconvert ! kmssink

Happy hacking!

Author: Javier Martinez

Javier Martinez Canillas was a Senior Linux Kernel Developer for the Samsung Open Source Group with a focus working on ARM and Exynos SoC support.

11 thoughts on “Video Decoding with the Exynos Multi-Format Codec & GStreamer”

  1. hello
    great guide,thanks ,much appreciated!
    I am trying to follow this tutorial but is different to what is in this output!!
    $$ v4l2-ctl -d /dev/video5 –list-formats-out
    Index : 0
    Type : Video Output Multiplanar
    Pixel Format: ‘YM12’
    Name : 4:2:0 3 Planes Y/Cr/Cb
    Index : 1
    Type : Video Output Multiplanar
    Pixel Format: ‘YM21’
    Name : 4:2:0 3 Planes Y/Cr/Cb
    Index : 2
    Type : Video Output Multiplanar
    Pixel Format: ‘VM12’
    Name : 4:2:0 2 Planes 16*16 tiles
    Index : 3
    Type : Video Output Multiplanar
    Pixel Format: ‘TM12’
    Name : 4:2:0 2 Planes 64*32 tiles
    Index : 4
    Type : Video Output Multiplanar
    Pixel Format: ‘NM12’
    Name : 4:2:0 2 Planes
    Index : 5
    Type : Video Output Multiplanar
    Pixel Format: ‘NM21’
    Name : 4:2:0 2 Planes Y/CbCr
    —–
    my output not similar to your guide!!
    What should I do? , any help here?

    1. Are you sure the /dev/video5 video device node is the MFC? What does v4l2-ctl –info -d /dev/video5 say?

      Also, what kernel version and board are you testing with?

      1. Thank you for your great answer!
        My output Terminal for this command is:
        $$>> v4l2-ctl –info -d /dev/video6
        Driver Info (not using libv4l2):
        Driver name : MFC
        Card type : decoder
        Bus info :
        Driver version: 1.0.0
        Capabilities : 0x04003003
        Video Capture
        Video Capture Multiplanar
        Video Output
        Video Output Multiplanar
        Streaming
        —————————————————————————-
        —————————————————————————–
        $$>> v4l2-ctl –info -d /dev/video7
        Driver Info (not using libv4l2):
        Driver name : MFC
        Card type : encoder
        Bus info :
        Driver version: 1.0.0
        Capabilities : 0x04003003
        Video Capture
        Video Capture Multiplanar
        Video Output
        Video Output Multiplanar
        Streaming

        Can you give me your image that I download It?
        Thank you very much!!

      2. Hello Madi,

        Ok, so then you should use /dev/video6 and not /dev/video5 (the video device nodes number may change since this depends on the driver’s probe order and that is not deterministic).

        No, I don’t have an image to download it sorry. You can use any armhf rootfs and install mainline Linux and get a GStreamer uninstalled setup.

  2. hi

    i use ubuntu image thats provided for odroid xu3 (can i do that?)
    i have a odroidxu4 and i am trying to show 4k video with gstreanmer but i have problems
    1- my video node is /dev/video6 and /dev/video7
    2- /dev/video6 is decoder and /dev/video7 is encoder
    3- driver name is MFC versin 1.0 instead of s5p-mfc
    4- the MFC driver dosent support h264
    can you help me?
    should i use specific kernel or device tree ?
    can you give me image that you used ?
    thanks

    1. Hello rezapop,

      I’m not familiar with the default image for Odroid XU3 provided by HardKernel, but AFAICT it has a very old 3.10 kernel so the instructions here may not work.

      This post talks about the upstream Linux and gstreamer capabilities, so I don’t think it applies to the vendor tree provided with the Odroid XU3.

      1. hi
        thanks i solve my problem with kernel
        now my problem is kmssink element of gstreamer i cant find it
        i use gst-inspect1.0 kmssink command and i fond no element
        what am i doing?

  3. Hello,
    I’m running on the ODROID XU4 single board computer. When I am logged in graphically and run:
    gst-inspect-1.0 | grep “v4l2”
    I can see the hardware decoders and encoders. When I am logged in remotely via SSH, and not graphically the same command returns no devices. I can still see the devices with v4l2-ctl commands and are present in the /dev/ folder. Any thoughts would be appreciated. I would like to be able to run a stand alone application written with gstreamer.

    Thanks!

Comments are closed.