How to Identify Media Pipeline Problems with GStreamer Regression Tests

Among the tests run daily on the GStreamer continuous integration server, there is a suite for detecting regressions and dangerous conditions called gst-validate. It’s a somewhat recent addition in the history of the project; in short, it activates a number of monitors in running pipelines to detect and report possible issues. These checks can be enabled for any pipeline, and wrap over all elements. gst-validate is run for a number of scenarios and formats that simulate real-world use cases, and it will catch issues for any element in the pipelines under these scenarios. Thibault Saunier, the lead maintainer of gst-validate, has written a very good summary of how it works.

Another important part of gst-validate is the media verification tooling, called media-check. This tool compares playback data from a pipeline using the latest git master of GStreamer with a reference file from a previous version, any differences are reported and can be manually inspected to verify if the changes are intentional or indeed represent a regression. This playback data includes timestamps of frames and format details such as resolution, framerate, metadata tags, etc. Running these tests for new commits makes it easy to see if a regression might have been introduced, and it is a powerful aid in maintaining the large number of features and scenarios work with newly merged patches.

Running gst-validate Integration Tests

gst-validate can be found in the project’s git repository, and it contains a tool called gst-validate-launcher that can be used to run the tests. To start, download and build gst-validate:

git clone cgit.freedesktop.org/gstreamer/gst-devtools

gst-devtools contains a collection of developer utilities, validate is one of those. Go into the gst-devtools/validate directory and build it:

cd gst-devtools/validate 
./autogen.sh 
make

autogen.sh will also run a configure script that checks to make sure your system has the required build dependencies. If any are missing, please install them using your distro’s package manager.

With validate built, you can now run the gst-validate-launcher. It is found inside the tools directory of validate. The usual way of running it is:

./tools/gst-validate-launcher --sync-all

In order to run some tests, gst-validate will need a set of media files of different formats, the –sync-all flag will make it download those files first. These are stored in a separate repository and are checked out to your home directory, at ~/gst-validate. After the download is over, it will start the tests. For every one of them, the result will be printed, and in the case of failure the path to log files will be printed.

A number of optional flags are available for this tool, such as:

  • -j X: to have ‘X’ tests running in parallel
  • -m: to ‘mute’ the tests, meaning that it won’t produce any sound or visual output. This makes it use different elements so the test results might be different. It is still useful if you want to keep using the computer while tests are running without being disturbed with sounds or pop up windows.
  • -gdb or -valgrind: to run tests with gdb or valgrind, respectively

Other interesting options are available, check the –help output to find more.

A Test Failed, What Now?

In the case that a test fails you should try to reproduce it with the latest GStreamer release or git. If it still persists, please file a bug if it is related to upstream elements, even if you are the one who wants to prepare a patch to fix it.

Together with the failure details, gst-validate-launcher  will print a command-line to reproduce the test, it is also possible to use the “-t” parameter to restrict the tests to be run. Suppose you want only to run the test “validate.dash.playback.change_state_intensive.dash_exMPD_BIP_TC1”, just use:

gst-validate-launcher -t validate.dash.playback.change_state_intensive.dash_exMPD_BIP_TC1

Or, if you want to run a subset of the tests, like all for media-check:

gst-validate-launcher -t media_check

This helps speeding up debugging and avoids the need to run the full test set, which can take a long time. To make things even better, if you run it with GST_DEBUG enabled  the GStreamer logs will automatically redirect to a file that can be used to analyze the issue. A test failure output looks like this:

Launching: validate.dash.playback.change_state_intensive.dash_exMPD_BIP_TC1
 Command: 'GST_VALIDATE_SCENARIOS_PATH=/home/thiagoss/gst/head/gst-devtools/validate/data/scenarios: GST_GL_XINITTHREADS=1 GST_VALIDATE_OVERRIDE=:/home/thiagoss/gst-validate/gst-integration-testsuites/medias/defaults/online-streams-infos/dash/dash.exMPD_BIP_TC1.override DISPLAY=:0 GST_VALIDATE_SCENARIO=change_state_intensive /home/thiagoss/gst/head/gst-devtools/validate/tools/gst-validate-1.0-debug  playbin uri=http://127.0.0.1:8079/defaults/exMPD_BIP_TC1/exMPD_BIP_TC1.mpd audio-sink='fakesink sync=true' video-sink='fakesink sync=true' --set-media-info "/home/thiagoss/gst-validate/gst-integration-testsuites/medias/defaults/online-streams-infos/dash/dash.exMPD_BIP_TC1.stream_info"'
 Logs:
 - /home/thiagoss/gst-validate/logs/validate/dash/playback/change_state_intensive/dash_exMPD_BIP_TC1
 - /home/thiagoss/gst-validate/logs/validate/dash/playback/change_state_intensive/dash_exMPD_BIP_TC1.validate.logs
 - /home/thiagoss/gst-validate/logs/validate/dash/playback/change_state_intensive/dash_exMPD_BIP_TC1.gstdebug

Continuous Integration

All of the gst-validate tests and the other continuous integration routines can be checked online. Aside from these tests, builds are run for all supported platforms, and the usual unit tests are also run for new commits. When changes are merged the server will trigger new builds, and in the case of failures, developers can quickly find the commits that caused regressions.

Given the number of scenarios and use cases that GStreamer plugins have to work on, it can be easy to break one use case when developing some other feature. gst-validate tests are a helpful tool to prevent these undesirable side-effects while developing.

Author: Thiago Sousa Santos

Based in northeast Brazil, Thiago is an Open Source developer and enthusiast focused on multimedia and GStreamer. When not hacking, you can find him attempting to play an electric guitar or traveling around.