Kselftest for Linux 4.13 to Include TAP13

Linux 4.13-rc1 was released on July 15th 2017  and it includes enhancements to the Kselftest framework to support The Test Anything Protocol v13 (TAP13).

TAP13 defines a human friendly output format for tests. Kselftest is run in test rings and is widely used for Linux kernel stable release regression testing. It’s important to make it easier to identify run-to-run differences; TAP13 adaption makes it easier to understand the test results, and helps pin point differences between one run to another run of the test suite.

Credit goes to Tim Bird for recommending TAP13 as a suitable format, and to Greg KH for kick starting the work with help from Paul Elder and Alice Ferrazzi.

The first phase of the TAP13 conversion is included in Linux 4.13. Future releases will include updates to rest of the tests. The following shows membarrier test results before and after TAP 13 conversion:

Before:

membarrier MEMBARRIER_CMD_QUERY syscall available.
membarrier: MEMBARRIER_CMD_SHARED success.
membarrier: tests done!

After:

TAP version 13
ok 1 sys_membarrier available
ok 2 sys membarrier invalid command test: command = -1, flags = 0. Failed as expected
ok 3 sys membarrier MEMBARRIER_CMD_QUERY invalid flags test: flags = 1. Failed as expected
ok 4 sys membarrier MEMBARRIER_CMD_SHARED test: flags = 0
1..4

Each test result is preceded with “ok,” “not ok,” or “skip” followed by the test number to clearly indicate the result of a test.

The Kselftest API has been enhanced to add new interfaces, and existing interfaces have been updated to print output in TAP13 format to abstract TAP13 details from the individual tests.

New API:
static inline void ksft_print_header(void);
static inline void ksft_print_msg(const char *msg, ...);
static inline void ksft_test_result_pass(const char *msg, ...);
static inline void ksft_test_result_fail(const char *msg, ...);
static inline void ksft_test_result_skip(const char *msg, ...);

Existing API:
static inline void ksft_print_cnts(void);
static inline int ksft_exit_pass(void);
static inline int ksft_exit_fail(void);
static inline int ksft_exit_xfail(void);
static inline int ksft_exit_xpass(void);
static inline int ksft_exit_skip(void);

With the framework in place, work is now underway to convert existing tests to TAP13 and ensure that new tests conform to the format.

Author: Shuah Khan

Shuah contributes to multiple aspects of the Linux Kernel, and she maintains the Kernel Selftest framework.