Kselftest for Linux 4.14 to Add Support For Test Object Relocation

In the upcoming Linux 4.14-rc3 release, work continues to develop the Kselftest TAP13 framework API and convert tests to TAP13. The new tests include Kselftest common RUN_TESTS in lib.mk that have been enhanced to print TAP13 to cover test shell scripts that won’t be able to use the Kselftest TAP13 API; this also covers test programs that aren’t converted yet.

Several fixes have been made to existing tests to prevent failure in unsupported cases as part of an ongoing work based on feedback from Kselftest stable release users that don’t want the tests to fail due to unmet dependencies, such as config options being disabled.

Additionally, a new watchdog test has been added and much needed cleanups to the existing watchdog tests have been made by Eugeniu Rosca.

A New Kselftest Use-Case

A notable change in this release is new support for the “make O=dir kselftest” use-case.  Several developers rely on this use-case to relocate objects and keep their source tree clean. It comes in handy when developers want to build kernels for several architectures and/or with different config files from the same source tree and keep it clean. Objects for each of those builds are saved in separate object directories.

I added support for this use-case by extending the existing Kselftest KBUILD_OUTPUT use-case. The majority of changes have been made to the kselftest common infrastructure. Some test make files are changed as needed to address the custom build and run_tests.

With this change the following methods to build and run Kselftest are possible.

Build all, relocate objects to /tmp/kselftest, and run tests

make O=/tmp/kselftest kselftest

or

make KBUILD_OUTPUT=/tmp/kselftest kselftest

Build TARGETS, relocate objects to /tmp/kselftest, and run tests

make O=/tmp/kselftest TARGETS="futex sync size" kselftest

or

make KBUILD_OUTPUT=/tmp/kselftest TARGETS="futex sync size" kselftest

Clean tests

make O=/tmp/kselftest kselftest-clean

or

make KBUILD_OUTPUT=/tmp/kselftest kselftest-clean

Better Test Summaries

One other important change is that messages from the Makefile are silenced during the test run, and the individual test results have been redirected to /tmp/testname files. This change makes it easier to understand the test results summary, while still being able to look at the detailed test output.

For example, running net tests after this change will output just the summary:

make --silent TARGETS=net kselftest

TAP version 13
selftests: reuseport_bpf
========================================
not ok 1..1 selftests:  reuseport_bpf [FAIL]
selftests: reuseport_bpf_cpu
========================================
not ok 1..2 selftests:  reuseport_bpf_cpu [FAIL]
selftests: reuseport_bpf_numa
========================================
not ok 1..3 selftests:  reuseport_bpf_numa [FAIL]
selftests: reuseport_dualstack
========================================
not ok 1..4 selftests:  reuseport_dualstack [FAIL]
selftests: reuseaddr_conflict
========================================
not ok 1..5 selftests:  reuseaddr_conflict [FAIL]
selftests: run_netsocktests
========================================
ok 1..6 selftests: run_netsocktests [PASS]
selftests: run_afpackettests
========================================
ok 1..7 selftests: run_afpackettests [PASS]
selftests: test_bpf.sh
========================================
not ok 1..8 selftests:  test_bpf.sh [FAIL]
selftests: netdevice.sh
========================================
ok 1..9 selftests: netdevice.sh [PASS]
selftests: rtnetlink.sh
========================================
ok 1..10 selftests: rtnetlink.sh [PASS]

The detailed test results can be found under /tmp for each test:

reuseport_bpf
reuseport_bpf_cpu
reuseport_bpf_numa
reuseport_dualstack
reuseaddr_conflict
run_netsocktests
run_afpackettests
test_bpf.sh
netdevice.sh
rtnetlink.sh

Please check the Kselftest documentation for supported use-cases: Documentation/dev tools/kselftest.rst

Author: Shuah Khan

Shuah contributes to multiple aspects of the Linux Kernel, and she maintains the Kernel Selftest framework.