Improving Linux Kernel Development Process Documentation

This article is part of a series on improvements being made to Linux kernel documentation:

This article will cover how the Linux kernel community handled the conversion of documentation related to the kernel development process.

Introduction

It’s not an easy task to properly describe the Linux development process. The kernel community moves at a very fast pace and produces about 6 versions per year. Thousands of people, distributed worldwide, contribute to this collective work; the development process is a live being that constantly adjusts to what best fits the people involved in the process. Additionally, since kernel development is managed per subsystems, each maintainer has their own criteria for what works best for the subsystem they take care of. To address this, the documentation provides a common ground for understanding the best practices all kernel developers should follow.

The Documentation/Development-Process Book

There are several files inside the kernel that describes the development process. In 2008, the Linux Foundation sponsored work from Jonathan Corbet to create a set of files that group the best development practices together into a development-process book. This book received additional updates in 2011 to describe the changes that happened since the original version, and reflect the practices followed for kernel 2.6.38. The kernel development practices for kernels 4.x are aproximately the same as they were during the late 2.6.x kernels; this document is not too far from the current practices.

The Essential Documentation for Kernel Developers

All kernel developers use a set of well-known documents that were written before the development-process files. These contain, among other things, the procedures to submit patches and drivers, the logic used to apply patches, how to solve conflicts, and the description of the kernel coding style. All kernel developers should be familiar with these documents, yet they’re in the middle of several other documents that include driver descriptions.

In order to get some sanity out of the chaos, I selected a group of documents that, in my opinion, contain the essence of the knowledge that all kernel developer should know, and added them into the development-process book:

  • Documentation/HOWTO is a sort of index for the other files in this set, it provides an introduction to Linux kernel development,
  • Documentation/adding-syscalls.txt describes the steps involved in adding a new system call to the Linux kernel,
  • Documentation/applying-patches.txt describes how to apply a patch to the Linux kernel using the patch tool, also describes the patch files stored at ftp.kernel.org
  • Documentation/Changes despite the name, describes the minimal requirement to build and to run the Linux kernel,
  • Documentation/CodeOfConflict describes the process to be used if someone feels personally abused, threatened, or otherwise uncomfortable with the development process,
  • Documentation/CodingStyle describes the preferred coding style for the C files inside the Linux kernel tree,
  • Documentation/email-clients.txt provides recommendations for developers to use git send-email to send e-mails with patches whenever possible. When not possible, provides information about how to setup e-mail clients to avoid mangling patches,
  • Documentation/kernel-docs.txt contains a list of external references and books related to the kernel development,
  • Documentation/magic-number.txt provides the set of magic numbers used inside the Linux kernel to help to protect kernel data structures,
  • Documentation/ManagementStyle provides a general guideline about the prefered management style for the Linux kernel,
  • Documentation/stable_api_nonsense.txt explains why the kernel internal API/ABI doesn’t have a stable kernel interface, and the advantages of such an approach,
  • Documentation/stable_kernel_rules.txt explains the process related to the kernel stable releases and the related development process,
  • Documentation/SubmitChecklist contains a list of actions that should be followed before submitting a new patch,
  • Documentation/SubmittingDrivers describes the process of submitting a new driver to the kernel,
  • Documentation/SubmittingPatches describes the process of submitting a new driver to the kernel, and
  • Documentation/volatile-considered-harmful.txt explains the proper procedure to work with data that could be modified by some other CPU or task (“volatile”).

There are other files that might be considered as part of the development process in the future.

Converting Kernel Development Files to Sphinx

As part of the effort to convert Linux documentation to Sphinx we converted the files listed above. Most of the work will be merged for kernel 4.9, but some conversions won’t be available until kernel 4.10.

Converting the Documentation/development-process files to a Sphinx book was trivial. First, Paragraph markups were added to the chapters and sections. Then, special characters that Sphinx uses were converted; specifically, emphasis required the correct markups. Finally, cross-references to other documents that were already converted to Sphinx were added.

The conversion of the other files was somewhat tricky. They’re all part of the development process, so ideally they should be located in the same directory: Documentation/development-process. However, moving several files is something that should be carefully considered, as some of them (like: CodingStyle, SubmittingDrivers, SubmittingPatches) are well known within the developer community. Moving them to any other directory would break a wide range of links that point to them, not only at the kernel tree itself, but also hyper-links all over the web. Worse than that, it will also break people’s mental references to those import documents. So, the approach should avoid physically moving them as much as possible, this results in a few choices:

  1. copy the files to the new directory, replacing the original contents with a link to the new place,
  2. create the files in the new location and use the Sphinx include directive, or
  3. use soft links at the new location.

Option 1 is the best long-term solution because the number of “orphan” files under Documentation/ will be reduced, making it easier to identify the files that haven’t been converted to Sphinx yet. Option 2 would require creating one file for every file that’s merged. Alternatively, the Sphinx include statements could be in a single RST file. However, in this case a large set of documentation files would be in the same HTML file since Sphinx generates only one output HTML file per RST source. Option 3 is easy to do and doesn’t prevent the adoption of another option in the future, so we opted to follow this direction. However, this has made it more challenging to figure out which files under Documentation/ haven’t been converted to Sphinx nor merged into a book, so we may need to review this decision in the future.

The final result of these efforts can be seen here.

Author: Mauro Carvalho Chehab

Mauro is the maintainer of the Linux kernel media and EDAC subsystems and Tizen on Yocto. He's also a major contributor to the Reliability Availability and Serviceability (RAS) subsystems.

One thought on “Improving Linux Kernel Development Process Documentation”

Comments are closed.