How to use V4L2 Cameras on the Raspberry Pi 3 with an Upstream Kernel

A V4L2 staging driver for the Raspberry Pi (RPi) was recently merged into the Linux kernel 4.11. While this driver is currently under development, I wanted to test it and to provide help with V4L2-related issues. So, I took some time to build an upstream kernel for the Raspberry Pi 3 with V4L2 enabled. This isn’t a complex process, but it requires some tricks for it to work; this article describes the process.

Prepare an Upstream Kernel

The first step is to prepare an upstream kernel by cloning a git tree from the kernel repositories. Since the Broadcom 2835 camera driver (bcm2835-v4l2) is currently under staging, it’s best to clone the staging tree because it contains the staging/vc04_services directory with both ALSA and V4L2 drivers:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git
$ cd staging
$ git checkout staging-next

There’s an extra patch that it is required for DT to work with the bcm2835-v4l2 driver:

[PATCH] ARM: bcm2835: Add VCHIQ to the DT.

Signed-off-by: Eric Anholt <eric@anholt.net>

commit e89ae78395394148da6c0e586e902e6489e04ed1
Author: Eric Anholt 
Date:   Mon Oct 3 11:23:34 2016 -0700

    ARM: bcm2835: Add VCHIQ to the DT.
    
    Signed-off-by: Eric Anholt 

diff --git a/arch/arm/boot/dts/bcm2835-rpi.dtsi b/arch/arm/boot/dts/bcm2835-rpi.dtsi
index 38e6050035bc..1f42190e8558 100644
--- a/arch/arm/boot/dts/bcm2835-rpi.dtsi
+++ b/arch/arm/boot/dts/bcm2835-rpi.dtsi
@@ -27,6 +27,14 @@
 			firmware = <&firmware>;
 			#power-domain-cells = <1>;
 		};
+
+		vchiq {
+			compatible = "brcm,bcm2835-vchiq";
+			reg = <0x7e00b840 0xf>;
+			interrupts = <0 2>;
+			cache-line-size = <32>;
+			firmware = <&firmware>;
+		};
 	};
 };

You need to apply this to the git tree, in order for the vciq driver to work.

Prepare a Cross-Compile Make Script

While it’s possible to build the kernel directly on any RPi, building it using a cross-compiler is significantly faster. For such builds, I do this with a helper script, rather than “make,” to set the needed configuration environment.

Before being able to cross-compile, you need to install a cross compiler on your local machine; several distributions come with cross-compiler patches, or you can download an arm cross-compiler from kernel.org. In my case, I installed the toolchain at /opt/gcc-4.8.1-nolibc/arm-linux/.

I named my make script rpi_make; it not only sets the cross-compiler and defines a place to install the final output files. It has the following content:

#!/bin/bash

# Custom those vars to your needs
ROOTDIR=/devel/arm_rootdir/
CROSS_CC_PATH=/opt/gcc-4.8.1-nolibc/arm-linux/bin
CROSS_CC=arm-linux-
HOST=raspberrypi

# Handle arguments
INSTALL="no"
BOOT="no"
ARG=""
while [ "$1" != "" ]; do
	case $1 in
	rpi_install|install)
		INSTALL=yes
		;;
	boot|reboot)
		INSTALL=yes
		BOOT=yes
		;;
	*)
		ARG="$ARG $1"
	esac
	shift
done

# Add cross-cc at PATH
PATH=$CROSS_CC_PATH:$PATH

# Handle Kernel makefile rules
if [ "$ARG" == "" ]; then
	rm -rf $ROOTDIR/

	set -eu

	make CROSS_COMPILE=$CROSS_CC ARCH=arm \
	    INSTALL_PATH=$ROOTDIR \
	    INSTALL_MOD_PATH=$ROOTDIR \
	    INSTALL_FW_PATH=$ROOTDIR \
	    INSTALL_HDR_PATH=$ROOTDIR \
	    zImage modules dtbs

	make CROSS_COMPILE=$CROSS_CC ARCH=arm \
	    INSTALL_PATH=$ROOTDIR \
	    INSTALL_MOD_PATH=$ROOTDIR \
	    INSTALL_FW_PATH=$ROOTDIR \
	    INSTALL_HDR_PATH=$ROOTDIR \
	    modules_install

	# Create a tarball with the drivers
	mkdir -p $ROOTDIR/install
	(cd $ROOTDIR/lib; tar cfz $ROOTDIR/install/drivers.tgz modules/)

	# Copy Kernel and DTB at the $ROOTDIR/install/ dir
	VER=$(ls $ROOTDIR/lib/modules)
	cp ./arch/arm/boot/dts/bcm283*.dtb $ROOTDIR/install
	cp ./arch/arm/boot/zImage $ROOTDIR/install/vmlinuz-$VER
	cp .config $ROOTDIR/install/config-$VER
	cp System.map $ROOTDIR/install/System.map-$VER
else
	make CROSS_COMPILE=$CROSS_CC ARCH=arm \
	    INSTALL_PATH=$ROOTDIR \
	    INSTALL_MOD_PATH=$ROOTDIR \
	    INSTALL_FW_PATH=$ROOTDIR \
	    INSTALL_HDR_PATH=$ROOTDIR \
	    $ARG
fi

echo "Build finished."
echo "  Install: $INSTALL"
echo "  Reboot : $BOOT"

# Install at $HOST
if [ "$INSTALL" == "yes" ]; then
	DIR=$(git rev-parse --abbrev-ref HEAD)
	if [ "$(echo $DIR|grep rpi)" == "" ]; then
		DIR=upstream
	fi

	echo "Installing new drivers at $HOST:/boot/$DIR"
	scp $ROOTDIR/install/drivers.tgz $HOST:/tmp
	ssh root@$HOST "(cd /lib; tar xf /tmp/drivers.tgz)"
	ssh root@$HOST "mkdir -p /boot/$DIR"
	scp $ROOTDIR/install/*dtb $ROOTDIR/install/*-$VER root@$HOST:/boot/$DIR
fi

# Reboots $HOST
if [ "$BOOT" == "yes" ]; then
	ssh root@$HOST "reboot"
fi

Please note, you need to change the CROSS_CC_PATH var to point to the directory where you installed the cross-compiler. You may also need to change the CROSS_CC variable on this script to match the name of the cross-compiler. In my case, the cross-compiler is called /opt/gcc-4.8.1-nolibc/arm-linux/bin/arm-linux-gcc. The ROOTDIR contains the PATH where driver, firmware, headers and documentation will be installed. In this script it’s set to /devel/arm_rootdir.

Prepare a Build Script

There are a number of drivers that are needed in.config to build the kernel, and new configuration data may be needed as kernel development progresses. Also, although RPi3 supports 64-bit kernels, the userspace provided with the NOOBS distribution is 32 bits. There’s also a TODO for the bcm2835 mentioning that it should be ported to work on arm64. So, I opted to build a 32 bit kernel; this required a hack because the device tree files for the CPU used in the RPi3 (Broadcom 2837) exist only under arch/arm64 directory. So, my build procedure had to change the kernel build systems to build it for 32 bit as well.

Instead of manually handling the required steps, I opted to use a build script, called build, to set the configuration using the scripts/config script. The advantage of this approach is that it makes it easier to maintain as newer kernel releases are added.

#!/bin/bash

# arm32 multi defconfig
rpi_make multi_v7_defconfig

# Modules and generic options
enable="MODULES MODULE_UNLOAD DYNAMIC_DEBUG"
disable="LOCALVERSION_AUTO KPROBES STRICT_MODULE_RWX MODULE_FORCE_UNLOAD MODVERSIONS MODULE_SRCVERSION_ALL MODULE_SIG MODULE_COMPRESS ARM_MODULE_PLTS BPF_JIT TEST_ASYNC_DRIVER_PROBE I2C_STUB SPI_LOOPBACK_TEST INTERVAL_TREE_TEST PERCPU_TEST TEST_LKM TEST_USER_COPY TEST_BPF TEST_STATIC_KEYS CRYPTO_TEST MODULE_FORCE_LOAD DRM_TEGRA_STAGING PRISM2_USB COMEDI RTL8192U RTLLIB R8712U R8188EU RTS5208 VT6655 VT6656 ADIS16201 ADIS16203 ADIS16209 ADIS16240 AD7606 AD7780 AD7816 AD7192 AD7280 ADT7316 AD7150 AD7152 AD7746 AD9832 AD9834 ADIS16060 AD5933 SENSORS_ISL29028 ADE7753 ADE7754 ADE7758 ADE7759 ADE7854 AD2S90 AD2S1200 AD2S1210 FB_SM750 FB_XGI USB_EMXX SPEAKUP MFD_NVEC STAGING_MEDIA STAGING_BOARD LTE_GDM724X MTD_SPINAND_MT29F LNET DGNC GS_FPGABOOT COMMON_CLK_XLNX_CLKWZRD FB_TFT FSL_MC_BUS WILC1000_SDIO WILC1000_SPI MOST KS7010 GREYBUS"

# Bluetooth
enable="$enable BNEP_MC_FILTER BNEP_PROTO_FILTER BT_HCIUART_H4 BT_HCIUART_BCSP BT_HCIUART_LL BT_HCIUART_3WIRE BT_BNEP_MC_FILTER BT_BNEP_PROTO_FILTER BT_HCIUART_BCM"
module="$module BT_BNEP BT_HCIUART BT_BCM"
disable="$disable BT_HCIUART_ATH3K BT_HCIUART_INTEL BT_HCIUART_QCA BT_HCIUART_AG6XX BT_HCIUART_MRVL BT_HCIBPA10X"

# Raspberry Pi 3 drivers
enable="$enable ARCH_BCM2835 I2C_BCM2835 BCM2835_WDT SPI_BCM2835 SPI_BCM2835AUX SND_BCM2835_SOC_I2S USB_DWC2_HOST DMA_BCM2835 BCM2835_MBOX RASPBERRYPI_POWER RASPBERRYPI_FIRMWARE STAGING BCM_VIDEOCORE"
module="$module BCM2835_VCHIQ SND_BCM2835 UIO UIO_PDRV_GENIRQ FUSE_FS CUSE"
disable="$disable BCM2835_VCHIQ_SUPPORT_MEMDUMP UIO_CIF UIO_DMEM_GENIRQ UIO_AEC UIO_SERCOS3 UIO_PCI_GENERIC UIO_NETX UIO_PRUSS UIO_MF624"

# Raspberry Pi 3 serial console
enable="$enable SERIAL_8250_EXTENDED SERIAL_8250_SHARE_IRQ SERIAL_8250_BCM2835AUX SERIAL_8250_DETECT_IRQ"
disable="$disable SERIAL_8250_MANY_PORTS SERIAL_8250_RSA"

# logitech HCI
enable="$enable HID_PID USB_HIDDEV LOGITECH_FF LOGIWHEELS_FF HIDRAW"
module="$module HID USB_G_HID I2C_HID HID_LOGITECH HID_LOGITECH_DJ HID_LOGITECH_HIDPP"
disable="$disable HID_ASUS LOGIRUMBLEPAD2_FF LOGIG940_FF"

# This is currently needed for bcm2835-v4l2 driver to work
#enable="$enable VIDEO_BCM2835"
module="$module VIDEO_BCM2835"
disable="$disable DRM_VC4"

# Settings related to the media subsystem
enable="$enable MEDIA_ANALOG_TV_SUPPORT MEDIA_DIGITAL_TV_SUPPORT MEDIA_RADIO_SUPPORT MEDIA_SDR_SUPPORT MEDIA_RC_SUPPORT MEDIA_CEC_SUPPORT DVB_NET DVB_DYNAMIC_MINORS RC_DECODERS LIRC VIDEO_VIVID_CEC MEDIA_SUBDRV_AUTOSELECT VIDEO_AU0828_V4L2 VIDEO_AU0828_RC VIDEO_CX231XX_RC SMS_SIANO_RC"
module="$module RC_MAP IR_NEC_DECODER IR_RC5_DECODER IR_RC6_DECODER IR_JVC_DECODER IR_SONY_DECODER IR_SANYO_DECODER IR_SHARP_DECODER IR_MCE_KBD_DECODER IR_XMP_DECODER VIDEO_AU0828 VIDEO_CX231XX DVB_USB DVB_USB_A800 DVB_USB_DIBUSB_MB DVB_USB_DIBUSB_MC DVB_USB_DIB0700 DVB_USB_V2 DVB_USB_AF9015 DVB_USB_AF9035 DVB_USB_AZ6007 DVB_USB_MXL111SF DVB_USB_RTL28XXU DVB_USB_DVBSKY SMS_USB_DRV IR_LIRC_CODEC VIDEO_CX231XX_ALSA VIDEO_CX231XX_DVB"
disable="$disable MEDIA_CEC_DEBUG MEDIA_CONTROLLER_DVB DVB_DEMUX_SECTION_LOSS_LOG RC_DEVICES VIDEO_PVRUSB2 VIDEO_HDPVR VIDEO_USBVISION VIDEO_STK1160_COMMON VIDEO_GO7007 VIDEO_TM6000 DVB_USB_DEBUG DVB_USB_UMT_010 DVB_USB_CXUSB DVB_USB_M920X DVB_USB_DIGITV DVB_USB_VP7045 DVB_USB_VP702X DVB_USB_GP8PSK DVB_USB_NOVA_T_USB2 DVB_USB_TTUSB2 DVB_USB_DTT200U DVB_USB_OPERA1 DVB_USB_AF9005 DVB_USB_PCTV452E DVB_USB_DW2102 DVB_USB_CINERGY_T2 DVB_USB_DTV5100 DVB_USB_FRIIO DVB_USB_AZ6027 DVB_USB_TECHNISAT_USB2 DVB_USB_ANYSEE DVB_USB_AU6610 DVB_USB_CE6230 DVB_USB_EC168 DVB_USB_GL861 DVB_USB_LME2510 DVB_USB_ZD1301 DVB_TTUSB_BUDGET DVB_TTUSB_DEC DVB_B2C2_FLEXCOP_USB DVB_AS102 USB_AIRSPY USB_HACKRF USB_MSI2500 DVB_PLATFORM_DRIVERS SMS_SDIO_DRV RADIO_ADAPTERS VIDEO_IR_I2C DVB_USB_DIBUSB_MB_FAULTY"

# GSPCA driver
module="$module USB_GSPCA USB_GSPCA_BENQ USB_GSPCA_CONEX USB_GSPCA_CPIA1 USB_GSPCA_DTCS033 USB_GSPCA_ETOMS USB_GSPCA_FINEPIX USB_GSPCA_JEILINJ USB_GSPCA_JL2005BCD USB_GSPCA_KINECT USB_GSPCA_KONICA USB_GSPCA_MARS USB_GSPCA_MR97310A USB_GSPCA_NW80X USB_GSPCA_OV519 USB_GSPCA_OV534 USB_GSPCA_OV534_9 USB_GSPCA_PAC207 USB_GSPCA_PAC7302 USB_GSPCA_PAC7311 USB_GSPCA_SE401 USB_GSPCA_SN9C2028 USB_GSPCA_SN9C20X USB_GSPCA_SONIXB USB_GSPCA_SONIXJ USB_GSPCA_SPCA500 USB_GSPCA_SPCA501 USB_GSPCA_SPCA505 USB_GSPCA_SPCA506 USB_GSPCA_SPCA508 USB_GSPCA_SPCA561 USB_GSPCA_SPCA1528 USB_GSPCA_SQ905 USB_GSPCA_SQ905C USB_GSPCA_SQ930X USB_GSPCA_STK014 USB_GSPCA_STK1135 USB_GSPCA_STV0680 USB_GSPCA_SUNPLUS USB_GSPCA_T613 USB_GSPCA_TOPRO USB_GSPCA_TOUPTEK USB_GSPCA_TV8532 USB_GSPCA_VC032X USB_GSPCA_VICAM USB_GSPCA_XIRLINK_CIT USB_GSPCA_ZC3XX"

# Hack needed to build Device Tree for RPi3 on arm 32-bits
ln -sf ../../../arm64/boot/dts/broadcom/bcm2837-rpi-3-b.dts arch/arm/boot/dts/bcm2837-rpi-3-b.dts
ln -sf ../../../arm64/boot/dts/broadcom/bcm2837.dtsi arch/arm/boot/dts/bcm2837.dtsi
git checkout arch/arm/boot/dts/Makefile
sed -i "s,bcm2835-rpi-zero.dtb,bcm2835-rpi-zero.dtb bcm2837-rpi-3-b.dtb," arch/arm/boot/dts/Makefile

# Sets enable/modules/disable configuration
for i in $enable; do ./scripts/config --enable $i; done
for i in $module; do ./scripts/config --module $i; done
for i in $disable; do ./scripts/config --disable $i; done

# Sets the max number of DVB adapters
./scripts/config --set-val DVB_MAX_ADAPTERS 16

# Use rpi_make script to build the Kernel
rpi_make

Please note, the script had to disable the VC4 DRM driver and use the simplefb driver instead. This is because the firmware is currently not compatible with both the bcm2835-v4l2 driver and vc4 driver. Also, as I wanted the ability to test a bunch of V4L2 and DVB hardware on it so I’ll be enabling several media drivers.

In order to build the kernel, I simply call:

    ./build 

Install the New Kernel on the Raspberry Pi3

I used an RPi3 with a NOOBS distribution pre-installed on its micro-SD card. I used the normal procedure to install Raspbian on it, but any other distribution should do the job. In order to make it easy to copy the kernel to it, I also connected it to my local network via WiFi or Ethernet. Please note, I’ve not yet been able to get the WiFi driver to work with the upstream kernel. So, if you want to have remote access after running the upstream kernel you should opt to use the Ethernet port.

Once the RPi3 is booted, set it up to run an SSH server; this can be done by clicking on the Raspberry Pi icon in the top menu, and selecting the Raspberry Pi Configuration application from the Preferences menu.  Switch to the Interfaces tab, select SSH, and click the button.

Then, from the directory where you compiled the kernel on your local machine, you should run:

$ export ROOTDIR=/devel/arm_rootdir/
$ scp -r $ROOTDIR/install pi@raspberrypi
pi@raspberrypi's password: [type the pi password here - default is "pi"]
System.map-4.11.0-rc1+                        100% 3605KB  10.9MB/s   00:00    
vmlinuz-4.11.0-rc1+                           100% 7102KB  11.0MB/s   00:00    
bcm2835-rpi-b-plus.dtb                        100%   11KB   3.6MB/s   00:00    
config-4.11.0-rc1+                            100%  174KB   9.7MB/s   00:00    
bcm2835-rpi-b-rev2.dtb                        100%   10KB   3.9MB/s   00:00    
bcm2837-rpi-3-b.dtb                           100%   10KB   3.7MB/s   00:00    
bcm2835-rpi-a.dtb                             100%   10KB   3.8MB/s   00:00    
bcm2836-rpi-2-b.dtb                           100%   11KB   3.8MB/s   00:00    
drivers.tgz                                   100% 5339KB  11.0MB/s   00:00    
bcm2835-rpi-zero.dtb                          100%   10KB   3.6MB/s   00:00    
bcm2835-rpi-b.dtb                             100%   10KB   3.7MB/s   00:00    
bcm2835-rpi-a-plus.dtb                        100%   10KB   3.7MB/s   00:00    
$ ssh pi@raspberrypi
pi@raspberrypi's password: [type the pi password here - default is "pi"]

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Mar 15 14:47:13 2017

pi@raspberrypi:~ $ sudo su -
root@raspberrypi:~# mkdir /boot/upstream
root@raspberrypi:~# cd /boot/upstream/
root@raspberrypi:/boot/upstream# cp -r ~pi/install/* .
root@raspberrypi:/boot/upstream# cd /lib
root@raspberrypi:/lib# tar xf /boot/upstream/drivers.tgz 

Another alternative is to setup the ssh server to accept root logins and copy the ssh keys to it, with

ssh-copy-id root@raspberrypi.

If you opt for this method, you could call this small script after building the kernel:

	echo "Installing new drivers"
	scp install/drivers.tgz raspberrypi:/tmp
	ssh root@raspberrypi "(cd /lib; tar xf /tmp/drivers.tgz)"
	scp install/*dtb install/*-$VER root@raspberrypi:/boot/upstream
	ssh root@raspberrypi "reboot"

This logic is included in the rpi_make script. So, you can call:

    rpi_make rpi_install

Adjust the RPi Config to Boot the New Kernel

Changing the RPi to boot the new kernel is simple. All you need to do is to edit the /boot/config.txt file and add the following lines:

# Upstream Kernel
kernel=upstream/vmlinuz-4.11.0-rc1+
device_tree=upstream/bcm2837-rpi-3-b.dtb

# Sets for bcm2385-v4l2 driver to work
start_x=1
gpu_mem=128

# Set to use Serial console
enable_uart=1

Use Serial Console on RPi3

Raspberry Pi3 has a serial console via its GPIO connector. If you want to use it, you’ll need to have a serial to USB converter capable of working with 3.3V. For it to work, you need to wire the following pins:

Pin 1 – 3.3V power reference
Pin 6 – Ground
Pin 8 – UART TX
Pin 10 – UART RX

This image shows the RPi pin outs. Once wired up, it will look like the following image.

How to use V4L2 Cameras on the Raspberry Pi 3 with an Upstream Kernel - raspberry_pi3
Raspberry Pi 3 with camera module v2

You should also change the Kernel command line to use ttyS0, e. g. setting /boot/cmdline.txt to:

dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p7 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Use the bcm2835-v4l2 Driver

The bcm2835-v4l2 driver should be compiled as a module that’s not loaded automatically. So, to use it run the following command.

modprobe bcm2835-v4l2

You can then use your favorite V4L2 application. I tested it with qv4l2, camorama and cheese.

Current Issues

I found some problems with upstream drivers on the Raspberry Pi 3 with kernel 4.11-rc1 + staging (as found on March, 16):

  • The WiFi driver didn’t work, despite brcmfmac driver being compiled.
  • The UVC driver doesn’t work properly: after a few milliseconds it disconnects from the hardware.

That’s it. Enjoy!

Author: Mauro Carvalho Chehab

Mauro is the maintainer of the Linux kernel media and EDAC subsystems and Tizen on Yocto. He's also a major contributor to the Reliability Availability and Serviceability (RAS) subsystems.

One thought on “How to use V4L2 Cameras on the Raspberry Pi 3 with an Upstream Kernel”

  1. Thanks for the step-by-step guide – I’ll be following it myself when I get some spare cycles.

    The firmware from 18th March 2017 (3845593 at https://github.com/raspberrypi/firmware/commits/master) onwards should be compatible with the full VC4 display side, so no need for the simplefb mod.
    The AWB algorithm now checks if the QPUs (part of the V3D block) are available or not before jumping in and using them.

Comments are closed.