Taking buildroot for a spin.
Buildroot helps prepare the four fundamentals with which to spin up a system: bootloader, kernel, root filesystem, and the underpinning toolchain. It is based on Make and it provides powerful meta-configuration systems to support mixing and matching the subordinate meta-configuration systems for each of the software suites that can build all of the above. Under the hood, there are several additional layers which implement configuration management, board support definitions, packaging, general tuils and documentation. The workflow is to set up a new configuration, then pull the correct versions of source code for all of the machinery, configure and build each of those (potentially using cross compilation) and then arrange the output in the correct format to enable a raw disk image or equivalent such that either a real machine (NFS booting laptop, Raspberry Pi or your favourite embedded evaluation board) or a virtualised/emulated machine (kvm/qemu) can boot and test the fruits of your labour.
Like all embedded Linux tasks, there is a lot to learn, and much that can go wrong. But there is also a lot of help out there, and mostly, someone will also have experienced your pain, and generally, someone else will have fixed the problem.
Small steps are good. So for my first round trip exploration of buildroot, I tried to go simple, based on the tutorial from pressreset. I also added a short bash script to my bash recipes repository to be able to try this again on other machines or in the future.
For today’s attempt, I am using a fairly stock Ubuntu 18.04 but one on which I have apt installed many packages for software development, and sufficient to build the Linux kernel from source. It is running on a nice Dell XPS 15 laptop with an i7-7700HQ CPU (4 cores, 8 with hyperthreading, 32K each L1 data and instruction cache, 256K L2, 6144K L3) with 32GB of RAM and an NVME drive (listed to permit comparison of build times according to hardware spec). My host kernel is 5.3.0-59-generic.
I cloned buildroot from git and checked out tag 2020.05. Using out of source build I configured buildroot with the qemu_arm_versatile_defconfig configuration, ran “make menuconfig” to enable the build option to “Enable compiler cache” and then ran “make”. After 33 minutes of work (user 30m27, sys 3m49) completed in 7m45 clock wall time (showing good parallel use of my cores) the kernel headers stage failed with the error :
(cd /work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/build/linux-headers-5.6.15; PATH="/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/bi
n:/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/sbin:/home/astephen/.cargo/bin:/home/astephen/.local/bin:/home/astephen/bin:/home/astephen/.nimble/bin:/home
/astephen/opt_x86_64:/work/astephen/anaconda3/bin:/work/astephen/anaconda3/condabin:/home/astephen/.nimble/bin:/home/astephen/opt_x86_64:/home/astephen/.cargo/bin:/home/astephen/.local/
bin:/home/astephen/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/astephen/git-wd/snippets/bin:/work/astephen/pycharm-commu
nity-2020.1/bin:/opt/bin:/home/astephen/git-wd/snippets/bin:/work/astephen/pycharm-community-2020.1/bin:/opt/bin" /usr/bin/make -j9 ARCH=arm HOSTCC="/work/astephen/kernels-qemu-direct/b
uildroot.org/build.pressreset.net/host/bin/ccache /usr/bin/gcc" HOSTCFLAGS="" HOSTCXX="/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/bin/ccache /usr/bin/g++
" INSTALL_HDR_PATH=/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr headers_install)
INSTALL /work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/include
if ! support/scripts/check-kernel-headers.sh /work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/build /work/astephen/kernels-qemu-direct/buildroot.org/build.pressre
set.net/host/arm-buildroot-linux-uclibcgnueabi/sysroot 5.4 strict; then exit 1; fi
Incorrect selection of kernel headers: expected 5.4.x, got 5.6.x
package/pkg-generic.mk:306: recipe for target '/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/build/linux-headers-5.6.15/.stamp_staging_installed' failed
From the default config, buildroot was targeting the latest kernel headers (BR2_KERNEL_HEADERS_LATEST=y) but this seems to conflict with the version of the kernel to be built, which was set to 5.4.35. My conclusion is that I managed to break the configuration in the act of enabling the compiler cache option.
So – time to climb up the learning curve a little. It’s all about the configuration. The top level configuration in a new out of source build directory is defined in the .config file. This is created through either the interactive menu editor, or is constructed from a command line request such as “make qemu_arm_versatile_defconfig” which uses a default config subset (a “defconfig” in buildroot terms) plus wider general configuration data from which to construct a per build ".config".
To check the difference, repeat the process in a clean out of source build directory, and skip the interactive “make menuconfig” stage. Sure enough, by inadvertent clickery I had contrived to break the provided clean default config. So repeat more carefully. i.e. cp .config{,.pristine}; make menuconfig. And this time, diff .config .config.pristine gives a better check
174,177c174
< BR2_CCACHE=y
< BR2_CCACHE_DIR="$(HOME)/.buildroot-ccache"
< BR2_CCACHE_INITIAL_SETUP=""
< BR2_CCACHE_USE_BASEDIR=y
---
> # BR2_CCACHE is not set
This time – no problems, as should be the case. So – what we have built and can now use to fire up an emulated machine includes : a kernel ; a device tree blob ; an ext2 root filesystem. All of these have been compiled for Arm architecture. The qemu invocation to spin up the machine is
qemu-system-arm -nographic -M versatilepb \
-kernel output/images/zImage \
-dtb output/images/versatile-pb.dtb \
-drive file=output/images/rootfs.ext2,if=scsi \
-append "root=/dev/sda console=ttyAMA0,115200"
After the typical kernel boot lines on the console, and a short delay while the network stack optimistically waits for a non-existent eth0 interface to appear, we end up with a Buildroot login console (login as root: null password) :
EXT4-fs (sda): mounting ext2 file system using the ext4 subsystem
EXT4-fs (sda): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext2 filesystem) readonly on device 8:0.
devtmpfs: mounted
Freeing unused kernel memory: 136K
This architecture does not have kernel memory protection.
Run /sbin/init as init process
EXT4-fs (sda): re-mounted. Opts: (null)
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving random seed: random: dd: uninitialized urandom read (512 bytes read)
OK
Starting network: Waiting for interface eth0 to appear............... timeout!
run-parts: /etc/network/if-pre-up.d/wait_iface: exit status 1
FAIL
Welcome to Buildroot
buildroot login: root
#
# random: crng init done
date
Thu Jan 1 00:01:40 UTC 1970
# uname -a
Linux buildroot 5.4.35 #1 Sun Jun 14 08:35:57 BST 2020 armv5tejl GNU/Linux
Poking around, and we have a nice minimal Linux system on a very small (60MB) root file system. The emulated processor is an ARM926ej-s
# cat /proc/cpuinfo
processor : 0
model name : ARM926EJ-S rev 5 (v5l)
BogoMIPS : 806.91
Features : swp half thumb fastmult edsp java
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant : 0x0
CPU part : 0x926
CPU revision : 5
Hardware : ARM-Versatile (Device Tree Support)
Revision : 0000
Serial : 0000000000000000
Otherwise, we don’t have very much that is useful. The editor, vi, is good. No compiler. No git. No externally facing network. Almost all of the commands, not suprisingly, come courtesy of busybox.
lrwxrwxrwx 1 root root 17 Jun 14 2020 wc -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 wget -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 which -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 who -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 whoami -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 xargs -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 xxd -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 xz -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 xzcat -> ../../bin/busybox
lrwxrwxrwx 1 root root 17 Jun 14 2020 yes -> ../../bin/busybox
So – most usefully, I now have a quick to build reference kernel+rootfs that works with qemu, which I want to use to check up how I can do this but on several other boxes I have access to, and in particular over ssh.
