Docker container for git+openssl

Last November I had some trouble with a git server that wouldn’t work with the out of the box version of git that comes with Ubuntu 18.04. The issue is with the TLS implementation.

Instructions on stack overflow show how to rebuild git from scratch and use the openssl implementation.

To avoid having to repeat this on other boxes in future, I have created a Dockerfile for this and put up a copy of the docker image on hub.docker.com where it can be pulled via avstephen/ub18-git-openssl:latest

Usage is as follows, for example

docker run -it -v `pwd`:/local -w `pwd` avstephen/u18-git-openssl:latest git clone https://vcis-gitlab.f4e.europa.eu/aneto/MARTe2.git /local/m2

It’s a little clunk since the resulting files will be owned by root, but it does the job and avoids messing around with finding equivalent build solutions for other Linux distros.

Hello World Petalinux

I am in the middle of trying to follow a very nice example from Greg Anders on building some qemu based Xilinx testing projects, and have diverted to figure out a few glitches and issues with the petalinux stuff.

First off, petalinux-config is refusing to play ball and generate something when fed a perfectly respectable ZCU102 base platform downloaded pre-built from Xilinx. As google helps me find out, I am not the only one to have experienced this (albeit I am using 2019.2). My compatriot got the usual set of advice that is relevant to Xilinx tools (are you using an up to date version, on a supported platform) but ultimately this boiled down to using the tools from a network file system install directory. This smacks of fragility, and does not directly explain my case. However, I am running inside a VirtualBox VM which is one step away from a “standard” environment, and could have aspects of filesystem performance difference.

Adding a -v (verbose) option gives me more of a steer.

petalinux-config -v --get-hw-description=/home/astephen/Xilinx/Vitis/2019.2/platforms/zcu102_base/hw -p petalinux
INFO: Getting hardware description...
INFO: Rename zcu102_base.xsa to system.xsa
[INFO] generating Kconfig for project
Initialization Failed:
Can't find a usable init.tcl in the following directories: 
    /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/tps/tcl/tcl8.5 /tmp/pabuild/tcl8.5.14/lib/tcl8.5 /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/bin/unwrapped/lib/tcl8.5 /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/bin/lib/tcl8.5 /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/bin/unwrapped/library /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/bin/library /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/bin/tcl8.5.14/library /home/astephen/Xilinx/petalinux-v2019.2/tools/xsct/tcl8.5.14/library

This probably means that Tcl wasn't installed properly.

    while executing
"error $msg"
    (procedure "tclInit" line 61)
    invoked from within
"tclInit"
ERROR: Failed to generate /home/astephen/git-wd/zcu102_example/petalinux/build/misc/config/Kconfig.syshw
ERROR: Failed to Kconfig project
ERROR: Failed to generate System hardware Kconfig file.

Well, find $PETALINUX -name "init.tcl" locates an existing (though not yet demonstrably “usable” init.tcl in webtalk/tps/tcl/tcl8.5 and yet that directory is notably absent from what I infer is the TCLLIBPATH. Now file $(which petalinux-config) tells us that the tool is an ELF binary, so we cannot just go looking for where that is setup. None of the Xilinx settings files (I am also set up for Vitis and XRT here) have populated my environment with a TCLLIBPATH. So some early parsed tcl script is to blame. At this point I would normally reach for strace and look at all the tcl file openings via the system call trace. Regrettably, when I did this, the process core dumped, which was unexpected and uncalled for. For some reason "strace petalinux-config" generates an infinite set of openat(AT_FDCWD, "/proc/self/status"...) calls.

More Googling as we go down the rabbit hole. AR51582 looks promising. It hints at dark consequences when the LD_LIBRARY_PATH accrues entries which get in the way, and the sage advice is to avoid sourcing the settings files from (in this case, a .cshrc file) but unless I do source the settings I will not have petalinux-config in my PATH. Some more of the advice says this can be indicative of something having executed a non Xilinx tcl shell. Well, in my PATH I have no other tclsh.

vivado mode -tcl works perfectly well and drops me into a tclsh.

stephen@h0208-ub1804:~/git-wd/zcu102_example$ vivado -mode tcl

****** Vivado v2019.2 (64-bit)
  **** SW Build 2708876 on Wed Nov  6 21:39:14 MST 2019
  **** IP Build 2700528 on Thu Nov  7 00:09:20 MST 2019
    ** Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.

Vivado% set a 5
5
Vivado% puts $a
5

Try the same tools on another machine? Why not. Only a 7.92GB download for the installer after I remember by Xilinx US export credentials login. Dont get me wrong. I love what Xilinx tools can do, when you finally manage to balance for a moment on the finely poised pyramid of complexity.

However, I now have a good answer when my hardware colleagues make fun of the real-time C++ middleware stack I work on. Quite rightly, the point out that to do true real-time and concurrent stuff, parallelism in hardware cannot be beat. However, in the real world, when I can git clone my middleware and compile an entire nuclear fusion protection application in 20 minutes (my day job has some interest), but it takes me a week to get hello world working on a Xilinx platform, I have a justification for using a general purpose CPU with carefully crafted approaches to avoid the OS getting in the way of deterministic behaviour.

If I manage to carve out the time, I will post a follow up that explains if I ever found a resolution to this glitch before it became more pragmatic to move on to yet a different generation of Xilinx tools.

Update 23 Jun. On the host of the VM which is running Ubuntu 18.04.4 LTS directly on a real filesystem, I repeated the installation of Petalinux 2019.2, and the ZCU102 base platform and made another attempt to generate the config for the same project. In this case, there were no Tcl related errors, but simply these errors instead:

astephen@H0208:~/petalinux_project_zcu102/vitis_example$ petalinux-config -v --get-hw-description=../zcu102_base/hw/ -p petalinux
INFO: Getting hardware description...
INFO: Rename zcu102_base.xsa to system.xsa
[INFO] generating Kconfig for project
INFO: [Hsi 55-2053] elapsed time for repository (/home/astephen/Xilinx/tools/xsct/data/embeddedsw) loading 0 seconds
hsi::open_hw_design: Time (s): cpu = 00:00:06 ; elapsed = 00:00:06 . Memory (MB): peak = 811.488 ; gain = 148.066 ; free physical = 10719 ; free virtual = 14436
[INFO] menuconfig project
ERROR: Failed to menu config project component 
ERROR: Failed to config project.
ERROR: Get hw description Failed!.

Working with Xilinx tools is like working for Edison. I have not failed. I’ve just found 10,000 ways that won’t work.

Update 2020-06-23. From another fine host, this time with Petalinux 2020.1 installed, another repeat of the same action. Finally, this time, we get to a menuconfig option for the system configuration (when run interactively). Taking the defaults from this, the tool proceeds to function as expected, up to the “generating workspace directory” which failed.

$>petalinux-config --get-hw-description=./platforms/zcu102_base/hw/ -p petalinux
WARNING: Your PetaLinux project was last modified by PetaLinux SDK version "2019.2",
WARNING: however, you are using PetaLinux SDK version "2020.1".
Please input "y" to continue. Otherwise it will exit![n]y
INFO: sourcing build tools
INFO: Getting hardware description...
INFO: Rename zcu102_base.xsa to system.xsa
[INFO] generating Kconfig for project
[INFO] menuconfig project
configuration written to /media/2TB/workspace/astephen/git-wd/petalinux_zcu102_example/petalinux/project-spec/configs/config

*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

[INFO] extracting yocto SDK to components/yocto
[INFO] sourcing build environment
[INFO] generating u-boot configuration files, This will be deprecated in upcoming releases
[INFO] generating kernel configuration files, This will be deprecated in upcoming releases
[INFO] generating kconfig for Rootfs
[INFO] silentconfig rootfs
yes: standard output: Broken pipe
[INFO] generating plnxtool conf
[INFO] generating user layers
[INFO] generating workspace directory
ERROR: Failed to create workspace directory
ERROR: Failed to config project.
ERROR: Get hw description Failed!.

Repeating, but with the verbose flag and the –silentconfig option…

auto-login (auto-login) [N/y/?] 
*
* user packages 
*
opencl-clhpp-dev (opencl-clhpp-dev) [Y/n/?] 
opencl-headers-dev (opencl-headers-dev) [Y/n/?] 
xrt (xrt) [Y/n/?] 
xrt-dev (xrt-dev) [Y/n/?] 
zocl (zocl) [Y/n/?] 
*
* PetaLinux RootFS Settings
*
Root password (ROOTFS_ROOT_PASSWD) [********] 
Add Extra Users (ADD_EXTRA_USERS) [] 
#
# configuration written to /media/2TB/workspace/astephen/git-wd/petalinux_zcu102_example/petalinux/project-spec/configs/rootfs_config
#
[INFO] generating plnxtool conf
[INFO] generating user layers
[INFO] generating workspace directory
ERROR: Failed to create workspace directory
ERROR: Failed to config project.
ERROR: Get hw description Failed!.
astephen@ukaea-fpga petalinux_zcu102_example master

Ubuntu 18.04 or Not Ubuntu 18.04

So during my mammoth Xilinx install exercise yesterday, I learned something about Ubuntu distributions that I had not known.

Xilinx Vitis 2019.2 release states that it is compatible with Ubuntu 18.04.2 LTS. I never really knew what the final minor version suffix indicated (the .2) and so I was sloppy in preparing a virtual box VM onto which to install the OS and downloaded the generic .iso from Ubuntu for “18.04” which gave me 18.04.4.

Several hours later, this came back to bite me when the XRT Xilinx run time install failed, because one of the kernel modules would not build, and also the pyopencl installation fell over. Looking at the first problem revealed that 18.04.4 provides a 5.x kernel and the kernel modules required a 4.x kernel. The pyopencl problem was not so clear cut but revolved about the usual Python2/Python3 issue (gee, thanks python guys for giving me some nostalgia about the bad old Perl4/Perl5 days).

Since the Vitis install involves a 25GB download, I was loathe to throw away the work so far, but nursing a non supported OS to support as complex a software stack as Xilinx could just be banging my head against a wall. On the other hand, a good opportunity to learn a few things, so what might we try?

Plan A was to consider downgrading the installed system. Numerous articles explain that this is possible by editing /etc/apt/sources.list to point at the desired version, then “pinning” the packages by tweaking the pin priority in /etc/apt/preferences and running an “apt update; apt upgrade; apt dist-upgrade” sequence.

However, most articles also caution YMMV and that if you have post installed other packages (and XRT pulls in no less than 136 packages) then the likely messed up root file system could be pretty flaky.

Plan B : no need to downgrade everything, since the problem is clearly with the kernel. Just back out, or install an older kernel and retry. The first challenge is to work out exactly which Ubuntu kernel was used in 18.04.2. In principle documented, but in practice I decided just to download the 18.04.2 install iso and make a second Virtual Box VM. Bad mistake in doing this was to check the “download other stuff while installing tickbox” because apparently by having done so I got a free 18.04.2 kernel upgrade to 5.3.0-59-generic and not the 4.x kernel I expected. Oddly, I still got a post install”would you like to download updates now” option. Just to prove I´m not dreaming I checked a 2019 article and sure enough (unless that article history gets rewritten which is not out of the question) it explained that 18.04.2 shipped with a 4.18 kernel.

Some more googling located what sounds like a handy Ubuntu Kernel Update Utility (UKUU)utility for my problem.

$ sudo add-apt-repository ppa:teejee2008/ppa
$ sudo apt-get install ukuu

But this failed. WTF? Checking Tony George´s ppa repo finds links to his ukuu utility for 18.04.1. This minor version on the Ubuntu releases is really starting to grate. However, while browsing the launchpad pages, I start to get a feel for the support that is present for managing “recipes” for packaging code for Ubuntu and this itself could lead to more resources for a forthcoming blog article on the packaging wars.

OK. I don´t really want to become a debian packaging ninja this morning, so we can install an older kernel without some flash GUI support, surely? Perhaps https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18.20/ Following the stackoverflow instructions reveals a couple of missing tips. Firstly, the modules are required in addition to the kernel image and they must be installed first. Second, after doing the installation, to make it possible to select the new kernel from the grub bootloader where a key tip is to edit /etc/default/grub and comment out the line that hides the timeout so you can select the correct kernel. OK. I now have 4.18.20 on my test 18.4.2 system and so I can replay this solution to backport my kernel in 18.04.4.

A couple of other possibilities were at the back of my mind had this approach failed.

  • Create a new VM with 18.4.2, then mount the VDI virtual disk from the 18.4.4 and copy over the directory where the Xilinx installations had been done, or mount and use the VDI disk (a good future exercise will be to check how to do this anyway)
  • Try an 18.4.2 install over the existing VDI disk, hoping that the installer would offer an option to install alongside and handle the housekeeping of partition wrangling to just make this work,

VOTTO II: HDF5 install

Part II in my series on variations on the theme of HDF5. Getting some demonstration C examples compiling and running on Ubuntu 18.04 but in contrast to part I we build the HDF libraries from scratch.

https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.0/src/hdf5-1.12.0.tar.gz

From release_docs/INSTALL we discover that the dependencies are on :

  1. Zlib : if zlib-1.1.2 or later is found, HDF5 will use it. Otherwise the compression stub will default to no compression.
  2. Szip (optional) provides another option to support Szip compression filter.
  3. MPI and MPI-IO. If the parallel version of the librariy is to be built then MPI and MPI-IO are required. Otherwise, only the serial version of HDF5 can be built.

The installation steps are :

  1. Download the source, either via git or wget for a released snapshot.
  2. If downloaded, extract and decompress the tar.gz archive.
  3. Out of source builds are always to be recommended and are supported with Make. Thus mkdir build; cd build; ../hdf5-X.Y.Z/configure options.

To summarise, the following script will work to build from scratch.

wget  https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.0/src/hdf5-1.12.0.tar.gz
tar zxvf hdf5-1.12.0.tar.gz
mkdir build-hdf5-1.12.0-make
cd build-hdf5-1.12.0-make
../hdf5-1-12.0/configure --prefix=/opt/hdf5-1.12.0-make
make
sudo make install

Now to use this installation, it is necessary to add the target install directory to PATH. i.e. PATH=/opt/hdf5-1.12.0/bin:$PATH.

To see a full working example, I have created github and docker repos with some test HDF5 code, a docker image where all of this has been done, and a github repo of the Dockerfile from which the image was made. See

Provenance

RepoCommit / DigestComment
hdf5-quickstartaca4cd0Commit
u18-hdf5-scratchsha256:e02bc5d93332df070c2e3dec49
ba5b6a0930afee867bcced212d7de04bab7c6d
Image digest
my-dockers658f051Commit
Provenance of the versions used in this article.

Related Articles

  • Series Overview
  • Part I
  • Part III: CPP bindings.
  • Part IV: Cross compiling for Arm
  • Part V: Arm on a ZynqMP under Qemu
  • Part VI: Yocto-ized Build Recipe
  • Part VII: Petalinux 2019.2 Integration

VOTTO HDF5 I : Ubuntu

Part I in my series on variations on the theme of HDF5. Getting some demonstration C examples compiling and running on Ubuntu 18.04

One route to installing the HDF5 tools is via the Anaconda python package manager. By default, installing the HDF5 module will give the python bindings. This includes the h5cc compiler script front end, but not by default the actual gcc toolchain configured for HDF5 compatibility. To be able to compile say the HDF group examples the following install is required.

conda install gxx_linux-64

Since this kind of task is always a bit fiddly, and since I work on a very large range of topics, I like to create github repos with sample code and dockerhub docker containers to latch my results. Where I can, I make these public. So – the repos below contain my HDF5 example code, a docker hub prebuilt image which can run it, and finally a repo where I keep my Dockerfiles since these are sometimes more useful than runnable docker container images from Docker hub.

A few notes as to which Ubuntu 18.04 packages[1] were found to be useful for HDF5 work :

  1. hdf5-tools : provides about 15 utilities for working with HDF5 files.
  2. libhdf5-dev : development headers and libraries. This adds the h5cc front end for compiling C HDF5 programs. Interestingly the dependencies in apt are not sufficient to cause this to pull in gcc.
  3. build-essential : always disappointing to me that stock Ubuntu requires custom installation of gcc. A linux box without gcc is totally pointless. It ought to be illegal.
  4. vim : Again, almost more essential than even a compiler. Should be even more illegal to have Linux without vim installed.
  5. cmake : Not strictly necessary, but important to placate my good friend Rashed Sarwar.

This is sufficient to build and run the C based examples. To see this for yourself, try this :

docker run -it avstephen/u18-hdf5:1.0
cd hdf5-quickstart/examples
make all
ls

Provenance

The tagged docker has the provenance of the github examples, but for reference, this was made from commit aca4cd0 of hdf5-quickstart and commit 4bdd100 of my-dockers and the u18-hdf5/Dockerfile.

The digest of avstephen/u18-hdf5:1.0 is sha256:b7fc21af122b875e69cb71b03bdb9dc0017590d8f4c03fd743643915099766c9

Related Articles

  • Series Overview
  • Part II : HDF5 from Scratch (Make version)
  • Part II(b) HDF5 from Scratch (cmake version)
  • Part III: CPP bindings.
  • Part IV: Cross compiling for Arm
  • Part V: Arm on a ZynqMP under Qemu
  • Part VI: Yocto-ized Build Recipe
  • Part VII: Petalinux 2019.2 Integration

Footnotes

[1] I know that 20.04 LTS is available, but I suspect it will be another year or so before the typical Ubuntu user upgrades, so I prefer to base tutorials on a slightly older version of Ubuntu. I am running with Ubuntu 14.04, 16.04, 18.04, 20.04, Centos 7, RHEL6, RHEL7, Scientific Linux 7.4 and Solaris. And for more interesting projects, forget distros, I build custom kernels and root filesystems for fun.

Variations on the theme of HDF5

HDF5 is a powerful toolkit for managing datasets, widely exploited in scientific research. It is developed and maintained by the HDF5 group.

I am working on a machine learning project where we intend to use HDF5 to exchange trained model weights between a python based tensorflow training project and FPGA based accelerated inference deployment engines.

So, I need to learn how to use HDF5 for application development, from python, C and C++. I also need to learn how to install it, in environments that range from the trivial (x86-64 Linux system with package manager) to the more complex (Yocto/Petalinux build for Xilinx Ultrascale+ family Arm Cortex A53 processor with a very embedded setup).

Rather than one uberlong monolithic post, I will break down the work into easily managed steps, and this top level post will act as a navigation starting point.

DateTask
2020-06-19Compile and run basic HDF5 examples on Ubuntu.
Sub tasks in the HDF5 on ZynqMPSoC Series

Fundamentals of Linux : I

This is the first in a series of many posts as I undertake self guided training in the skills of building Linux systems from scratch. i.e. Downloading and compiling the kernel sources, building and using cross-compiling toolchains, curating and generating root file systems, preparing and understanding device tree blobs and other board support related matters, bootloaders and the tools which bootstrap a machine startup. Along the way, I will experiment with doing some of these things from first principles, and more generally, using the very powerful toolkits which support these workflows, such as buildroot and yocto.

Fundamentals of Linux : II

Taking buildroot for a spin.

Buildroot helps prepare the four fundamentals with which to spin up a system: bootloader, kernel, root filesystem, and the underpinning toolchain. It is based on Make and it provides powerful meta-configuration systems to support mixing and matching the subordinate meta-configuration systems for each of the software suites that can build all of the above. Under the hood, there are several additional layers which implement configuration management, board support definitions, packaging, general tuils and documentation. The workflow is to set up a new configuration, then pull the correct versions of source code for all of the machinery, configure and build each of those (potentially using cross compilation) and then arrange the output in the correct format to enable a raw disk image or equivalent such that either a real machine (NFS booting laptop, Raspberry Pi or your favourite embedded evaluation board) or a virtualised/emulated machine (kvm/qemu) can boot and test the fruits of your labour.

Like all embedded Linux tasks, there is a lot to learn, and much that can go wrong. But there is also a lot of help out there, and mostly, someone will also have experienced your pain, and generally, someone else will have fixed the problem.

Small steps are good. So for my first round trip exploration of buildroot, I tried to go simple, based on the tutorial from pressreset. I also added a short bash script to my bash recipes repository to be able to try this again on other machines or in the future.

For today’s attempt, I am using a fairly stock Ubuntu 18.04 but one on which I have apt installed many packages for software development, and sufficient to build the Linux kernel from source. It is running on a nice Dell XPS 15 laptop with an i7-7700HQ CPU (4 cores, 8 with hyperthreading, 32K each L1 data and instruction cache, 256K L2, 6144K L3) with 32GB of RAM and an NVME drive (listed to permit comparison of build times according to hardware spec). My host kernel is 5.3.0-59-generic.

I cloned buildroot from git and checked out tag 2020.05. Using out of source build I configured buildroot with the qemu_arm_versatile_defconfig configuration, ran “make menuconfig” to enable the build option to “Enable compiler cache” and then ran “make”. After 33 minutes of work (user 30m27, sys 3m49) completed in 7m45 clock wall time (showing good parallel use of my cores) the kernel headers stage failed with the error :

(cd /work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/build/linux-headers-5.6.15; PATH="/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/bi
n:/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/sbin:/home/astephen/.cargo/bin:/home/astephen/.local/bin:/home/astephen/bin:/home/astephen/.nimble/bin:/home
/astephen/opt_x86_64:/work/astephen/anaconda3/bin:/work/astephen/anaconda3/condabin:/home/astephen/.nimble/bin:/home/astephen/opt_x86_64:/home/astephen/.cargo/bin:/home/astephen/.local/
bin:/home/astephen/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/astephen/git-wd/snippets/bin:/work/astephen/pycharm-commu
nity-2020.1/bin:/opt/bin:/home/astephen/git-wd/snippets/bin:/work/astephen/pycharm-community-2020.1/bin:/opt/bin" /usr/bin/make -j9 ARCH=arm HOSTCC="/work/astephen/kernels-qemu-direct/b
uildroot.org/build.pressreset.net/host/bin/ccache /usr/bin/gcc" HOSTCFLAGS="" HOSTCXX="/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/bin/ccache /usr/bin/g++
" INSTALL_HDR_PATH=/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr headers_install)                             
  INSTALL /work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/include                                               
if ! support/scripts/check-kernel-headers.sh  /work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/build  /work/astephen/kernels-qemu-direct/buildroot.org/build.pressre
set.net/host/arm-buildroot-linux-uclibcgnueabi/sysroot  5.4 strict; then exit 1; fi                                                                                                      
Incorrect selection of kernel headers: expected 5.4.x, got 5.6.x                                                                                                                         
package/pkg-generic.mk:306: recipe for target '/work/astephen/kernels-qemu-direct/buildroot.org/build.pressreset.net/build/linux-headers-5.6.15/.stamp_staging_installed' failed 

From the default config, buildroot was targeting the latest kernel headers (BR2_KERNEL_HEADERS_LATEST=y) but this seems to conflict with the version of the kernel to be built, which was set to 5.4.35. My conclusion is that I managed to break the configuration in the act of enabling the compiler cache option.

So – time to climb up the learning curve a little. It’s all about the configuration. The top level configuration in a new out of source build directory is defined in the .config file. This is created through either the interactive menu editor, or is constructed from a command line request such as “make qemu_arm_versatile_defconfig” which uses a default config subset (a “defconfig” in buildroot terms) plus wider general configuration data from which to construct a per build ".config".

To check the difference, repeat the process in a clean out of source build directory, and skip the interactive “make menuconfig” stage. Sure enough, by inadvertent clickery I had contrived to break the provided clean default config. So repeat more carefully. i.e. cp .config{,.pristine}; make menuconfig. And this time, diff .config .config.pristine gives a better check

174,177c174
< BR2_CCACHE=y
< BR2_CCACHE_DIR="$(HOME)/.buildroot-ccache"
< BR2_CCACHE_INITIAL_SETUP=""
< BR2_CCACHE_USE_BASEDIR=y
---
> # BR2_CCACHE is not set

This time – no problems, as should be the case. So – what we have built and can now use to fire up an emulated machine includes : a kernel ; a device tree blob ; an ext2 root filesystem. All of these have been compiled for Arm architecture. The qemu invocation to spin up the machine is

qemu-system-arm -nographic -M versatilepb \
    -kernel output/images/zImage \
    -dtb output/images/versatile-pb.dtb \
    -drive file=output/images/rootfs.ext2,if=scsi \
    -append "root=/dev/sda console=ttyAMA0,115200"

After the typical kernel boot lines on the console, and a short delay while the network stack optimistically waits for a non-existent eth0 interface to appear, we end up with a Buildroot login console (login as root: null password) :

EXT4-fs (sda): mounting ext2 file system using the ext4 subsystem
EXT4-fs (sda): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext2 filesystem) readonly on device 8:0.
devtmpfs: mounted
Freeing unused kernel memory: 136K
This architecture does not have kernel memory protection.
Run /sbin/init as init process
EXT4-fs (sda): re-mounted. Opts: (null)
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving random seed: random: dd: uninitialized urandom read (512 bytes read)
OK
Starting network: Waiting for interface eth0 to appear............... timeout!
run-parts: /etc/network/if-pre-up.d/wait_iface: exit status 1
FAIL

Welcome to Buildroot
buildroot login: root
# 
# random: crng init done
date
Thu Jan  1 00:01:40 UTC 1970
# uname -a
Linux buildroot 5.4.35 #1 Sun Jun 14 08:35:57 BST 2020 armv5tejl GNU/Linux

Poking around, and we have a nice minimal Linux system on a very small (60MB) root file system. The emulated processor is an ARM926ej-s

# cat /proc/cpuinfo 
processor       : 0
model name      : ARM926EJ-S rev 5 (v5l)
BogoMIPS        : 806.91
Features        : swp half thumb fastmult edsp java 
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant     : 0x0
CPU part        : 0x926
CPU revision    : 5

Hardware        : ARM-Versatile (Device Tree Support)
Revision        : 0000
Serial          : 0000000000000000

Otherwise, we don’t have very much that is useful. The editor, vi, is good. No compiler. No git. No externally facing network. Almost all of the commands, not suprisingly, come courtesy of busybox.

lrwxrwxrwx    1 root     root            17 Jun 14  2020 wc -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 wget -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 which -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 who -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 whoami -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 xargs -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 xxd -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 xz -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 xzcat -> ../../bin/busybox
lrwxrwxrwx    1 root     root            17 Jun 14  2020 yes -> ../../bin/busybox

So – most usefully, I now have a quick to build reference kernel+rootfs that works with qemu, which I want to use to check up how I can do this but on several other boxes I have access to, and in particular over ssh.

Git version hell.

It’s deeply ironic. I’m working with arguably the best tool ever developed on the planet to help large groups of people work effectively together on software projects by delivering sophisticated, powerful and effective version control : git. And yet, not only does it have rough edges in terms of the UI (particularly from the cli point of view) but it even has a lack of consistency across different versions of itself.

To be fair, most of my gripes about the UI have been addressed over time, and it will always be true that powerful software has complexity that can be difficult to fully understand without putting in the effort. And yet I don’t make the same complaints to my colleagues about vim or emacs that I do about git. I suppose this is because the main function of git, while complex at low levels, is conceptually trivial. Keep track of my project and help my team get the right files at the right time.

I’m currently working on a real-time control application which sits on top of a very sophisticated real-time C++ framework middleware. There are around 80 distinct software modules, which originally were organised in one monolithic git repository. However, some of the modules represent core functionality which is mandatory for all of the applications that leverage the framework. Some are optional per application, and indeed optional per supported OS platform per application. Finally, some actually form the application and hence are mandatory for that application and prohibited (or at least useless) for other applications. The lifecycle and dependencies for these 80 modules are complex. We recently decided to migrate the repository to a submodule based model with a superproject that aggegrates the code.

So, over the last month or two, there has been a lot of work. We’ve also adopted cmake over make as the build toolchain, and during the transition, managing releases of products moving from the monolithic to the submodule module has been challenging. Git has generally been superb and has supported some worfklows I would not have believed possible (automatically repeat the rewriting of history that splitting a monolithic project in two requires, and then merging the original history rewrite, all the work that went on the in meantime and a second rewrite of history from a later period of time). Indeed, I get the feeling that the English language lacks sufficient depth of expression in terms of tenses and moods to be able to discuss some of this temporal evolution – we need a supersubjunctive.

I’m now on the last lap, tidying up, automating builds of different branches that weave themselves across the modules and sub-projects. I’m doing this work from many machines, with a range of Linux distributions including Ubuntus 16.04/18.04/20.04, Scientific Linux 7.4, Centos 7 and RHEL6,7. And of course each of these has a different vintage of git. Mostly I had never noticed.

The first irritation came yesterday as I worked on a wrapper script that can iterate over sets of several tens of git submodules to manage branch and tag handling in batch. I needed a few commands to report on the commits in useful ways to help build trust in the team. One of the fun parts of git is the evolution of trust. Initially it seems like black magic that does amazing things. Then you learn to break it in simple ways and start to be nervous. Then you understand the underlying model and regain trust that it will never lose anything. Detached head has no fear and you recklessly rewrite working trees in amazing ways. Then you find that the reflog has a 90 day garbage collection limit and that hence some meta history can be lost. Then you rewrite history and realise that while the file set changes are generally inviolate, the chain of SHA hashes can change when you rebase, or rewrite history, and then you start to lean on the crutch of commit dates and descriptions to believe that the work is still there. Then you rewrite history totally and realise that you do have the power to do anything. And with great power comes great responsibility.

Anyway – it was annoying that as I wrote my development support script, I found that using some log formatting strings (%gD) was only valid in later versions of git. I could live with that, though spotting that when I do a Google search for instructions means some recommendations may not work for some versions of git was a bit of a warning flag. And the drop down selector to look at the documentation version per git version on the main is another indication.

This version related selection has a very typical version control pattern. Why would I need different versions of documentation? Well, firstly, and hopefully primarily, the documentation can have a lifecycle in which it gets better and better. It becomes clearer. It has better presentation. But secondly, it can refer to functional differences between different versions of the tools.

The positive way to look at this is the mature recognition that learning git is a bit like learning science, and just as the simple lessons about Newtonian mechanics get overturned by the deeper understanding of Quantum mechanics, the thin veneer of git add/commit/push cycles needs to be supplanted by a better appreciation of what is under the hood.

And hey – it means I get to write the gazillionth plus one article on how something in git tripped me up and I can explain it to others. And the number of such articles is the greatest indication that a much better job on the UI and teaching how to use the tool might have saved the world a lot of work. One point of view would say – don’t blame the tool for offering massive flexibility which can be interpreted, used and explained in myriad ways. And the dolphins would whistle.

Patterns and AntiPatterns : Culture Clash

An hour or so lost due to somewhat obfuscated difficulties for Make when presented with filename wildcards that expand to include pathnames that include colons.

When writing simple, literal Makefile rules it is of course usual to see the colon character as the separator between the targets and the dependencies.

In the course of working towards fully automated testing, I decided to introduce a timestamped folder convention. And the format for the timestamp included %Y-%m-%d:%H:%M:%S. Unfortunately, in the same change, I also made minor updates to a Makefile that referenced files in this path. And then I buried all of this way down the full build stack.

Out popped an unexpected error.

Makefile:21: *** target pattern contains no `%’. Stop

It was entirely unobvious to me that the cause of this was the new colon containing directory. Eventually bisecting the changes reduced this to the only possibility and then a Google search indicated that it is a not uncommon problem.

Most ugly? The Make maintainer recognises that this is an issue but has no solution as fixing this problem would break existing Makefiles.

I’m surprised I’ve never run into this in 30 years of working with Make.