The Raspberry Pi is a great little computer for makers. However, it lacks the performance to compile big software packages in an acceptable timeframe. So I set out to create a fast and easy-to use docker-based cross-compiler for the Pi, which runs on much more powerful machines like for example a VPS.
Continuous Integration
This server does not only host the WordPress blog you are currently reading, but also, among other things, a full CI stack using Docker (https://www.docker.com/):
- Source code hosting: Gitea (https://gitea.io/)
- Integration server: Drone (https://drone.io/)
- Asset storage: Docker registry (https://hub.docker.com/_/registry)
- Asset management: Portus (http://port.us.org/), running a fork (https://github.com/StarGate01/Portus)
Now, I want to be able to write Dockerfiles, push them into a Git repository, and have them built into a Docker image which supports multiple architectures. Then I just pull the image on to my Raspberry Pi and run the program.
Docker BuildX and QEMU-BinFMT
But how do we run binaries for a different architecture on our server? That is where QEMU (https://www.qemu.org/), an open-source machine emulator comes in. Because most (cheap) VPS do not allow hardware virtualisation, we run QEMU in the user space (https://wiki.debian.org/QemuUserEmulation). QEMU provides the needed modules for binfmt_misc (https://en.wikipedia.org/wiki/Binfmt_misc), a linux kernel function to execute arbitrary binaries via user space modules.
Now we could build our docker image using the good old docker build
command and then assemble a multi-arch image using docker manifest
, but that is quite a lot of manual work. Instead, we use the docker buildx
experimental feature, which is able to build for a given range of architectures.
The BuildX Docker command is still an experimental feature, so you either have to install it from a binary, or use a recent Docker version and enable experimental features.
Conveniently, there is a docker container which installs the needed QEMU modules:
docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
You should check for new versions of that image if you want to follow along on this adventure. After having installed the modules into the host, we can create a builder which supports the new architectures, and bootstrap it:
docker buildx create --use --name crosscomp && docker buildx inspect --bootstrap
The driver then tells us its capabilities:
Name: crosscomp Driver: docker-container Nodes: Name: crosscomp0 Endpoint: unix:///var/run/docker.sock Status: running Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
To create a multi image that runs on PC and as well on the Raspberry PI, we only need to add linux/arm/v7
, and maybe linux/arm64
:
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 ...
To enable multiple parallel builds on the CI server using multiple independent runners, we give each runner an unique name based on the current container ID:
export BUILDER_ID="crosscomp-$(cat /proc/self/cgroup | head -1 | cut -d '/' -f 3)" && docker buildx create --use --name $BUILDER_ID && ... && docker buildx rm $BUILDER_ID
Integration Server and Registry Setup
The CI server must be able to bind to the docker socket of the host, in oder to spawn the BuildKit (https://github.com/moby/buildkit) container, on which BuildX is based. Apart from that, the compiler needs privileged
permissions to install the QEMU modules. These features are not commonly available on cloud services, so I host my own pipeline.
I also run a private Docker registry. This registry has to support the docker protocol v2, and especially docker manifest lists for the multi-arch images.
Docker Layer Caching
The CI server runs all tasks inside a new context, to guarantee ephemeral builds. But it would be nice to cache the Docker layers to reduce build times on subsequent builds! Because we do not have access (nor should we) to the host layer cache, we use a second repository location for our cache. This cache holds all layers, while the public repository location only holds the last stage from the Dockerfile. This enables us to cache private build stages without exposing them in the same public location.
BuildX provides the --from-cache
and --to-cache
flags to control this mechanic:
--cache-to=type=registry,ref=registry.chrz.de/cache/hello-ci,mode=max --cache-from=type=registry,ref=registry.chrz.de/cache/hello-ci
The mode=max
argument tells BuildX to cache all stages, not only the last public one.
A Simple Example
To demonstrate this compiler, I wrote a simple example container containing a native binary which has to be compiled.
The Binary Payload
#include <iostream>
#include <sys/utsname.h>
using namespace std;
struct utsname unameData;
int main()
{
cout << "Hello from binary!" << endl;
uname(&unameData);
printf("Running on %s, %s\n", unameData.sysname, unameData.machine);
return 0;
}
This simple C++ program just prints hello and the architecture, and then exits.
Container Configuration
Then, I wrote a two-stage Dockerfile, with one stage for build and one stage for execution, to emulate real-world usecases:
# build stage
FROM ubuntu:18.04 AS build
ENV DEBIAN_FRONTEND=noninteractive TZ=Europe/Berlin
RUN apt-get update && \
apt-get -y install --no-install-recommends g++ && \
rm -rf /var/lib/apt/lists/*
COPY main.cpp /app/main.cpp
WORKDIR /app
RUN g++ -o main main.cpp
# run stage
FROM ubuntu:18.04 AS run
ENV DEBIAN_FRONTEND=noninteractive TZ=Europe/Berlin
RUN apt-get update && \
apt-get -y install --no-install-recommends file && \
rm -rf /var/lib/apt/lists/*
COPY --from=build /app/main /app/main
COPY run.sh /app/run.sh
RUN chmod +x /app/run.sh
CMD ["/app/run.sh"]
As you can see, the Dockerfile sets up a simple build environment and then copies the compiled binary into an execution environment. The entrypoint shell script outputs some meta information of the binary:
#!/bin/bash
echo "Hello from entrypoint!"
file /app/main
ldd /app/main
/app/main
Integration Server Configuration
The CI configuration for Drone basically consists of the steps discussed above:
kind: pipeline
name: default
volumes:
- name: docker_socket
host:
path: /var/run/docker.sock
steps:
- name: build
image: alexviscreanu/buildx
commands:
- docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
- export BUILDER_ID="crosscomp-$(cat /proc/self/cgroup | head -1 | cut -d '/' -f 3)"
- docker buildx create --use --name $BUILDER_ID --driver-opt image=stargate01/buildkit
- docker buildx inspect --bootstrap
- docker login --username $DOCKER_USERNAME --password $DOCKER_PASSWORD registry.chrz.de
- docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 --output=type=image,push=true --progress tty -t registry.chrz.de/public/hello-ci .
- docker buildx rm $BUILDER_ID
volumes:
- name: docker_socket
path: /var/run/docker.sock
environment:
DOCKER_USERNAME:
from_secret: docker_username
DOCKER_PASSWORD:
from_secret: docker_password
It uses the image alexviscreanu/buildx
, which is just Docker with BuildX installed. It also instructs the server to push the compiled image to my registry. Registry login data is provided via secrets by the CI server.
Note that the repository has to be configured as “trusted” inthe CI server in order to mount the Docker socket and run in privileged mode.
Testing the Image
To run the image, I use the command docker run registry.chrz.de/public/hello-ci
, which downloads and executes the image.
When I run the image on my x64 PC, the output looks like this:
Hello from entrypoint! /app/main: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=f354df5ffd19fa277ef28d97fe4e8760ee630302, not stripped linux-vdso.so.1 (0x00007fff1676b000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1819ef2000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1819b01000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1819763000) /lib64/ld-linux-x86-64.so.2 (0x00007f181a47d000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f181954b000) Hello from binary! Running on Linux, x86_64
Now, on my Raspberry Pi, using the exact same command and meta-image, Docker knows to download the sub-image for the current architecture. The output then looks like this:
Hello from entrypoint! /app/main: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib, for GNU/Linux 3.2.0, BuildID[sha1]=ef78a3c9e7f58a215bafa2839394746dfd2f5013, not stripped linux-vdso.so.1 (0x7ed38000) libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x76e61000) libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x76e38000) libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76d40000) libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76cbf000) /lib/ld-linux-armhf.so.3 (0x76f86000) Hello from binary! Running on Linux, armv7l
Success! Not only does it run, but it also shows that the QEMU cross-compiler did in fact create valid ARM executables.
Download Sources
I made the sources for this example public:
- Git repository: https://git.chrz.de/chonal/hello-ci
- Docker image: registry.chrz.de/public/hello-ci
Conclusion
I am very pleased with this setup. It enables me to outsource builds to my VPS, and provides a robust docker build system. My Raspberry Pi would often crash or overheat on long builds.
Future Work
In the future, I would like to see my Docker registry interface Portus to support Docker v2 manifest list images as well. Currently, they are accessible using the registry, but do not show up in the web UI. But you have to consider, this is still an experimental container type – maybe in the future. To compensate for this, you could run a second UI like docker-registry-ui (https://github.com/Quiq/docker-registry-ui), which handles v2 manifests, but does neither provide authentification nor authorization.
Also, at the time of writing, the CI build is not that well behaved when it gets manually cancelled. The buildkit containers can stay alive and have to be killed manually, because Drone CI at the time of writing does not provide a way to trigger pipeline cleanups on cancel. To compensate for this, you could write a simple garbage collect cron job or container, which correlates every running buildkit container to a drone runner container using the name of the buildkit container and the container ID of the runner container. It then removes orphan buildkit containers.
Update and why Open Source is awsome
So, I patched Portus in order to fix the issues above. I also patched docker-registry-ui to correctly display multi-arch images. All of this hotfixing and debugging would have never been possible if the code was not freely available.
Image source: https://www.pexels.com/de-de/foto/industrie-warenhaus-geschaft-draussen-122164/