Raspberry Pi Cross Compilation using Docker

Posted on by chrz

The Raspberry Pi is a great little computer for makers. However, it lacks the performance to compile big software packages in an acceptable timeframe. So I set out to create a fast and easy-to use docker-based cross-compiler for the Pi, which runs on much more powerful machines like for example a VPS.

Continuous Integration

This server does not only host the WordPress blog you are currently reading, but also, among other things, a full CI stack using Docker (https://www.docker.com/):

Now, I want to be able to write Dockerfiles, push them into a Git repository, and have them built into a Docker image which supports multiple architectures. Then I just pull the image on to my Raspberry Pi and run the program.

Docker BuildX and QEMU-BinFMT

But how do we run binaries for a different architecture on our server? That is where QEMU (https://www.qemu.org/), an open-source machine emulator comes in. Because most (cheap) VPS do not allow hardware virtualisation, we run QEMU in the user space (https://wiki.debian.org/QemuUserEmulation). QEMU provides the needed modules for binfmt_misc (https://en.wikipedia.org/wiki/Binfmt_misc), a linux kernel function to execute arbitrary binaries via user space modules.

Now we could build our docker image using the good old docker build command and then assemble a multi-arch image using docker manifest, but that is quite a lot of manual work. Instead, we use the docker buildx experimental feature, which is able to build for a given range of architectures.

The BuildX Docker command is still an experimental feature, so you either have to install it from a binary, or use a recent Docker version and enable experimental features.

Conveniently, there is a docker container which installs the needed QEMU modules:

docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

You should check for new versions of that image if you want to follow along on this adventure. After having installed the modules into the host, we can create a builder which supports the new architectures, and bootstrap it:

docker buildx create --use --name crosscomp && docker buildx inspect --bootstrap

The driver then tells us its capabilities:

Name:   crosscomp
 Driver: docker-container
 Nodes:
 Name:      crosscomp0
 Endpoint:  unix:///var/run/docker.sock
 Status:    running
 Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

To create a multi image that runs on PC and as well on the Raspberry PI, we only need to add linux/arm/v7, and maybe linux/arm64:

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 ...

To enable multiple parallel builds on the CI server using multiple independent runners, we give each runner an unique name based on the current container ID:

export BUILDER_ID="crosscomp-$(cat /proc/self/cgroup | head -1 | cut -d '/' -f 3)" && docker buildx create --use --name $BUILDER_ID && ... && docker buildx rm $BUILDER_ID

Integration Server and Registry Setup

The CI server must be able to bind to the docker socket of the host, in oder to spawn the BuildKit (https://github.com/moby/buildkit) container, on which BuildX is based. Apart from that, the compiler needs privileged permissions to install the QEMU modules. These features are not commonly available on cloud services, so I host my own pipeline.

I also run a private Docker registry. This registry has to support the docker protocol v2, and especially docker manifest lists for the multi-arch images.

Docker Layer Caching

The CI server runs all tasks inside a new context, to guarantee ephemeral builds. But it would be nice to cache the Docker layers to reduce build times on subsequent builds! Because we do not have access (nor should we) to the host layer cache, we use a second repository location for our cache. This cache holds all layers, while the public repository location only holds the last stage from the Dockerfile. This enables us to cache private build stages without exposing them in the same public location.

BuildX provides the --from-cache and --to-cache flags to control this mechanic:

--cache-to=type=registry,ref=registry.chrz.de/cache/hello-ci,mode=max --cache-from=type=registry,ref=registry.chrz.de/cache/hello-ci

The mode=max argument tells BuildX to cache all stages, not only the last public one.

A Simple Example

To demonstrate this compiler, I wrote a simple example container containing a native binary which has to be compiled.

The Binary Payload

#include <iostream>
#include <sys/utsname.h>

using namespace std;

struct utsname unameData;

int main()
{
    cout << "Hello from binary!" << endl;
    uname(&unameData);
    printf("Running on %s, %s\n", unameData.sysname, unameData.machine);
    return 0;
}

This simple C++ program just prints hello and the architecture, and then exits.

Container Configuration

Then, I wrote a two-stage Dockerfile, with one stage for build and one stage for execution, to emulate real-world usecases:

# build stage
FROM ubuntu:18.04 AS build
ENV DEBIAN_FRONTEND=noninteractive TZ=Europe/Berlin

RUN apt-get update && \
    apt-get -y install --no-install-recommends g++ && \
    rm -rf /var/lib/apt/lists/*

COPY main.cpp /app/main.cpp
WORKDIR /app
RUN g++ -o main main.cpp


# run stage
FROM ubuntu:18.04 AS run
ENV DEBIAN_FRONTEND=noninteractive TZ=Europe/Berlin

RUN apt-get update && \
    apt-get -y install --no-install-recommends file && \
    rm -rf /var/lib/apt/lists/*

COPY --from=build /app/main /app/main

COPY run.sh /app/run.sh
RUN chmod +x /app/run.sh
CMD ["/app/run.sh"]

As you can see, the Dockerfile sets up a simple build environment and then copies the compiled binary into an execution environment. The entrypoint shell script outputs some meta information of the binary:

#!/bin/bash

echo "Hello from entrypoint!"
file /app/main
ldd /app/main
/app/main

Integration Server Configuration

The CI configuration for Drone basically consists of the steps discussed above: 

kind: pipeline
name: default

volumes:
  - name: docker_socket
    host:
      path: /var/run/docker.sock

steps:
- name: build  
  image: alexviscreanu/buildx
  commands:
    - docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
    - export BUILDER_ID="crosscomp-$(cat /proc/self/cgroup | head -1 | cut -d '/' -f 3)"
    - docker buildx create --use --name $BUILDER_ID --driver-opt image=stargate01/buildkit
    - docker buildx inspect --bootstrap
    - docker login --username $DOCKER_USERNAME --password $DOCKER_PASSWORD registry.chrz.de
    - docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 --output=type=image,push=true --progress tty -t registry.chrz.de/public/hello-ci .
    - docker buildx rm $BUILDER_ID
  volumes:
    - name: docker_socket
      path: /var/run/docker.sock
  environment:
    DOCKER_USERNAME:
        from_secret: docker_username
    DOCKER_PASSWORD:
        from_secret: docker_password

It uses the image alexviscreanu/buildx, which is just Docker with BuildX installed. It also instructs the server to push the compiled image to my registry. Registry login data is provided via secrets by the CI server.

Note that the repository has to be configured as “trusted” inthe CI server in order to mount the Docker socket and run in privileged mode.

Testing the Image

To run the image, I use the command docker run registry.chrz.de/public/hello-ci, which downloads and executes the image.

When I run the image on my x64 PC, the output looks like this:

Hello from entrypoint!
 /app/main: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=f354df5ffd19fa277ef28d97fe4e8760ee630302, not stripped
         linux-vdso.so.1 (0x00007fff1676b000)
         libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1819ef2000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1819b01000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1819763000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f181a47d000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f181954b000)
 Hello from binary!
 Running on Linux, x86_64

Now, on my Raspberry Pi, using the exact same command and meta-image, Docker knows to download the sub-image for the current architecture. The output then looks like this:

Hello from entrypoint!
 /app/main: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib, for GNU/Linux 3.2.0, BuildID[sha1]=ef78a3c9e7f58a215bafa2839394746dfd2f5013, not stripped
         linux-vdso.so.1 (0x7ed38000)
         libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x76e61000)
         libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x76e38000)
         libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76d40000)
         libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76cbf000)
         /lib/ld-linux-armhf.so.3 (0x76f86000)
 Hello from binary!
 Running on Linux, armv7l

Success! Not only does it run, but it also shows that the QEMU cross-compiler did in fact create valid ARM executables.

Download Sources

I made the sources for this example public:

Conclusion

I am very pleased with this setup. It enables me to outsource builds to my VPS, and provides a robust docker build system. My Raspberry Pi would often crash or overheat on long builds.

Future Work

In the future, I would like to see my Docker registry interface Portus to support Docker v2 manifest list images as well. Currently, they are accessible using the registry, but do not show up in the web UI. But you have to consider, this is still an experimental container type – maybe in the future. To compensate for this, you could run a second UI like docker-registry-ui (https://github.com/Quiq/docker-registry-ui), which handles v2 manifests, but does neither provide authentification nor authorization.

Also, at the time of writing, the CI build is not that well behaved when it gets manually cancelled. The buildkit containers can stay alive and have to be killed manually, because Drone CI at the time of writing does not provide a way to trigger pipeline cleanups on cancel. To compensate for this, you could write a simple garbage collect cron job or container, which correlates every running buildkit container to a drone runner container using the name of the buildkit container and the container ID of the runner container. It then removes orphan buildkit containers.

Update and why Open Source is awsome

So, I patched Portus in order to fix the issues above. I also patched docker-registry-ui to correctly display multi-arch images. All of this hotfixing and debugging would have never been possible if the code was not freely available.

Image source: https://www.pexels.com/de-de/foto/industrie-warenhaus-geschaft-draussen-122164/

Creator of unconventional and sometimes useful things.

Leave a Reply

Your email address will not be published. Required fields are marked *