How to create your own Docker images that run on multiple CPU architectures.
With the recent introduction of Docker’s buildx functionality it becomes possible and relatively easy for everybody to build and publish Docker images that work on multiple CPU architectures. This article focuses exclusively on Linux multi-architecture docker images, shows how to go about creating such images, and what to look out for to make it work in different host environments. It’ll cover Ubuntu and Debian distributions in particular, which are used in a number of CI/CD pipelines such as Github Actions or Travis, but it’s generally applicable to other Linux distributions too. You just need to make sure to check which kernel and userspace tool versions you’ve got.
The article assumes you’re generally familiar with using Docker. If you don’t know Docker yet, you can familiarize yourself with the basics with Docker’s Getting Started guide. Make sure you get the Hello World example working before continuing here.
How Docker Buildx Compiles for Non-Native Architectures
Docker buildx multi-architecture support can make use of either native builder nodes running on different architectures or the QEMU processor emulator. We’re only going to discuss QEMU here as it’s a pure software solution that doesn’t require you to have access to hosts that run on different CPU architectures.
QEMU works by simulating all instructions of a foreign CPU instruction set on your host processor. E.g. it can simulate ARM CPU instructions on an x86 host machine. With the QEMU simulator in place you can run foreign architecture binaries on your host. But to do so, you’d have to write every command with a prefix qemu-<arch> <your-command> on the command line.
Luckily, Linux also has built-in support for running non-native binaries, called binfmt_misc. Whenever Linux tries to execute a binary, it checks if there is a handler for that binary format registered with binfmt_misc. If there is, the handler is executed instead and pointed to the binary. The handler in turn executes the binary however it sees fit. An example of this is executing java byte code binaries with a JVM which interprets each java byte code. In our case we’ll make use of binfmt_misc to transparently execute foreign CPU binaries with QEMU.
Software Requirements for Buildx Non-Native Architecture Support
There are several software requirements that need to be met so docker buildx can create multi-architecture images:
- Docker >= 19.03: Docker itself needs to be new enough to contain the buildx feature.
- Experimental mode for the docker CLI needs to be turned on since buildx is an experimental feature.
- Linux kernel >= 4.8: The kernel side of binfmt_misc needs to be new enough to support the fix-binary (F) flag. The fix-binary flag allows the kernel to use a binary format handler registered with binfmt_misc inside a container or chroot even though that handler binary is not part of the file system visible inside that container or chroot.
- binfmt_misc file system mounted: The binfmt_misc file system needs to be mounted such that userspace tools can control this kernel feature, i.e. register and enable handlers.
- Either a Host installation or Docker image based installation of QEMU and binfmt_misc support tools.
- Host installation:
- QEMU installed: To execute foreign CPU instructions on the host, QEMU simulators need to be installed. They need to be statically linked since dynamic library resolution depends on those dynamic libraries being visible in the file system at time of use, which is not typically the case inside a container or chroot environment.
- binfmt-support package >= 2.1.7: You need to install a package that contains an update-binfmts binary new enough to understand the fix-binary (F) flag and actually use it when registering QEMU simulators.
- Docker image based installation: You can use a Docker image that contains both QEMU binaries and setup scripts that register QEMU in binfmt_misc similar to what the binfmt-support package does.
- Host installation:
If you happen to run on a system that has Docker Desktop >= 2.1.0 installed, e.g. on Mac OSX or Windows, you’re in luck since it comes configured meeting all the above requirements. In this case you can skip the rest of this section. However, if you’re running on a system where Docker Desktop is not available or installed, e.g. Linux, you’ll have to install the necessary support yourself. The rest of this section assumes you’re running on Linux x86. Now let’s go through these requirements one by one.
Docker
Docker gained buildx support with version 19.03, so you need at least this version installed. You can check your docker version with:
1 | docker --version |
If you don’t have docker installed on your system you can try to install it from your Linux distribution’s default package sources. The package typically comes by the name of docker-ce or docker.io (see also the table of popular Linux environments below):
1 | sudo apt-get install -y docker-ce |
It’s quite possible though that the docker version that comes by default with your Linux distribution is not new enough. In that case you can add Docker’s own package repository and get a newer docker version from there:
1 | DOCKER_APT_REPO='https://download.docker.com/linux/ubuntu' |
Docker Experimental Features
As of this writing (early 2020), buildx is an experimental feature. If you try to use it without turning on experimental features it’ll fail:
1 | docker buildx |
You can turn on experimental Docker CLI features in one of two ways. Either by setting an environment variable
1 | export DOCKER_CLI_EXPERIMENTAL=enabled |
or by turning the feature on in the config file:
1 | { |
If you choose the environment variable, put the setting into you shell startup script, e.g. $HOME/.bashrc for bash, otherwise the setting only sticks around in your current shell until you log out. Once you have turned on experimental features either way, you can check that it has taken effect with:
1 | docker version |
Note that this output also shows you the status of the Experimental flag of Server: Docker Engine. But this doesn’t concern us for now. With experimental mode now turned on, you should have access to the docker buildx command:
1 | docker buildx |
Linux Kernel
You need a kernel that supports the binfmt_misc feature and has it enabled. In particular, the binfmt_misc support needed to use QEMU transparently inside containers is the fix-binary (F) flag which requires a Linux kernel version >= 4.8 (commit, commit). You can check your kernel version with:
1 | uname -r |
Binfmt_misc File System
The binfmt_misc kernel features are controlled via files in /proc/sys/fs/binfmt_misc/. This file system must be mounted. E.g. on a Ubuntu 18.04 (bionic) system the script responsible for mounting that file system is /lib/systemd/system/proc-sys-fs-binfmt_misc.automount which is part of the systemd package and runs automatically at boot time (and also during package installation). You can check if the file system is mounted with:
1 | ls /proc/sys/fs/binfmt_misc/ |
Host Installation: QEMU
An easy way to install statically linked QEMU binaries is to use a pre-built package for your host Linux distribution. E.g. for Debian or Ubuntu you can install it with:
1 | sudo apt-get install -y qemu-user-static |
That has installed QEMU for a number of foreign architectures, e.g. 64-bit ARM (aarch64), as you can see by checking:
1 | ls -l /usr/bin/qemu-aarch64-static |
Other Linux distributions might use different package managers or package names for the QEMU package. Alternatively you can install QEMU from source and follow the build instructions.
Host Installation: update-binfmts Tool
The update-binfmts tool is typically part of the binfmt-support package. If you look back at the installation of qemu-user-static above you’ll see that it has automatically pulled in the recommended binfmt-support package, so in our case it’s already installed. But if you’ve specified the --no-install-recommends flag (or that is set by default on your system), binfmt-support might not yet be installed. If it’s missing on your system you can also install it manually with:
1 | sudo apt-get install -y binfmt-support |
Here again, we need support for the fix-binary (F) flag, which was added to update-binfmts with version 2.1.7. You can check the version with:
1 | update-binfmts --version |
Checking Your Host System
Putting everything together, you can check if the aforementioned environment is in place for using QEMU with docker buildx with the following check-qemu-binfmt.sh script:
Problem: QEMU Not Registered With (F) Flag
In some environments you can run into the situation that the appropriate kernel and update-binfmts support is present, but the qemu-user-static post-install script does not register QEMU with the fix-binary (F) flag. The checker script above will point that out. One such environment is e.g. AWS EC2 instances running Ubuntu 18.04 (bionic). In such a case you can fix up the installation by re-registering QEMU with the fix-binary (F) flag with the following reregister-qemu-binfmt.sh script:
Docker Image Based Installation
As an alternative to installing the QEMU and binfmt-support packages on your host system you can use a docker image to satisfy the corresponding requirements. There are several docker images that do the job, among them multiarch/qemu-user-static and docker/binfmt. They come loaded with QEMU simulators for several architectures and their own setup script for installing those QEMU simulators in the host kernel’s binfmt_misc with the fix-binary (F) flag. The QEMU simulators stay registered and usable by the host kernel after running that docker image as long as the host system remains up (or you explicitly unregister them from binfmt_misc). That is what also makes them usable by later runs of docker buildx. Unlike the host installation of packages though, you’ll need to re-run that docker image after every system reboot.
Using those images doesn’t release you from having the right docker and kernel version on the host system, but you do get around installing QEMU and binfmt-support packages on the host. I like to use multiarch/qemu-user-static:
1 | docker run --rm --privileged multiarch/qemu-user-static --reset -p yes |
Status of Popular Linux Environments
The following table shows the current status of docker buildx support on various popular Linux environments. Only Ubuntu >= 19.10 (eoan) and Debian 11 (bullseye/testing) come with sufficient support by default to be able to run docker buildx out of the box. All older versions of these Linux distributions need updates of various components in order to be compatible with docker buildx usage. For example Ubuntu 18.04 (bionic) requires re-registration of QEMU with the fix-binary (F) flag or usage of the docker image installation method for QEMU as described above as well as an upgraded docker package.
| Environment | Docker Package | Kernel | binfmt-support | (F) Flag |
|---|---|---|---|---|
| Requirements | >= 19.03 | >= 4.8 | >= 2.1.7 | yes |
| Ubuntu: | ||||
| 18.04 (bionic) | 17.12.1 docker.io | 4.15.0 | 2.1.8 | no |
| 19.04 (disco) | 18.09.5 docker.io | 5.0 | 2.2.0 | yes |
| 19.10 (eoan) | 19.03.2 docker.io | 5.3 | 2.2.0 | yes |
| 20.04 (focal) | 19.03.2 docker.io | 5.5 | 2.2.0 | yes |
| Debian: | ||||
| 9 (stretch) | - | 4.9.0 | 2.1.6 | no |
| 10 (buster) | 18.09.1 docker.io | 4.19.0 | 2.2.0 | yes |
| 11 (bullseye/testing) | 19.03.4 docker.io | 5.4 | 2.2.0 | yes |
| AWS EC2: | ||||
| Ubuntu 16.04 (xenial) | 18.09.7 docker.io | 4.4.0 | 2.1.6 | no |
| Ubuntu 18.04 (bionic) | 18.09.7 docker.io | 4.15.0 | 2.1.8 | no |
| Travis (on GCP): | ||||
| Ubuntu 14.04 (trusty) | 17.09.0 docker-ce | 4.4.0 | 2.1.4 | no |
| Ubuntu 16.04 (xenial) | 18.06.0 docker-ce | 4.15.0 | 2.1.6 | no |
| Ubuntu 18.04 (bionic) | 18.06.0 docker-ce | 4.15.0 | 2.1.8 | no |
| Github Actions (on Azure): | ||||
| Ubuntu 16.04 (xenial) | 3.0.8 moby-engine | 4.15.0 | 2.1.6 | no |
| Ubuntu 18.04 (bionic) | 3.0.8 moby-engine | 5.0.0 | 2.1.8 | no |
Building Multi-Architecture Docker Images With Buildx
With all the software requirements on the host met, it’s time to turn our attention to how buildx is used to create multi-architecture docker images. The first step is setting up a buildx builder.
Creating a Buildx Builder
The docker CLI now understands the buildx command, but you also need to create a new builder instance which buildx can use:
1 | docker buildx create --name mybuilder |
You can check your newly created mybuilder with:
1 | docker buildx inspect --bootstrap |
Note how the Platforms line reports support for various non-native architectures which you have installed via QEMU. If it only reports support for linux/amd64 and linux/386 you either still haven’t met all software requirements, or you had created a builder before you have met the software requirements. In the latter case remove it with docker buildx rm and recreate it.
You can also see your just created mybuilder with buildx’ ls subcommand:
1 | docker buildx ls |
Using Buildx to Build
Alright, now we’re ready to build multi-architecture docker images with buildx. To have something concrete to work with we’re going to use the following example:
1 | FROM alpine:latest |
It’s a simple stand-in for whatever you’d like to build yourself in your own Dockerfile. It uses the latest Alpine distribution - which itself is a multi-architecture docker image - and prints out the architecture on which it is executing. That will allow us to check which kind of image we’re running.
The docker buildx build subcommand has a number of flags which determine where the final image will be stored. By default, i.e. if none of the flags are specified, the resulting image will remain captive in docker’s internal build cache. This is unlike the regular docker build command which stores the resulting image in the local docker images list. The important flags are:
--load: This flag instructs docker to load the resulting image into the localdocker imageslist. However, this currently only works for single-architecture images. If you try this with multi-architecture images you’ll get an export error:docker buildx build ... --load ...
...
=> ERROR exporting to oci image format 0.0s
------
exporting to oci image format:
------
failed to solve: rpc error: code = Unknown desc = docker
exporter does not currently support exporting manifest lists--push: This flag tells docker to push the resulting image to a docker registry. Your image tag has to contain the proper reference to the registry and repository name. This currently is the best way to store multi-architecture images.
We’re going to use the default Docker Hub registry. First we have to log in:
1 | export DOCKER_USER='arturklauser' |
Now we can build and use the --push flag to push the image to Docker Hub. In our example we’re going to build for three different architectures - x86, ARM, and PowerPC - which are specified with the --platform flag:
1 | docker buildx build -t "${DOCKER_USER}/buildx-test:latest" \ |
We can check the image with the imagetools subcommand which confirms that three architecture versions are included in the image:
1 | docker buildx imagetools inspect "$DOCKER_USER/buildx-test:latest" |
Also, on the Docker Hub web site we see it reported as:
To verify that you’ve actually got what you’ve been promised, let’s try to run the image:
1 | docker run --rm "$DOCKER_USER/buildx-test:latest" |
As expected, since we’re running on a 64-bit x86 host, the default architecture version that was used by docker was the amd64 which reports running on x86_64. If you check the local image in docker it confirms that:
1 | docker inspect --format "{{.Architecture}}" "$DOCKER_USER/buildx-test:latest" |
To pull and run a specific architecture version, use the image name including its full sha256 value that was reported by imagetools:
1 | docker run --rm "$DOCKER_USER/buildx-test:latest@sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c" |
Since the sha256 value we requested here was that of the PowerPC image version, we see that the image is reporting to run on ppc64le as expected.
Optionally, we can pull and run non-native image versions by platform name. For that though we need to turn on another experimental feature, this time in the docker engine, that’ll allow us to specify a --platform. If docker engine experimental features are not turned on you’ll get an error instead:
“–platform” is only supported on a Docker daemon with experimental features enabled
Change the docker engine configuration file or create one if it doesn’t exist already:
1 | { |
After changing the configuration file you’ll also need to restart dockerd for the change to take effect:
1 | sudo systemctl restart docker |
Let’s purge the image that we’ve already pulled and try a different architecture:
1 | docker rmi "$DOCKER_USER/buildx-test:latest" |
Now we see that the architecture version of the image we’ve pulled and run is the one for 64-bit ARM aarch64, as can also be verified by looking at the image metadata:
1 | docker inspect --format "{{.Architecture}}" "$DOCKER_USER/buildx-test:latest" |
With this you’ve got to the point where you can start to build your own multi-architecture docker images with buildx.
Happy developing!
Appendix
The following script shows how you can use what was described above to build multi-architecture docker images in CI/CD pipelines like Github Actions or Travis.