This article is about combining container images from various sources and architectures and combining them into a single multi-arch container image. If you’re familiar with docker buildx, then maybe you’ve built multi-arch images before from the same Dockerfile.

docker-seleniarm is one example where we use docker buildx to build container images for multiple architectures. In this fork of docker-selenium, we build x86_64, arm64, and armv7l images for Chromium and Firefox on Debian. The source of the browser images is the same Dockerfile, just built 3 times, one for each supported architecture.

Here’s a quick review of how to build a multi-arch image. First, use aptman/qus to register other architectures:

1

$ docker run --rm --privileged aptman/qus -s -- -p

Afterwards, build the images with buildx and push to a registry:

1

$ docker buildx build --push --platform linux/arm/v7,linux/arm64,linux/amd64 -t repo/image:tag .

At the end of this process, you’ll have a multi-arch docker image in a registry somewhere. If you need to know more about this process, you can find a lot of information on the Internet about this.

Let’s take a moment to explore ARM on Linux. It is still considered a bit experimental in some circles, at least it is on the Selenium project where the latest browser binaries and drivers aren’t always available for ARM. Up until we forked docker-seleniarm from docker-selenium, the only option for those wanting to test inside containers was to use x86_64 machines, or to build the images themselves.

The main differences between docker-selenium and docker-seleniarm browser container images is that Google Chrome doesn’t yet exist on Linux ARM, and both Chromium and Firefox oftentimes are at least one or two versions behind those available on x86_64. So docker-selenium browser images have the same browsers used by most users, whereas docker-seleniarm is considered more experimental, and there’s no guarantee we’ll have the latest browser binaries.

Some developers and testers work on teams where some members use Intel machines and some use Apple Silicon M1/M2 machines. One common workaround to allow everyone to work together is to detect the system architecture and load docker-selenium images for x86_64 and docker-seleniarm for ARM machines. Here’s a small sample of such logic in Node.js:

1
2
3
4

// if OS is not Apple M1, then assume Intel and use selenium; otherwise, use seleniarm.
const image = !os.cpus()[0].model.includes('Apple M1')
            ? 'selenium/standalone-chrome:latest'
            : 'seleniarm/standalone-chromium:latest');

But some people don’t like having to add that extra logic. First, the x86_64 people will be using different browser versions than the ARM folks, and second, it’s extra code in config files to make this work.

So some teams just use docker-seleniarm everywhere, which means both x86_64 and ARM developers and testers are using the same experimental images.

But let’s say we wanted to have the x86_64 people use the stable images while the M1 users use the seleniarm ones, but without the extra logic. This is where tools like docker manifest or docker buildx imagetools enter the picture. Let’s look at some examples.

Multi-Arch from Different Sources

Using docker manifest

Below is an example that uses docker manifest in combination with a CLI tool I wrote called get-image-sha256-digest.go to make it easier to obtain the sha256 strings for each architecture. The below example creates a manifest jamesmortensen1/standalone-firefox:combo and then adds the x86_64 image built from docker-selenium and adds the arm64 and armv7l images built from docker-seleniarm. The result is essentially a multi-arch images but built from different Dockerfiles:

1
2
3
4
5
6

docker manifest create jamesmortensen1/standalone-firefox:combo \
  --amend selenium/standalone-firefox@`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/selenium/standalone-firefox/tags/latest/ | grep -w "amd64" | awk '{print $2}'` \
  --amend seleniarm/standalone-firefox@`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/seleniarm/standalone-firefox/tags/latest/ | grep -w "arm" | awk '{print $2}'` \
  --amend seleniarm/standalone-firefox@`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/seleniarm/standalone-firefox/tags/latest/ | grep -w "arm64" | awk '{print $2}'`

Created manifest list docker.io/jamesmortensen1/standalone-firefox:combo

Note that the image is in the build cache. It isn’t pushed to the registry until running the following command:

1

$ docker manifest push jamesmortensen1/standalone-firefox:combo

You can also inspect the manifest and confirm that all three architectures are present. If you compare the sha256 images to what you see in Docker Hub, you can confirm they’re the same:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

docker manifest inspect jamesmortensen1/standalone-firefox:combo
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 6364,
         "digest": "sha256:0118c6dbe298bd8f42a03cab865546774e6756cc5c3360af8fa4ee5903acc0d5",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 6378,
         "digest": "sha256:6118383e83deaae3b2086086665594fe8ee92ba5b9b342a909fa5afe14fed01c",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 6802,
         "digest": "sha256:a081d355c2b30503b040d3bc738ed4d85e6b3f7ee643b88dbd2a4bb059dfbf65",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Using docker buildx imagetools

docker buildx imagetools is a new tool from Docker which allows you to combine images built from different sources and architectures. It works very similar to docker manifest except it pushes the images to the registry in one step. The syntax is very similar.

The command below creates an image called jamesmortensen1/standalone-firefox:imagetools that combines the x86_64 docker-selenium standalone-firefox image with the arm64 and armv7l images created from docker-seleniarm’s standalone-firefox image. The command also pushes the image to the registry. It’s very similar to docker manifest, and we also leverage the get-image-sha256-digest.go CLI tool to help obtain the sha256 hashes for each architecture:

1
2
3
4
5
6
7

docker buildx imagetools create -t jamesmortensen1/standalone-firefox:imagetools \
  selenium/standalone-firefox@`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/selenium/standalone-firefox/tags/latest/ | grep -w "amd64" | awk '{print $2}'` \
  seleniarm/standalone-firefox@`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/seleniarm/standalone-firefox/tags/latest/ | grep -w "arm" | awk '{print $2}'` \
  seleniarm/standalone-firefox@`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/seleniarm/standalone-firefox/tags/latest/ | grep -w "arm64" | awk '{print $2}'`

[+] Building 9.6s (1/1) FINISHED                                                                                                                
 => [internal] pushing docker.io/jamesmortensen1/standalone-firefox:imagetools

Again, we can also inspect the newly created image using docker manifest inspect to validate that the image sha256 digests are what we see in Docker Hub for selenium/standalone-firefox and seleniarm/standalone-firefox:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

docker manifest inspect jamesmortensen1/standalone-firefox:imagetools
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 6802,
         "digest": "sha256:a081d355c2b30503b040d3bc738ed4d85e6b3f7ee643b88dbd2a4bb059dfbf65",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 6364,
         "digest": "sha256:0118c6dbe298bd8f42a03cab865546774e6756cc5c3360af8fa4ee5903acc0d5",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 6378,
         "digest": "sha256:6118383e83deaae3b2086086665594fe8ee92ba5b9b342a909fa5afe14fed01c",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}
sha256 hash for selenium/standalone-firefox sha256 hashes for seleniarm/standalone-firefox sha256 hashes for jamesmortensen1/standalone-firefox showing one sha256 hash from selenium and two from seleniarm

Run the containers

Thanks to imagetools (or the more legacy docker manifest option) we can now start containers from different Dockerfiles on different machine architectures but using the same image names. Here are some examples below of running the Firefox containers on ARM and x86_64:

On an aarch64 machine:

1

$ docker run --rm -it -p 7900:7900 --shm-size 2g jamesmortensen1/standalone-firefox:imagetools

On an x86_64 machine:

1

$ docker run --rm -it -p 7900:7900 --shm-size 2g jamesmortensen1/standalone-firefox:imagetools

Note that it’s the same command in both cases.

The difference here is that on the x86_64 machine, we’re actually pulling selenium/standalone-firefox:latest, but on the aarch64 machine, we’re pulling seleniarm/standalone-firefox:latest. When running the containers in both machine architectures and then viewing the desktop via noVNC port 7900, we confirm the operating systems are different. The images are definitely not built from the same Dockerfiles!

Ubuntu desktop showing the operating system. Debian desktop showing the operating system.

What’s more, in each environment, run the arch command in the terminal to confirm the architectures are different:

Ubuntu:

1
2

$ arch
x86_64

Debian:

1
2

$ arch
aarch64

Conclusion - Should we merge images from various sources

Just because we can do something doesn’t necessarily mean we should. So we’re now left with the question of whether or not this makes sense for Selenium and Seleniarm. Here’s a little background on what we do today:

Normally, when we build with buildx, we do so from a single Dockerfile. We instruct docker to build the image for armv7l, arm64, and x86_64. I want to emphasize that this is from the same Dockerfile. So when developers and testers who are using seleniarm on both x86_64 and ARM machines decide to use seleniarm/standalone-firefox on both machines, those who are on the x86_64 machines are also getting the experimental images.

But by using docker manifest, or docker buildx imagetools, we can take an x86_64 image built from one Dockerfile and combine it with ARM images built from an entirely different Dockerfile.

This may make sense for some projects, but for Selenium and Seleniarm, it seems more like a fun experiment. I am not sure this makes sense for Selenium, and I’ll explain why.

The docker-selenium images are considered stable at this point in time. They’re officially maintained by the Selenium team. On x86_64 platforms, browser binaries and drivers are widely available and kept up to date by maintainers on Ubuntu and various other Linux distributions. Ubuntu is the base image for all of these container images, and with the exception of the move toward snaps, it generally works quite well.

But docker-seleniarm is an entirely different story. It’s highly experimental, and binaries are limited. First, Google only compiles binaries for Chrome on Linux for x86_64, not ARM. This means that, while the browser used in Selenium is Google Chrome, on Linux ARM it’s open source Chromium. It’s a subtle difference, but it can matter. Maybe it’s ok in some situations, and maybe in other situations it’s not ok. Additionally, the browser and driver versions may not always match up. Linux ARM tends to be at least one or two versions behind x86_64, both for Chromium as well as Firefox.

Moreover, we use Debian as the base for docker-seleniarm images. First, Debian tends to have more up to date binaries than Ubuntu. When developers and testers use docker-selenium, they’re running Ubuntu. But when using docker-seleniarm, they’re using Debian, regardless of whether they’re running x86_64 or ARM. Second, the snap images were quite confusing, and Debian still offers browsers and drivers for installation via apt-get, both for Chromium and Firefox.

Now, what happens if we build a multi-arch image where x86_64 is based on the stable container images and ARM is based on the experimental images. This solves the problem where developers and testers need to swap out the image name based on the OS/architecture, but it creates a problem where the same browser type and versions are not the same. It seems much clearer to have docker-seleniarm use the same browser binaries across all three architectures, and if someone on x86_64 needs more stability, or Google Chrome, or Ubuntu, then that person should use docker-selenium images instead. This is currently what I recommend, but it would be interesting to hear what others think as well.