Cross Browser Testing with Safari Alternative: WebKitWebDriver-Epiphany Docker Container Image

Users of Web applications use a variety of different browsers. Neglecting to test an application in one of them may result in lost revenue. Testing in the browsers your users use is important for maximizing revenue.

All of our end to end UI tests are running only on Google Chrome. I want to change that and ensure we’re covering more of the browsers used by our users. There was no low-cost method available for testing on Safari in the cloud, and according to browser usage statistics, it’s the browser we should be targeting next.

Google Chrome is the browser developers and testers reach for first, and for good reason: It is the most widely used browser in the world. However, Chrome isn’t the only browser, and 5%, 8%, 10% of 100,000 is still a big number for the bottom line. It is still important, from a financial and business perspective, to support other browsers.

Let’s examine the most recent browser usage stats, from statcounter GlobalStats, as of August 2022:

While Google Chrome is the most used browser at 65.51%, Safari is the second at 18.8%, perhaps because it comes with macOS by default and because it is so small and lightweight that many people are happy using it instead of the Chromium-based goliaths.

But Safari poses a challenge for CI setups. How can we run tests on Safari in the cloud or in our CI pipeline without it costing an arm and a leg?

Apple Safari only runs on macOS. It does not run on Windows or Linux, and Apple imposes some strict licensing requirements when it comes to virtualizing macOS. This means things like Hackintosh, or running macOS in a VM on a Windows or Linux server, violate Apple’s terms and conditions.

We want to focus more on cross browser testing since our users don’t all just use Google Chrome. Here is our current setup: We use WebdriverIO. It communicates with either a WebDriver or Selenium server. Each browser requires a separate WebDriver implementation in order for the testing framework, WebdriverIO, to communicate with the browser. While WebdriverIO controls Google Chrome via chromedriver, it would need to control Safari via safaridriver.

The e2e tests for one of our modules run on a series of Ubuntu 20.04 GitHub Actions runners. We run the tests in parallel on 6 Chrome browsers. On each runner, we’ve deployed 3 selenium/node-chrome images and 1 selenium/hub. That’s 6 browsers if two jobs run concurrently and 9 browsers if 3 jobs run concurrently. There is one job which just runs some payment tests on a single selenium/standalone-chrome container, but for the most part, everything else follows this pattern of 3 Chrome browsers and 1 Hub per job.

To see what this setup looks like, see the diagram below. The jobs with Google Chrome exist today, and the WebKit job is what we intend to setup next:

Since there are several jobs executing at once, we needed to split the test spec files between them. We wrote a WebdriverIO service that splits a single array of spec files into different equal sub-arrays, based on the number of workers/jobs. For instance, if there are 2 jobs then the service creates 6 sub-arrays of spec files and if there are 3 jobs then it generates 9 sub-arrays, one sub-array for each browser. Each sub-array is then assigned to one of the browsers running in one of the node-chrome containers. We make sure each worker uses a login/password combination that is unique from the other workers so that the activity in one test will not affect another.

This process greatly reduces the amount of time it takes to run tests and get feedback. We reduced the test execution runtime for this particular module from over 40 minutes to 7 minutes. However, at this time, all the tests run only on Chrome.

There are some tests we should run on other browsers, and Safari is next, based on the browser statistics we saw earlier.

Hopefully, after seeing the above setup, you’re starting to see the problem: Safari doesn’t run on Ubuntu, and the Selenium container images all run Ubuntu. Also, SeleniumHQ only maintains container images for Google Chrome, Mozilla Firefox, Microsoft Edge, and Opera.

We could use a macos-10.15 GitHub Actions runner. Safari will run, and we’d be able to run tests, but it would cost 10 times more than using the Ubuntu runners. However, this is not the only option.

Ivan Krutov, a software engineer at Aerokube and core developer on the Selenoid project, built a container image called browsers/safari, which runs on their Go-based testing platform. Thinking Safari cannot possibly run in Linux, I decided to investigate. Upon further investigation, I discover that the image installs a WebkitGTK based browser, but not Safari. WebkitGTK is a port of WebKit, which runs on various Linux distributions.

Epiphany and Minibrowser are two examples of browsers built on the WebkitGTK browser engine. They’ve been around for years, and they run on Linux. WebKitGTK is a port of the WebKit browser engine. These browsers are fundamentally very similar to Safari: They all share the same WebKit browser engine.

It turns out that there isn’t much difference between how web pages are rendered in different browsers built on the same browser engine. This means testing in Epiphany or MiniBrowser is very likely to get you the same results as testing on Safari, just without the shiny Apple icons and without forking out a pile of cash for expensive hardware, a 10x macos-10.15 GitHub Actions runner, or MacStadium.

Consequently, the same is true for the plethora of Chromium based browsers. Google Chrome, Microsoft Edge, Yandex, Brave, Epic, and of course the open source Chromium browser, which the latter browsers are built on top of, all run on the Blink browser engine; the rendering is identical.

Since these browsers are all built on the same engine, do we need to test on them all? In many cases, it is enough to just target the big 3 browser engines: Blink (Chromium based browsers), Gecko (Firefox), and WebKit (Safari, Epiphany, MiniBrowser).

There may of course be small exceptions. I once saw something fail on one of our applications – only once – in Yandex that ran smoothly in Google Chrome. However, in the three years I’ve been using Yandex, I cannot recall any other time I’ve had a problem with a site where that same problem didn’t also exist in Google Chrome. Regardless of my own personal experiences, it is up to each engineering team to determine what it should and shouldn’t include in the test plan. For instance, if 50% of your users use Microsoft Edge, then perhaps you also stand to benefit from testing in this specific browser, even though its adoption is lower than Google Chrome and Safari worldwide.

As I mentioned earlier, SeleniumHQ maintains several browser container images, but they don’t maintain a WebKit container image. To fill this gap, I made one. I discovered there is a WebKitWebDriver implementation, and it is capable of communicating with any WebKitGTK browser, which includes Epiphany, which is the default browser on the GNOME Linux desktop, and MiniBrowser, an extremely lightweight WebKitGTK browser which is really just intended for developers working on WebKit to test out new features on that browser engine.

Unlike the SeleniumHQ container images, there is no Selenium server on WebKitWebDriver-Epiphany. I initially tried to utilize selenium/node-base and connect WebKitWebDriver to the Selenium Server running in that container image, but it wasn’t clear to me how to get Selenium to talk to WebKitWebDriver. However, I observed that, unlike Safaridriver, WebKitWebDriver can accept remote connections, as long as the –host flag is specified and the address is set to 0.0.0.0. As a result, I rebuilt the container image from scratch, using Debian 11 as a base image. I installed supervisor and followed the pattern I saw in selenium/node-base, including fluxbox, xvfb, VNC, and noVNC. I also included WebKitWebDriver. I configured supervisor to start all of the services when the container starts. I also installed Epiphany and MiniBrowser.

As an aside, I work on a Mac M1, an ARM64 architecture. Unlike other browser binaries, WebKitGTK2 is up to date on both x86_64 and ARM64. So I built the image as a multi-arch image. If you’re running a Mac M1 laptop, you’re not left out in the cold and can also use jamesmortensen/webkitwebdriver-epiphany. I also maintain a fork of the SeleniumHQ container images called Seleniarm, which are great for Mac M1 QA engineers, but just be aware that the browser binaries will be slightly out of date due to lack of support by the community.

One limitation with jamesmortensen/webkitwebdriver, as opposed to the SeleniumHQ images, is that it can only run as a standalone instance. Because Selenium isn’t installed, we cannot run the system as part of a Selenium Grid, but we can run tests locally on WebKit browsers on Windows and Linux host systems, and we can run the standalone jamesmortensen/webkitwebdriver-epiphany container images on ubuntu-latest runners in GitHub Actions for a reasonably low cost.

There are two ways you can obtain the container image. First, you can clone the WebKitWebDriver-Browser-Images repo and then build the images yourself:

1

$ docker build -f Dockerfile.epiphany -t jamesmortensen/webkitwebdriver-epiphany:latest .

Or you can pull the image from Docker Hub directly. The best way to retrieve the image is to simply use docker run:

1

$ docker run --rm -it --shm-size=3g -p 7900:7900 -p 4444:4444 -p 5900:5900 jamesmortensen/webkitwebdriver-epiphany:latest

I opened ports 7900 and 5900 for noVNC and VNC respectively. I prefer using the browser to view the desktop at http://localhost:7900, but you can also use a VNC client on port 5900 if you prefer.

To configure WebdriverIO to run tests against Epiphany, change the capabilities in wdio.conf.js so that it looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

  capabilities: [{
        maxInstances: 1,
        browserName: 'Epiphany',
        browserVersion: '3.38.2',
        'webkitgtk:browserOptions': {
            args: [
                '--automation-mode'
            ],
            binary: 'epiphany'
        }
    }],
};

Or if you want to test in the MiniBrowser, use this instead:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

capabilities: [{
        maxInstances: 1,
        browserName: 'MiniBrowser',
        browserVersion: '2.34.1',
        'webkitgtk:browserOptions': {
            args: [
                '—automation'
            ],
            binary: '/usr/lib/${ARCH}-linux-gnu/webkit2gtk-4.0/MiniBrowser'
        }
    }],

It’s mandatory to specify the browser binary location in the capabilities, unlike with Google Chrome or Firefox. The binary property for MiniBrowser includes the full path, and part of that path requires the architecture, such as x86_64 or aarch64. In wdio.conf.js, specify the following, depending on whether or not you’re running on Intel or ARM64:

1

const ARCH = 'x86_64';  // for Intel

1

const ARCH = 'aarch64';  // for ARM64, such as Mac M1

Make sure the container is running. If you ran the above docker run command earlier, then it should already be running. You can verify WebKitWebDriver is listening for connections by going to http://localhost:4444/status and observing that the WebDriver returns “ready”. You should see the following response:

1

{“value”:{“ready”: true, “message”:”No sessions”}}

Once confirmed, start running the tests as normal:

1

$ npx wdio

While this example involves running locally, a GitHub Actions ubuntu-latest runner can run the same commands.

Ideally, it would be great to be able to run WebKit browsers in the Selenium Grid, and that’s still an open challenge. If you’re interested in contributing and know enough about the Selenium infrastructure to make this happen, this repo is accepting pull requests, as well as ideas on things to try to make that be a possibility.

Even though Google Chrome is the most widely used browser by leaps and bounds, a lot of users use the other browsers, and it’s important for the bottom line to make sure applications work properly on them. Our goal is to have cross-browser testing on all 3 major browser engines: Blink, Gecko, and WebKit. This container image is a step in that direction without needing to pay 10 times more on GitHub Actions. While we don’t yet have the ability to include it in the Grid, there’s nothing stopping us from configuring WebdriverIO and GitHub Actions to run a subset of tests on a standalone WebKitWebDriver container image. There’s also nothing stopping you from doing the same. I hope this container image is helpful for others in their cross browser testing journey.