Introduction to front-end testing. Part V. Visual testing

YEAR 2023
Technology partner

Visual Testing

Software Testing has always been a passion of mine. These days, I find it difficult to write code without having tests for it. The idea of just running the code in order to verify its correctness seems so beginning of century to me. You mean to tell me that in the past every time a developer changed their code, somebody needed to manually verify that everything that previously worked is still working? Really?

So I write tests and also talk and write about Software Testing. And when the opportunity arose to join a company whose vision is to enhance Software Testing, to write code that helps other people write tests, and to evangelize their products, I did not hesitate one bit.

So, I joined Applitools. And because their product, Applitools Eyes, has a direct connection to the content of this series, you’re now reading one more post in this series — a post about “Visual Testing”.

Remember my amazement at the fact that developers actually need to always run their application every time they change their code? Well, up to now, one aspect of software products necessitated manual testing — the visual aspect of the application. There was no way to check that the application still looks good — the fonts are correct, the alignment is right, the colors are not off, etc.

Theoretically, you could write code that checks this. We saw in part three how we can use Selenium Webdriver to test our web application’s UI. We could use Selenium’s getScreenShot API to take a screenshot of the page, save it as a baseline, and then each test will also check the page screenshot against the baseline:

Ha! If only it were that simple. I tried this solution once, and found so many problems that I had to abandon it, and, yes, run the application every time I changed the code. The main problem is somewhat technical: there are subtle variations in the image related to how the browser renders the content — depending on the screen or the GPU, the content is anti-aliased in subtly different ways. No two screenshots will have exactly the same pixels. This is almost imperceptible to the human eye, but it means that a pixel by pixel comparison is pointless. You need image analysis techniques to solve this.

But there are other problems, problems which I understood only when I started working at Applitools:

You can’t take a full page screenshot — you can only take a screenshot of what’s in the viewport.
If the page has animations, then you will never be able to compare them to the base image.
Dynamic data, like ads, will complicate figuring out the real differences against the baseline.
When is the page “fully loaded”? When can I take that screenshot? These days, taking a picture when the DOM is loaded is not enough. Figuring out when you can take that screenshot is difficult.

Yes We Can

But it seems we can write automated visual tests. A myriad of tools, which I was unaware of, exists to enable you to take good screenshots and compare them against base images. Some of these are:

Wraith
WebdriverCSS
And, of course, Applitools Eyes
(There are others, but this blog post is already getting a bit large…)

These tools solve all or some of the problems described above. In this part of the series, I want to show how to use Applitools Eyes to write a visual test.

Writing a Visual Test

Visual tests, since they test the final product, should be used in frontend E2E browser tests. So this is where I put our visual test. The code, interestingly enough, is smaller than our regular E2E test. It consists of three parts — setting up the browser, setting up Applitools Eyes, and the test itself.

Let’s look again at the selenium driver browser setup, which is the same setup used in the E2e test in part four:


let driver
before(async () => {
  driver = new webdriver.Builder().forBrowser('chrome').build()
})
after(async () => await driver.quit())

This opens a browser and waits for driver commands. But before starting with the test, we need to setup (and teardown) Applitools Eyes:

We create some new Eyes (line 5), and open them (line 8) — cute terminology, isn’t it? Don’t forget to get an API Key from Applitools, which we’ll talk about in the next section, and give it to the Eyes (line 6).

Now that we’ve setup the browser and eyes, we can write the test, which is very similar to our E2E test:

Comparing it to the E2E test in the previous post in the series, you can see that this one’s similar, but shorter. The main difference is that in this code, the validation of specific elements is replaced by one simpler line:


await eyes.checkWindow(‘<check-name>’)

In the E2E test, we did this instead of this one liner:

We waited, using retries, for the page to “stabilize”. But doing Visual Testing, you don’t need to wait for the page to visualize — eyes.checkWindow will do this for you!

eyes.checkWindow will take a screenshot of the page, and compare it against a baseline image from a previous test (see how it works in the section “Running the Visual Test” below). If the comparison says that the new image is equal to the baseline, then the test succeeds, otherwise it fails.

Visual Testing as A Better Validation Tool in E2E Tests

And this huge benefit of doing visual tests — stabilization is dealt with by the system. Moreover — you’re not just checking one element or two elements — you’re checking the whole page in one assertion. You will probably find problems that you weren’t even looking for!

In general, it seems that visual testing should be the only way you do assertions in E2E tests. Unfortunately, visual assertions are currently somewhat slower, so you need a good mix of regular assertions that check a specific element and visual assertions that check the whole page.

Remember — there are no panaceas: no one type of test can do everything! A mix of various types of testing will give you the best balance, and it takes experience in testing to determine what this mix is. So start testing now! Yes, testing takes time and commitment. But once you start testing, you really can’t go back.

Running the Visual Test

How do we run the visual test and see the outcome?

npm test will skip the visual tests if you do not supply an API key in the environment variable APPLITOOLS_APIKEY. To get an API key so that you can also run the test, go to the Applitools site, and register. In your account in the Applitools UI, you will find the API Key. Copy it to the clipboard, and then run the test with (on Linux/MacOS):


APPLITOOLS_APIKEY=<the-api-key> npm test

or, if you’re using Windows, use:


set APPLITOOLS_APIKEY=<the-api-key> && npm test

Once you have that, you can run the test. The first time, the test will fail with the error EYES: NEW TEST ENDED.

This is because there is no baseline to test against. On the other hand, if we look at the Applitools Eyes interface, we will see this:

From Applitools point of view, the test passed, because this is a baseline, and it assumes that the baseline is correct. You can make it “fail” by using the interface “Like/Unlike buttons” for each screenshot.

The second time you use npm test, the test will succeed:

And Applitools UI will show this too:

If we explicitly fail the test, by (for example) clicking 43 * 3 instead of 42 * 2

,

the test will fail, and the Applitools UI will show the test and highlight the difference:

Fixing the “bug” will again make the test pass, both in Mocha and in Applitools.

This Concludes

This concludes the series about testing frontend code. Feel free to comment the article and add your thoughts.