Visual unit tests 💋

I hope that most of us use unit tests in day to day development because it saves a lot of time for us to make something new instead of repeat the same mistakes again. In this article, I’ll talk about our approach to deal with visual tests

Standard approach and problems 📌

All visual tests* based on the same approach.

• You should run the server with your application.
• You should write tests which run by NodeJS
• As a glue between our application and tests, we use Puppeteer or Playwright

// Average visual regression test code
const page = await browser.newPage();
// navigate to URL where located the code which we want to test
await page.goto('https://localhost:3000');
...
const image = await page.screenshot();
expect(image).toMatchImageSnapshot();

Pros:

• Access to the file system API and other OS related capabilities
• Run test code in debug is much easier because it’s NodeJS

Cons:

• Complicated setup and teardown. To run tests, we usually should set up two servers. One for code under test and another for tests itself
• Slow. Because test and code under test use different processes. Usually such communication goes over TCP that add communication latency
• Hard to debug code under test, which typically placed in the browser, which run in the separate process than test code
• Hard to write fixture or use cases which we want to test. Because test code and code which we want to test placed in different locations, it’s hard to maintain and write code which we want to test

All these problems and consequences prevent to write visual tests and add significant cost to maintain them. But visual issues haven’t gone anywhere.

Wishful thinking 🧚‍♀️

Let's think how the process of writing visual tests should look. The first code which we want to test should be placed close to the test itself. For example, like we do for unit tests. The second, we still should have possibility to make a screenshot and store it on the disk. So the code for visual test should look like this:

render(<UserProfile/>) // Code under test
const image = await page.screenshot(); // Take a screenshot
expect(image).toMatchImageSnapshot(); // Test with reference

Removing test, we also remove use case which we have tested

Solution 🥇

Now, when we have stated how visual test should look. We can try to find a solution. The first what we can see is that the shape of our visual test looks exactly like the unit test. The difference only in possibility to get a screenshot and magic matcher, which should get screenshot and much it with reference one.

For unit test we are using Karma. Karma is flexible, fast and solid tool to test frontend JavaScript in browser. I have used it in different projects and even had experience to set up it for very custom process with own build tool and own evaluation context. It gave me hope that we can do what we want using Karma. Because Karma run tests inside browser we are getting for free runtime which we need including the server which host our tests and engine which run them. Now left the hard part is to teach Karma how to make a screenshot and save it on the disk. We already can make a screenshot by using canvas API in the Browser, but how to store them 🤔. It would be ideal if we can have a special version of browser where we allow JavaScript more access than in regular browser. And…here gotcha ✨ what about Puppeteer and Playwright. They allow not only to grant more access but even better they allow you to expose custom functions to JavaScript runtime.
So to make it possible, we take Puppeteer or Playwright as launcher for our tests where we expose screenshot functionality and possibility to compare screenshot with reference image on the disk for our test code through expose function API. Also, we have adopted jest-image-snapshot for karma and jasmine to make API more familiar and voilà!

Conclusion 💭

The result is the boom of visual unit tests in our product. Because now write visual unit test as easy as write regular unit test.

Every approach has pros and cons and this one also not exception. The most notable cons, that all tests are running on the same page, so you should tear down properly it after each test to prevent test clash

But we love this approach because it already brings benefits of visual tests. And it does not require much effort. We pack it as a NPM package for karma. The repository itself contains tests, so you can open it via Gitpod or GitHub Codespace and play with it without additional setup.

* Different browsers

We are using only Playwright(Chrome) to simplify the process of installation, reduce headache with different browsers and improve developer experience to encourage writing visual unit tests. It works for us because it already brings benefits. But this solution allow you to use any browser or even remote server with browsers