Visual UI Testing for Android Apps with Espresso and Visual Studio App Center

Visual Studio App Center

July 13th, 2018 0

This is a guest post from Daniel Puterman, engineering lead at Applitools.

Suppose you’ve built a new Android app and want to build some visual test automation scripts to ensure that it appears correctly on the devices it might run on. Not all the Android devices out there — just the top 20 for now. Suppose further that your app isn’t all that complex; just ten screens, with an average of ten visual elements each. And you won’t check every last attribute of those visual elements, just five: x, y position, height, width, and text.

Simple, right?

Unfortunately, no.

If you apply traditional functional testing techniques to visual testing, you just signed yourself up for writing test automation code with 10,000 checkpoints (= 20 devices * 10 screens * 10 elements * 5 attributes).

Of course, there’s no way you’re going to write 10,000 lines of checkpoint code. That could be more than all the code in your Android app itself. So you cut corners. Reduce your test matrix. Write less validation code.

But then your users find bugs in production, on a platform you didn’t test on.

It’s a dilemma that developers face regularly: Spend all your time writing test code, or scramble to fix production bugs.

Luckily, there’s a way to make it simpler with visual UI testing, which uses artificial intelligence to test an entire screen’s visual appearance with just one API call. With this approach, 200 lines of test code is all that’s needed to test on 20 devices and ten screens.

In this tutorial, you’ll learn how to build and run a visual user interface test of a native Android mobile app using the following tools: Espresso, Visual Studio App Center, and Applitools.

As an engineering lead at Applitools, I rely on visual UI testing for many apps and will cover the following topics:

  1. A quick introduction to visual UI testing
  2. Preparing the Ground
  3. Run your first mobile visual UI test on Visual Studio App Center

Get Started with Visual UI Testing

If you’re not familiar with visual UI testing, here’s a brief introduction.

Here’s the Hello World app from Applitools’ “Getting Started” guide for the Espresso SDK:

How would you test this? You could verify that a button exists and the button’s text is “CLICK ME!”. What about the “HELLO WORLD!” title? Again, you can verify the text, but what about the colors? What about the positions of the title element? The position of the button? What about the rest of the elements?

Without visual testing, you need to write a lot of validations (also called assertions or checkpoints), one for every property you want to test (content, position, color, etc.), and even then you cannot be sure the final result is displayed properly to the user.

Visual testing solves this problem easily. Let’s show you how using Espresso, App Center, and Applitools.

With a single “check” command, you verify all these properties for the entire content of the application. So your advantage is double:

  1. You don’t need to write a separate assert statement for each element.
  2. You get full coverage, not just coverage for specific elements.

The result of visual testing this application using Applitools are shown below. The known good baseline version is on the left; the new test build, on the right. Visual differences are detected automatically using Applitools’ AI and highlighted in magenta:     Since visual tests capture the entire screen — not just what’s in the viewport, but also content the user needs to scroll in order to view — we have validated the entire application while we ran the tests.

The command which validates the entire screen is:

 
eyes.checkWindow("Hello World screen");

That’s a lot easier than writing dozens of lines of checkpoint code, right?

If you agree, let’s dive into how to get this working with Espresso, Visual Studio App Center, and Applitools.

Preparing the Ground

You’ll need this software to get started:

You need to create accounts on:

Set up the above before continuing.

Downloading the Demo Application

For this guide, we’ve prepared a demo application which includes a visual Espresso UI test: https://applitools.bintray.com/Examples/app-center-eyes-android-hello-world.zip

You can find the source code for the application on GitHub: https://github.com/applitools/eyes-android-hello-world/tree/xamarin_cloud

If you already have an existing application with Espresso tests just waiting to get visual validations, the quickest way to start would be through the Applitools getting started guide. It walks you through adding the Applitools SDK for Espresso to your application.

Set up an Applitools Account

Create your Applitools account so you get your API key for running visual tests. You can find it on the menu in the top right corner of the Applitools Test Manager. Update the Applitools API key in the ExampleInstrumentedTest.java file of the demo application, line 42.

Set up App Center

Once you create your App Center account, you need to get credentials to run tests on App Center. First, you need to create your application, which allows you to describe the application you want to run on App Center. Click Add New > Add New App.

Our application is a native Android Java application, so we’ll mark it as such. The page that follows explains what to update in your application to run tests in App Center. In brief, the instructions are:

  1. Add dependencies to your build.gradle file
  2. Update your main activity:
    1. Add imports
    2. Add AppCenter API call, with the application secret.

In our demo application, we’ve already added the imports and the API call. However, if you would also like analytics on the app in App Center, you need to update the proper credentials in the API call with your App Secret (the 2nd parameter to the AppCenter.start method). The application secret should be updated in the MainActivity.java file:     Updating the application is not required to run wherever your test, however.

Create an App Center Test Run

Next, create a test run. Click New Test Run to do this.

Select the devices to use for the tests. (With App Center, you can test your app on thousands of real devices in the cloud.) Then select “master” as the test series, and Espresso as the test framework.

The next (“Submit”) step of creating a new test run explains how to run tests on App Center cloud. This requires app-center-cli.

It also specifies the command to run the test. Copy this command. You’ll use it soon.

Run Your First Mobile Visual Test on App Center

Next, build the application in App Center. To do this, click “build” under the “app” section in the “Gradle” menu in Android Studio:     To run the visual test, paste the command you copied from the last step in the “Create an App Center Test Run” section, in a command line of your choice, such as Windows PowerShell or macOS Terminal. The command has the following template:

appcenter test run espresso –app “” –devices “app-center-username/named-device-set” –app-path pathToFile.apk –test-series “master” –locale “en_US” –build-dir pathToEspressoBuildFolder

Where: —test-series is the test series to run in. Defaults to “master” —app-path is the path to the application APK file —build-dir is the path to the tests build folder. E.g., app/build/outputs/apk/androidTest/debug/

Visual Validation Results

After you run the test, open Applitools Test Manager to see the results.     As you can see, you have an Android test named “My first Espresso Android test!”, which includes two new steps. Both should be green, indicating they passed.

Here’s the visual UI test code sample (also on GitHub).

 
try {
        // Start a visual test in Applitools
        eyes.open("Hello World!", "My first Espresso Android test!");

        // Visual checkpoint #1.
        eyes.checkWindow("Hello!");
	  
	  // reportHelper creates labeled steps on App Center
  reportHelper.label("Check window method");

        onView(withId(R.id.click_me_btn)).perform(click());
        reportHelper.label("Click on ClickMe btn");

        

        // Visual checkpoint #2.
        eyes.checkWindow("Click!");

        // End the test in Applitools.
        eyes.close();
    }

In this example, two eyes.checkWindow commands are in the test. The first command executes when the application starts, the other after the user clicks a button. Each of these commands corresponds to a step (screenshot) you see in the Applitools Test Manager. Each call to the checkWindow validates an entire screen of the application in a specific state.

Pretty cool.

Wrapping up

At this point, you should have everything you need to start doing visual UI testing of Android applications using Applitools on Visual Studio App Center. To learn more:

  About the author Daniel Puterman is R&D director at Applitools – a leader in AI-powered visual testing and monitoring solutions. Daniel is a veteran programmer and team lead with experience in a wide variety of fields, from temperature loggers and web applications to location-based mobile applications and image processing.

Visual Studio App Center

Read next

Improved App Distribution is Here: Redesigned Tester Experience, Distribution Statistics, and More
Getting your release installed on a user’s device is the most important part of the distribution process. We've been working on incremental improvements to our tester experience since launching App Center in 2017, but after months of interviews and feedback from hundreds of customers, we’ve completely overhauled the user experience and visual design for our tester install portal.
Garrett Knoll
July 16, 2018

Feedback usabilla icon