Mobile Visual Test Automation with Applitools Eyes and Appium

In this article, I explained how to do automated mobile visual testing with Applitools and Appium. In the web world, there are many solutions for visual regression testing such as PhantomCSS, Webdrivercss, Huxley, Wraith, and much more. With Applitools Eyes you can do a visual comparison of both web, mobile web, and mobile native apps. You can use the Eyes class’s functions in your selenium codes.

I explained how to do Appium setup and how to write mobile automation code in this article. That’s why I do not want to go into those details in this article. I want to focus on how to integrate Applitools Eyes with Appium and do a simple visual mobile test automation.

First, open Genymotion Emulator and create a new device. Installation of Genymotion is described on this page. You can select personal use to use genymotion and you need to install Oracle VirtualBox.

When you open the GenyMotion, you will see a screen like the below based on the latest version. I did these steps on mac but the steps are similar on Windows as well.

Note: If you have a .apk with ARM-based, then you should use Android Studio and create a virtual device with API 30 as shown below.  In this way, you will not get “failed to extract native libraries, res=-113” error. You can check this here: https:/how-to-run-arm-apk-on-x86-systems/

For this example, I will go with Genymotion and simple x86 based bitbar.apk.

Then, you can select a phone model as shown below. 

Continue with the Install button.

Wait for installation.

Then, click start and lets wait the emulator starts.

And, here we go! Galaxy 10S emulator started. :-)

Now, open Appium and check settings.

We will use sample bitbar.apk. First, you need to put this apk under C:\ApplitoolsApk\BitbarSampleApp.apk folder and then set this path in Appium settings.

Then, we need to install Appium and start it. The installation is described in this article.

Then, we need to move the apk to genymotion as shown in the below screenshot. 

Bitbar will be installed as shown below.

For the demo, Galaxy S10 is so new to the demo app that’s why I did the demo with Galaxy Nexus 4 device.

Drag and drop the bitbar.apk to the device and then updated the below setting, check Unknown Sources.

Now, we can write our test code.

Open IntelliJ and click File -> New Project and select Maven and click Next.

Then, write:

GroupId: AppliToolsEyesDemo

ArtifactId: AppliToolsEyesDemo

And click Next.

Write “AppliToolsDemo” as the project name and click “Finish”.

Then, open pom.xml and copy-paste the below codes.

Project GitHub Page: https://github.com/swtestacademy/MobileVisualTestingApplitoolsEyes

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>AppliToolsEyesDemo</groupId>
    <artifactId>AppliToolsEyesDemo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M4</version>
                <configuration>
                    <useSystemClassLoader>false</useSystemClassLoader>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.testng/testng -->
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.4.0</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/io.appium/java-client -->
        <dependency>
            <groupId>io.appium</groupId>
            <artifactId>java-client</artifactId>
            <version>7.5.1</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.applitools/eyes-selenium-java3 -->
        <dependency>
            <groupId>com.applitools</groupId>
            <artifactId>eyes-selenium-java3</artifactId>
            <version>3.199.1</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>3.141.59</version>
        </dependency>
    </dependencies>
</project>

Click test -> java and right click and create a new Java Class and give “AppliToolsDemo” as a class name then click the OK button.

Right click “AppliToolsEyesDemo” folder and create a new File and give “TestNG.xml” as a name.

After that, open “TestNG.xml” file and write the below code it.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="AppliTools Demo Tests">
<test name="RunTest" preserve-order="true">
    <classes>
        <class name="AppliToolsDemo" />
    </classes>
</test>
</suite>

Now, we can write our test code but before that, I want to summarize our test scenario:

• Open BitbarSampleApp.apk
• Visually check that the start screen is visually as expected?
• Click the second option “Use Testdroid Cloud”
• Visually check that the Second Option is clicked?
• Write “SW Test Academy” to the text bar
• Click Answer button
• Wait until the second page appears
• Visually check that the answer is correct
• Close the driver.

As you can see in the scenario, we will make 3 visual comparisons. When you run your test, at first, AppliTools Eyes takes screenshots of the checkpoints and puts them under our test as baseline images. When you run your tests second, third, fourth times, and so on, AppliTools Eyes’s “eyes.checkWindow()” method will compare the actual screen with the baseline screen with given comparison match level as STRICT, EXACT, LAYOUT, CONTENT. In this article, we will make a STRICT comparison.

For match levels please read below page:

https://help.applitools.com/hc/en-us/articles/360007188591-Match-Levels

Now, Let’s log in to Applitools cloud test management panel from http://www.applitools.com/ If you do not have an account, first you need to create a new account.

When you logged-in https://eyes.applitools.com at the right corner, you will reach your profile details and you will get your API key from here.

At left pane, you can see your test runs highlighted with green when they passed and red when they failed. Also, you can all checkpoints when they passed they comprises of green + sing and when they failed, you see red + sign. 

When you open an image, you can reach the image control panel and in this panel, you can many operations as shown below image.

I think this is enough for Applitools Eyes control panel. Now, let’s start to write our test code. I wrote detailed inline comments in the code.

import com.applitools.eyes.MatchLevel;
import com.applitools.eyes.selenium.Eyes;
import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.remote.MobileCapabilityType;
import java.net.MalformedURLException;
import java.net.URISyntaxException;
import java.net.URL;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class AppliToolsDemo {
    final   String        APP_NAME  = "BitbarSampleApp";
    final   String        TEST_NAME = "BitbarDemo";
    final   String        DEVICE    = "Galaxy Nexus 4";
    private AndroidDriver driver;
    private WebDriverWait wait;
    private Eyes          eyes;

    //Do the Setup before tests
    @BeforeClass
    public void setUp() throws MalformedURLException {
        //Setup of Appium
        DesiredCapabilities caps = new DesiredCapabilities();
        caps.setCapability(MobileCapabilityType.DEVICE_NAME, "Galaxy Nexus 4");
        caps.setCapability(MobileCapabilityType.UDID, "192.168.56.102:5555"); //DeviceId from "adb devices" command
        caps.setCapability(MobileCapabilityType.PLATFORM_NAME, "Android");
        caps.setCapability(MobileCapabilityType.AUTOMATION_NAME, "UiAutomator1"); //UIAutomator2 is only supported since Android 5.0
        caps.setCapability(MobileCapabilityType.PLATFORM_VERSION, "4.4");
        caps.setCapability("skipUnlock", "true");
        caps.setCapability("app", "/Users/onur/OneDrive/swtestacademy/stuff/bitbarapk/BitbarSampleApp.apk");
        caps.setCapability("noReset", "false");
        driver = new AndroidDriver<MobileElement>(new URL("http://127.0.0.1:4723/wd/hub"), caps);
        //Set Wait Time
        wait = new WebDriverWait(driver, 30);

        //Setup of Applitools Eyes
        eyes = new Eyes();
        //Set API Key of Eyes
        eyes.setApiKey("YOUR API KEY");
        //Set Match Level
        eyes.setMatchLevel(MatchLevel.STRICT);
        //Set host operating System as our device
        eyes.setHostOS(DEVICE);
    }

    @Test
    public void bitBarDemoTest() throws MalformedURLException, InterruptedException, URISyntaxException {
        //Open Eyes and start visual testing
        eyes.open(driver, APP_NAME, TEST_NAME);

        //Visual check point #1
        eyes.checkWindow("Start Screen");

        //Click second radio button

        driver.findElement(By.id("com.bitbar.testdroid:id/radio1")).click();
        //"/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.ScrollView"
        //                + "/android.widget.LinearLayout/android.widget.LinearLayout[1]/android.widget.RadioGroup/android.widget"
        //                + ".RadioButton[2]"

        //Visual check point #2
        eyes.checkWindow("Answer selected");

        //Write "SW Test Academy" to text box
        driver.findElement(By.id("com.bitbar.testdroid:id/editText1")).sendKeys("SW Test Academy");

        //Click answer button
        driver.findElement(By.id("com.bitbar.testdroid:id/button1")).click();

        //Synchronization after click. Wait until "You are right!" text appear on second screen
        wait.until(ExpectedConditions.presenceOfElementLocated(By.id("com.bitbar.testdroid:id/textView1")));

        // Visual validation point #3
        eyes.checkWindow("Answer is correct");

        // End visual testing. Validate visual correctness.
        eyes.close();
    }

    @AfterClass
    public void teardown() {
        //close the app
        driver.quit();
        //Abort eyes if it is not closed
        eyes.abortIfNotClosed();
    }
}

Before running our code please make sure that your Appium server and Genymotion emulator running. If these preconditions satisfied, then you can run your test code. After that, the Bitbar application will open and a new bitbar test created in the Applitools control panel as shown below.

At the first run, Applitools will save all images at checkpoints as baseline images.

We can also add some ignored comparison regions with rectangular selection. It is shown below.

For the second run, if your test will fail, Applitools Eyes will highlight problematic mismatches and your test will turn a red color.

When your test passed, your test is highlighted as red as all the checkpoints are displayed without mismatch error.

Thanks,
Onur Baskirt

Onur Baskirt

Onur Baskirt is a Software Engineering Leader with international experience in world-class companies. Now, he is a Software Engineering Lead at Emirates Airlines in Dubai.