Autify and the Future of AI Testing (Nauman Mustafa)

AI has been revolutionizing every industry for the last decade, and the test automation industry is no different. The Autify ML team explores the cutting AI research and how that can be used to accelerate test automation. In this blog post, we are going to see how Autify is already using the latest developments in AI for some of its current and upcoming features.

MLUI (Mobile)

When interacting with an app, the user only sees and interacts with the screen (not the source code). However, the majority of the test automation tools rely upon interacting with the app’s source code to generate and execute test scenarios. We want to provide our customers with test systems that are similar to how humans (QAs) test applications. And this is the motivation behind MLUI.

MLUI (acronym of Machine Learning-based UI extraction) is one of Autify’s flagship AI projects. It uses state-of-the-art deep learning techniques to extract UI element information purely from the screenshots. This allows us to provide our customers with more human-like test automation.

Imagine a test scenario where a button becomes hidden due to a mistake in the code. Since the button still exists, an automation tool might still be able to click that button. However, MLUI only looks at the screenshot of the app. So, if a button is not visible, the MLUI-based testing would not be able to locate it and thus test would fail.

Intelligent Assertion

Assertions are a very crucial part of test automation. When a (human) tester is quality checking different parts of a web page, sometimes he/she would assert the exact value of an element like the name of the company, and other times he/she would look at the category of the value like if the price is being correctly displayed or not. In most of the test automation tools, asserting the exact value of an element is simple but asserting the category is not possible.

Intelligent assertion provides extraction of categories and their relevant information without having to write complex regexes. Imagine a scenario where we want to test whether a particular element is displaying date or not. Normally, one would have to use a DateTime parser or write a regex if the format is slightly more complex. But if the format is changed, even if the element is still displaying the date, the test would fail. And regex also needs to be updated.

Intelligent assertion (coming soon in Autify) allows us to assert whether some element is dated or not without writing and maintaining complex regexes. It supports many categories besides DateTime, like price, number, person name, etc. Following is an example of how the Autify Intelligent assertion system categorizes different texts and extracts the relevant information into a clean and parsable format.

Test Scenario Generation

To achieve market leadership, we at Autify want to boost the productivity of QA by