The year 2017 saw an impressive number of important events as far as UI automation testing tools go.
In 2017, we witnessed the end of developer support for Selenium IDE and PhantomJS. We’ve also had a full year to evaluate how the 2016’s acquisition by Micro Focus has influenced UFT. Finally, the releases of Angular 5 and React Fiber have introduced new challenges for QAs, making GUI testing tools even more relevant.
Given all these news, what automation frameworks and tools will become the mainstream options for guarding the user interface against bugs in 2018? Read on for an overview of top web automation solutions across three categories:
|Protractor||UFT Pro (LeanFT)||Ghost Inspector|
*Yep, thats us — and we’ve got awesome features to back up our claim for leadership.
UI automation frameworks. Selenium WebDriver, Protractor, and Cucumber
Code-based testing frameworks have their drawbacks, but they will persist driving UI automation in teams that have enough time to write UI tests. For these teams, Selenium, Protractor, and Cucumber will remain the preferred options 2018.
Selenium WebDriver: old but gold still got it
In 2017, people still consider it a must for every specialist involved in GUI testing to learn Selenium. And there’s a good reason to. Selenium WebDriver underlies the W3C WebDriver standard, it’s native to major browsers, and it’s used in all higher-level automated testing frameworks.
Being low-level, Selenium requires a solid grasp of good testing practices. The low-level nature of this tool also makes Selenium less productive than higher-level tools. Inspite of these two facts, Selenium has a reputation of the “golden standard” of automation testing of the GUI, which is why many teams still choose it over dedicated high-level frameworks. Specifically, many React teams prefer to test web applications with Selenium.
Protractor: test automation framework for Angular apps
While React doesn’t seem to have a UI testing champion, the situation is different in the Angular realm. When it comes to the automated testing of Angular-based web applications, Protractor enjoys the status of the go-to solution. Specifically, software testing engineers name several advantages of Protractor:
- Protractor is the brainchild of the people who wrote AngularJS.
- The framework has a clever way of handling Angular waits.
- Protractor’s accessors make for an effective Angular-specific locator mechanism.
- It’s always nice to work with a framework that ships with a test runner out of the box. In case with Protractor, the test runner is Jasmine, and there are many Jasmine fans out there.
Cucumber: BDD champion for UI
BDD frameworks stand apart from other test automation solutions. If you buy into the idea of “business-facing” Gherkin scripts, Cucumber is pretty much the only game in town. Capybara, the web-based automation framework for Cucumber, is also something you’d want to use for BDD-centric acceptance testing.
Desktop automation testing tools for enterprise: TestComplete, UFT Pro (LeanFT), and EggPlant
It’s not uncommon for large enterprises to gravitate towards all-in-one suites of automated testing tools that work for both web and mobile testing and have every feature one can imagine. The all-in-one nature of these products also manifests itself in how they automate tests.
IBM Rational Functional Tester, for instance, combines coded tests with screenshot comparison and “storyboard testing” which uses natural language for test editing. In the meantime, Telerik Test Studio emphasizes the comprehensive coverage of GUI, API, Load, and Performance testing.
While both of these two GUI testing tools offer a slew of neat features, this review will focus on other solutions. Specifically, we will zoom in on the tools featured in the Gartner’s Quadrant for automated software testing. The table below offers a comparison of TestComplete, UFT Pro (LeanFT), and EggPlant Functional.
|Automation testing tool|
|Developer||SmartBear||Micro Focus (the app was previously owned by Hewlett-Packard)||Testplant Inc.|
|Position in the Gartner’s Quadrant for 2017||Challengers||Leaders||Visionaries|
|Runs on||Windows (desktop app)||Windows, Mac, Linux||Windows, Mac OS X, or Linux|
|Web testing||Chrome, Firefox, Internet Explorer/Edge, Safari||Chrome, Firefox, Internet Explorer/Edge, Safari||Chrome, Firefox, Internet Explorer/Edge, Safari|
|Mobile testing||iOS, Android||iOS, Android||iOS, Android|
|Visual testing||Image comparison||No (image comparison and record-playback are available with a UFT license)||Object recognition, image search, image recognition, and text recognition|
|Selenium support||Yes, via Java’s JUnit||Yes, via Object Identification Center, Objects locators & utilities||No|
|Supported IDE||Microsoft Visual Studio, Eclipse||Microsoft Visual Studio, Eclipse, IntelliJ||Microsoft Visual Studio, Eclipse|
|CI support||Jenkins||Jenkins||Jenkins, Bamboo|
|Known issues||Slow to support new IE versions, weak with Java applets||Large memory consumption footprint, extensive infrastructure||The software is still buggy (even though the TestPlant team fixes them fast).|
|Pricing model||Node-Locked License tied to a machine
Floating User License tied to a user account
|Seat-based license / 3 months
Custom pricing for enterprises
|Pricing information only available via contacting TestPlant|
|Free version||Free trial||Free trial||Free trial|
Cloud testing solutions for codeles and low-code automation: Screenster, Ghost Inspector, UseTrace
In the last couple of years, a slew of web services and platforms emerged to enable efficient low-code automation of UI and functional testing. Like Selenium-based frameworks, these testing tools drive the browser to emulate user interactions with websites and web apps. Unlike Selenium and similar solutions, these tools test the whole UI instead of targeting individual elements.
Even though people sometimes compare cloud testing platforms to Selenium IDE, their functionality goes beyond simplistic test recording and screenshot comparison.
For instance, modern cloud-based UI testing tools are able to recognize on-page objects and UI areas with dynamic content. They can automatically ignore visual differences caused by rendering and handle timeouts with little to no human input. Being web-based, these tools also feature collaboration functionality. Finally, they support coded test scripts and continuous integration.
The table below will compare the three tools that offer advanced functionality for powerful test automation.
|Functionality||Functional and smoke testing (UI)|
|Visual verification||Yes||No (Screenshot comparison only)||No (Screenshot comparison only)|
|Management of dynamic UI areas||Yes||No||No|
|Built-in Selenium commands for in-browser coding||Yes||No||Yes|
|Importing / exporting tests||Importing Selenium tests||Exporting tests as JSON||Importing and exporting Selenium tests|
|Parameterizable tests / Variables||Yes||Yes||Yes|
|Support of different screen sizes||Yes||Yes||Yes|
|Creating a basic test takes < 3 minutes||Yes||Yes||Yes|
|Fast test editing||Yes||Yes||Yes|
|Supported browsers||Chrome, Firefox, IE||Chrome, Firefox, IE||Chrome, Firefox|
|Collaboration functionality||Shared access to dashboards and test editing||Shared access to dashboards, tagging tests||Shared access to dashboards and test editing|
|Reports||Screenshots and test steps||Screenshots and test steps||Video, screenshots, test steps console information|
|Notifications||Email, Flowdock, HipChat, JIRA, Slack||Email, Slack, Webhooks, HipChat, PagerDuty|
|Installation/setup footprint||Fully web-based||Fully web-based||Requires a dedicated Chrome plugin|
|Test server||Cloud-based or local||Cloud-based; NGrok tunnels to local machines||Cloud-based; NGrok tunnels to local machines|
|Concurrent test runs||Yes||Yes||Yes|
|CI support||Jenkins, TeamCity, Travis, Bamboo||Jenkins, Bamboo, CircleCI, Codeship, Visual Studio Online||Jenkins, CircleCI|
Automated functional testing of the GUI with Screenster
Screenster, Ghost Inspector, and UseTrace narrow down on test automation on the UI level. Instead of offering you the broad functional scope of enterprise platforms, these tools do one thing, but they do it really well. Here’s what our very own product Screenster can offer for powerful website and web application testing:
1. Pain-free test maintenance
Screenster enables you to record and execute automated tests without hand-coding. While this is the same sort of functionality that you get with some enterprise IDEs, there’s one major difference. The tests generated by the record-playback tools of enterprise IDEs typicaly come as auto-generated code which is often a pain to handle.
Instead of making you fumble through autogenerated test code, Screenster allows for quick and fully codeless test editing. Given how frequent the UI changes can be, you’ll appreciate how much sense this feature actually makes.
2. Self-healing tests with smart locators
When creating the baseline of the UI, Screenster automatically builds complex locators that include all HTML attributes and structure-based selectors for every element. Should a UI change of any kind break one of the selectors, Screenster will automatically substitute it with another one. This way, you can perform test automation without having to worry about UI updates.
3. Intelligent visual testing with verification of visuals, DOM, and content
Screenster automates UI testing by capturing the DOM structure and the content of the page, in addition to taking screenshots of every UI state. All of this information is stored as baselines of each UX step, and Screenster checks new UI versions against these baselines during every test run.
When it comes to visual testing,Screenster does more than mere screenshot comparison. The platform distinguishes between style changes and content updates, providing you with valuable information on the nature of every UI bug. When looking at screenshots, Screenster sorts out the differences associated with anti-aliasing. It also detects UI elements with dynamic content and provides you with a simple functionality for managing these regions.
There’s a lot that can be said about the advantages of low-code cloud-based solutions, but giving them a try is worth a thousand words. Check out our demo, and you will see if cloud-based solutions have what it takes to rank among the top automation testing tools of 2018.