Let’s suppose that you’re a hospital, and have just realized that you need an automated solution for your testing. Why automated? Because automation is the only way to get the excellent coverage you need without requiring days or even weeks of testing time. We want to test functionality and integration and performance, on a system that is heavy on database content, including files of very large size (medical images).
When automating your testing, selecting the right tool for your needs is the most important early decision you can make. This foundation commits you to a path, and the test scripts you produce will be linked to your choice. All automation testing tools have different benefits and deficits, but a deficit that is irrelevant to your needs is not really a deficit, is it? So, let’s take a look at 3 popular tools (QTP/UFT, Selenium, TestComplete) and when each of them may be the preferred choice for you, or the most obvious choice to avoid. They each complement different personalities. They are all popular, allow front-end functional and regression testing, and allow you to code directly at the element level. And please do not feel like these are the only good automation tools out there.
HP UFT (formerly QTP) benefits: This uses VBScript, one of the easier languages to learn. It provides you with easy access to the UI layer, the service layer, and the back-end database layer. It is easy to use for navigating, validating, and generating reports and log files. You have the option of using GUI-based tools throughout the suite. It supports data-driven testing, meaning that data values can be input or output from external tables instead of being hard-coded in the scripts. The scripts can be edited at the keyword or source code levels, enabling access by scripters of low or high programming abilities. It has a logical built-in suite of supporting tools (as compared to Selenium, where you need to assemble and install separate plug-ins if you want to do common things like logging and reporting). And, it can be used for non-UI testing like file system operations and database testing.
HP Deficits: You’re limited to VBScript as your programming language, which is object-based but not object-oriented (its documentation is fairly poor too). This is the most expensive of the 3 automation tools, by a lot, including recurring licensing fees. It does not work on non-Windows, and is limited in which Windows browsers (and versions of them) it supports. QTP’s Object Repository is neither dynamic nor customizable, unlike TestComplete’s Name Mapping. Execution time is on the high side, as is hardware resource consumption during execution. It requires a different HP product for mobile device support. It cannot run multiple/simultaneous tests per machine, and is not designed for cloud testing.
Selenium (plus Appium) benefits: Selenium is free with no licenses, and plays nicely with many third party frameworks and tools that complement it as free plug-ins. It lets you access platforms other than Windows (like UNIX, Solaris, Linux, OS X, iOS and Android), and likewise wins the “most browsers available” contest. It is a good fit for people who already have developer skills, and allow for a wide variety of languages to choose from. It is open source, so there are many libraries available, or you can create your own. It has a very large community. It can be installed on a server, and run like a client. It’s extremely configurable and highly reliable once working. It is well-suited for testing web and mobile apps (Appium expands into mobile testing by extending the Selenium API).
Selenium deficits: It is not designed for testing desktop apps. It does not come with supporting tools, so you need to build out a framework and customize it which is a time drain (for instance, it has no logger or report generator, although free plug-ins are available). It requires developer skills, and takes longer to create scripts. It has no image verification. It may run slower than other test automation solutions. Support comes only from a community, not a vendor. Some of the programming languages are complex and take a while to learn. It has no object mapping repository. It may require some tweaking when switching around inside the wide variety of browsers.
Smartbear’s TestComplete Benefits: TestComplete’s test scripts are easy to maintain due to the easy readability, so a high level of programmer knowledge is not required. It can be used on desktop, web, or mobile apps. You can use it for front-end (like functional testing) or back-end (like database testing). Tests can be created by record and playback, by straight coding, or a hybrid of both methods. Unlike UFT, TestComplete allows languages other than the older language of VBScript, like JavaScript. While not free (like Selenium), TestComplete is relatively inexpensive (compared to UFT), and has a one-month try-before-you-buy period for familiarization. It has excellent technical support. The XML comparer is easy to use. Scripts are easy to debug. Supports client server apps as well as web apps.
SmartBear’s TestComplete deficits: Adjustments are needed when dealing with different browsers. Once keyword scripts are translated into programming scripts, they lose synchronization. TestComplete uses older versions of Jscript, missing out on new features since added to the language in the past decade. It is difficult to create object-oriented reusable frameworks due to each script unit being independent from others.
Now that we have looked at these three options, let’s determine which is the best fit for our hospital-based testing. While Selenium is the cheapest, it does not work well out-of-the-box, and will not work on desktop apps, so we eliminate Selenium as a choice. We have a budget that makes QTP/UFT look rather expensive, so let’s start comparing UFT and TestComplete, with a bias towards TestComplete due to cost. UFT has problems with different browsers. UFT is limited to VBScript as a programming language. TestComplete has more responsive with problems. None of TestComplete’s deficits are deal-breakers – at worst, they are limitations to keep in mind. For these 3 automation tool solutions, TestComplete is the obvious choice.
