Automation Engine Comparison !

Introduction

Automation market is full of tools that support automation for web applications along with the support for mobile automation as well. So, the aim of this post is to collect useful features for each listed above engine and present it in some comparable form. Thus we have:

• CasperJS
• AngularJS E2E
• Jasmine
• Sahi
• Ranorex
• Geb
• Selenium WebDriver
• Watin

This list isn't complete as there can be many other similar engines which are simply less popular. All of them have some common set of supported features but there're some restrictions and abilities specific to the actual engine. Key features to be mentioned are:

• Documentation availability

• Auto-complete

• Reporting

• CI Server Support

• Browser Support

• Language Complexity

• Setup Complexity

• Screenshots

• Technology Support (like SPA, Ajax)

                                                    

Each feature has some sub-features which reflect some specific part of functionality. And those little things make each engine different and unique in some cases. So, let's take a closer look at each feature. In order to make some comparison characteristics let's introduce the scale of support quality. Each feature can be values with the grade from 0 to 3 by the following criteria:

Grade	Criteria
0	No support at all
1	Very Less Support
2	Partially Support
3	Full-featured support

Documentation Availability

One of the key factors demonstrating the maturity of the engine is the documentation availability and its completeness. Indeed, when we start using new engine the first thing we usually do is documentation reading where we can take some examples, list of features etc. Actually, the documentation is the part of software product (this is one of the differences between application and software product), so if there's a lack of documentation it indicates that product isn't complete. Additional source of documentation is various online materials we can find in Internet from specialized resources, blogs etc. So, when I estimate the grade of documentation availability I check the following criteria:

1. Documentation is available in general (it makes grade 1 at once)
2. Every feature is described and has examples (if it fits it makes grade 2)
3. There're additional well-grown resources (forums, blogs, user groups) where we can find additional information about the engine

These are external resources which can be found pretty quick (for me it took less than a minute to find them). So, given all the above information we can evaluate the grades for each engine. The table will look like:

Engine	Documentation availability
CasperJS	3
AngularJS E2E	3
Jasmine	3
Sahi	3
Ranorex	3
Geb	3
Selenium	3
Watin	3

Auto-complete

I'd say it's one of the most useful features while writing tests using natural language instructions. You just have to type some key parts of entire phrase and select the most appropriate option. If there's such ability it's definitely great. Unfortunately this feature is quite rare and mostly represented as some IDE plugin for writing tests. Since most of these tools are bounded to the language implementation like in CasperJS you can write your unit tests using Coffee Script & Java Scripts only, on the other side when it comes to language support Selenium & Watin actually wins the race which can be easily integrated to the IDE’s like Eclipse & Visual studio that supports Auto-Complete feature. Selenium’s most useful features have been its support for multiple programming languages, and is support for testing a wide variety of browsers, whereas Watin is bounded to only two browsers that are IE & Firefox. Since most of the mentioned engines are bounded to Java scripts only which is a dynamic language and it least supports auto- complete/ intelligence.  So, support table for this feature looks like:

Engine	Auto-complete support
CasperJS	1
AngularJS E2E	1
Jasmine	1
Sahi	2
Ranorex	2
Geb	2
Selenium	3
Watin	3

As it's seen from the table the auto-complete is the weakest feature at the moment for most of the engines.

Reporting

Reporting is one of the most important parts of the testing and automation as well. It's not enough to say that test is passed or failed. Very often we should identify where it fails. It's vital for functional tests where we perform sequence of steps with some interim checks and such test can fail everywhere. And we should be able to say what the actual reason of fail was. We have the built in reports in Sahi, Ranorex, Geb & E2E on the other hand Selenium & Watin can easily integrate with any other reporting extensions like MSTest, Nunit, Junit & TestNG. However when it comes to CasperJS & Jasmine their execution usually happens on the headless browser in Jasmine using HTML Reporter, we can provide reporting for each spec and each suite & CasperJS provides Xunit XML report. So, support table for this feature looks like:

Engine	Reporting
CasperJS	2
AngularE2E	3
Jasmine	2
Sahi	3
Renorex	3
Geb	3
Selenium	3
Watin	3

CI Server support

Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. These days almost every tool has the CI server support for tools like CasperJS & AngularE2E we can easily export the results of a test suite to an Xunit XML file, which is compatible with continuous integration tools such as Jenkins. On the other side if talk about other tools like Jasmine, Sahi, Ranorex , Geb, Selenium & watin all these tools can be integrated with the CI server support. We are currently using Selenium WebDriver for our ongoing projects which are already integrated to the CI server (Jenkins) & we get the daily execution report for our automation workflows.

Engine	CI server support
CasperJS	3
AngularJS E2E	3
Jasmine	3
Sahi	3
Ranorex	3
Geb	3
Selenium	3
Watin	3

Browser Support

Browser support now a day is like a challenge for all organizations to perform Cross browser compatibility testing. If we think of this ability we would point to the automation tool that can perform cross browser compatibility testing in an efficient way & surely we would want such tools that can be run over multiple browsers with least effort. CasperJS & Jasmine engines will be run over the headless browser whereas AngularE2E scripts can be run in parallel on multiple browsers like Firefox, Chrome & IE, Geb builds on the WebDriver browser automation library, which means that Geb can work with any browser that WebDriver can, the existing drivers are the ChromeDriver,InternetExplorerDriver, FirefoxDriver, OperaDriver and HtmlUnitDriver. For more information about each of these, including their relative strengths and weaknesses, please follow the links to the relevant pages. There is also support for mobile testing via the AndroidDriver, OperaMobileDriver and IPhoneDriver. Sahi tool is a record & play web automation tool which also has multiple browser support open source tool. WatiN is developed in C# and aims to bring you an easy way to automate your tests with Internet Explorer and FireFox using .Net.

Engine	Browser Support
CasperJS	2
AngularJS E2E	2
Jasmine	2
Sahi	3
Ranorex	3
Geb	3
Selenium	3
Watin	2

Language Complexity

While choosing an automation tool we always want that our tests can be written in a language which we are familiar with so that least effort shall be required in writing our tests. If the language complexity is high then the effort will be very high for writing our tests. Now, if we talk about CasperJS we can write the tests in Coffee scripts & Java scripts those tests eventually would run over headless browser kit viz. PhantomJS. In AngularE2E & Jasmine these tools are widely used for Java scripts unit testing so it is quite obvious that we can achieve unit testing with the help of Java Scripts only. On the other hand Sahi is a record & play tool which generates the code as soon as we stop it’s recording. If we talk about Geb, the tests will be written with the help of Groovy scripts.  Selenium has its support for multiple programming languages but Watin is bounded to only .net Languages since it is developed in C#. So, the support table for this feature looks like:

Engine	Language Complexity
CasperJS	2
AngularJS E2E	2
Jasmine	2
Sahi	3
Ranorex	3
Geb	2
Selenium	3
Watin	3

Setup Complexity

While starting off with a new tool we always want that less set up configuration should be required in order to run the unit test. If the setup is huge & it is going through lot of complex configurations then it will surely affect the “Time required” factor. If we look at the engines there are many in the list that require external server to be set up on the system whereas there are others which is node independent & platform independent. Selenium WebDriver we can set up with any popular IDE like Eclipse & Visual Studio. Jasmine we can easily set up after downloading its library. CasperJS runs over headless browser webkit viz. PhantomJS so we can easily set up this tool after appending the path environment variable. Sahi tool we can easily set up after integrating it to the browser. Ranorex & Geb also do not require complex set up. Watin tool we can easily integrate it with Visual studio after adding its reference to the project. So, support table for this feature looks like:

Engine	Setup Complexity
CasperJS	3
AngularJS E2E	2
Jasmine	3
Sahi	3
Ranorex	3
Geb	3
Selenium	3
Watin	3

Screenshots

Screenshot is indeed one of the most important part of an automation tool. It gives more clarity to our automation scripts since the automation efforts does not require much human efforts so we will surely go through lot of challenges to know where exactly our tests was failed, in reports we would get the failed or passed log but we would not get a clear idea that why the test was failed what exactly was happening on the UI, was the test failed due to automation code or was it failed due to connection or any application issue so all these questions will be answered by a clean screenshot that will explain the exact behavior running at that time. AngularE2E and Jasmine does not support screen shot feature whereas CasperJS & Sahi has the ability of taking the screenshots. In Ranorex the Report class also supports a method to create screenshots to log images from applications or unexpected dialogs. Every Ranorex test should handle exceptions and generate at least one screenshot from the system's desktop. Geb uses the ScreenshotAndPageSourceReporter implementation that dumps the page source as html and attempts to take a screenshot as a .png file. Selenium & watin engines are well equipped with this ability of taking the screenshots. Watin uses CaptureWebPageToFile function to get the screenshot. So, support table for this feature looks like:

Engine	Setup Complexity
CasperJS	3
AngularJS E2E	0
Jasmine	0
Sahi	3
Ranorex	3
Geb	3
Selenium	3
Watin	3

Technology Support

When it comes to picking an automation engine we would always want that the engine would fully support almost every technical platforms, it’s not a good practice that down the time your application would introduce some new technical stuff which your existing automation tool cannot automate, it would surely add lot of pain to the developers/testers because no one wants to work on different automation tools for a single project. Hence while picking a tool we should first confirm whether the tool can support our existing technical platform or not. We have many new technologies that have been introduced in the market like SPA (using Angular), we cannot confidently take this call that our automation tool will support this platform & can easily automate the things. Support table for this feature looks like:

Engine	SPA & Ajax Compatibility
CasperJS	3
AngularJS E2E	3
Jasmine	3
Sahi	3
Ranorex	3
Geb	3
Selenium	3
Watin	3

Conclusion

If you’re a CasperJS user & you think of automated web testing as a car. CasperJS is the steering wheel & dashboard which interact with the engine. Allowing CasperJS users to use AngularE2E & Jasmine is like providing an additional engine choice, but keeping the steering wheel and dash board the same because the main objective of these three tools are almost same but if we look at the above records among these three Jasmine wins the race. However when it comes to the UI web automation part we believe Selenium & Watin are ahead of the above mentioned tools because of their multiple browser support & their strong reporting makes them the top priority, only flaw in Watin is it can run only on two browser viz. IE & FF & supports only .net languages. Whereas Ranorex is a paid tool & for Geb one has to learn groovy scripting. If we talk about Sahi its interface is very confusing & it is least developed. Hence as per the final conclusion for UI automation the best available tool in the market is Selenium & if an organization wants some of the automated unit tests should be developed by the developer itself then Watin can come in the picture because of its rich API. Hence the overall ratings are:

Engine	Documentation availability	Auto-Complete	Reporting	CI server support	Browser Support	Language Complexity	Setup complexity	Screen shots	Technology Support	Overall
CasperJS	3	0	2	3	2	2	3	3	3	21
AngularE2E	3	0	3	3	2	2	2	0	3	18
Jasmine	3	0	2	3	2	2	3	0	3	18
Sahi	3	2	3	3	3	3	3	3	3	26
Ranorex	3	2	3	3	3	3	3	3	3	26
Geb	3	2	3	3	3	2	3	3	3	25
Selenium WebDriver	3	3	3	3	3	3	3	3	3	27
Watin	3	3	3	3	3	3	3	3	3	26