Powered By Blogger

Search Here!

Wednesday, November 27, 2013

E2E Protractor for .NET

Protractor is an end to end test framework for AngularJS applications built on top of WebDriverJS. In this tutorial I’ll guide you through the process of getting it up and running. Protractor can run Jasmine and Mocha tests so feel free to choose the one that suits your needs best.
The .NET port of Protractor, an end to end test framework for Angular applications.
Protractor for .NET is built on top of Selenium WebDriver C# binding.
Get it from NuGet!
PM> Install-Package Protractor
Supports Microsoft .NET Framework 3.5 and higher..

Write Tests!

         
///
/// User Login.
///

private void UserLogin()
{
//Create Webdriver Instance
webDriver = new ChromeDriver();
webDriver.Manage().Timeouts().SetScriptTimeout(TimeSpan.FromSeconds(10));
IWebDriver ngDriver = new NgWebDriver(webDriver);
//Navigate Browser
ngDriver.Navigate().GoToUrl("https://mail.gmail.com");
//Enter User Name                    ngDriver.FindElement(NgBy.Input("user.Email")).SendKeys("dummy.user@gmail.com");
//Enter User Password
ngDriver.FindElement(NgBy.Input("user.Password")).SendKeys("qwertyuio");
//Wait For Element
webDriver.FindElement(By.CssSelector("button[data-qa-id*='login-button']")).Click();
//Click Save Button
webDriver.FindElement(By.CssSelector("span[class*='ico-profile']")).Click();       
}

Saturday, November 16, 2013

Changes in Visual Studio 2012 with respect to test features !

Welcome to the first post in this series which will give an overview of the differences in Visual Studio 2012 (VS12) retrospect to Visual Studio 2010 (VS10). In this particular post, we will cover the basic differences between VS2012 and VS2010 Test features, mostly with respect to navigation, running the tests, results and some features that have been deprecated in VS12. Let us now dive into this.

Test List Editor which is very commonly used to run tests in VS10 is deprecated in VS12 and is replaced by Test Explorer. Below screenshots show this difference with a red highlight.


Next question would obviously be, how do I run my tests from Test Explorer and how will this be different from VS2010. The experience of running the tests in VS12 is seemingly different from VS10 for a first time user. When you select Test Explorer in VS12 as shown in the above screen shot (Test – Windows – Test Explorer), it will open the Test Explorer window and you will see ‘Run All’ or ‘Run’ option to execute your tests as compared to ‘Run Checked Tests’ or ‘Debug Checked Tests’ in VS10. Below screen shots will show this difference


In VS12 ‘checkbox’ option is removed which means that if you want to select any particular tests to run, you have to click ‘CTRL + Select’.
Now that we understand the differences on how to run the tests in VS12, let us look at the features that have changed in VS12 while the tests are in progress. Unlike, VS10 where the progress of the test is shown with a distinct ‘In Progress’ icon, this icon is replaced in VS12 with a simple status bar. We have highlighted the differences in the below screenshots






Thursday, September 26, 2013

A Case Study on the Building a 21st Century Test Automation Framework!

A Case Study on the Building a 21st Century Test Automation Framework!

Abstract
Re-usable test automation frameworks coupled with open-source tools and technologies is a key solution to shrink test cycle times and related costs. This paper covers an approach for creating a script based web application automation framework around Selenium2 which is robust, flexible, and extensible. Our experimental results on this framework show increased productivity and ROI, reduced learning curve and dependency on skilled resources, eliminated usage of commercial tools and manual configuration. The framework provides ease of script generation & management, page object model, method level logging with screenshots, concurrent and remote execution using continuous integration environment, test data generation and customized reports. Preliminary statistics indicate that the automation framework reduces ~ 40% of total test execution effort per release cycle under a given scenario.

Biography
Frameworks are built at each level of the testing phase/cycle viz. unit, integration, and system testing. Each framework is built on a set of rules which are derived based on need analysis. This paper describes the methodology used in developing a robust web based automation framework.
This paper describes the “Plug and Play” data driven framework architecture in .Net on top of Behavior Driven layer as a best practice this architecture. As per the current design and implementation this framework is scalable because the controller and the execution scripts are designed as individual modules / utilities which give extensibility and ease of maintenance provides the best scalable solution to run tests in parallel.

Architecture and Design Approach


Basic components of an automation framework consist of selenium singleton class, in memory database, object repository, test data, execution engine, reporting and error logging.
Selenium singleton class is the core of the framework which is used to govern the execution work flow and modules listed below-

Application Configuration File: This file will load the required test script execution environment to govern the execution work flow.
BDD Layer: The feature files that are used by SpecFlow to store the acceptance criteria of the features (use cases, user stories) of your application are described in a format that is called Gherkin. Each feature file is associated with its definition file to drive the test scripts written in the page class.
In Memory Database: No physical database needed to avoid the extra burden for installing; all the created data through scripts primarily relies on main memory for computer data storage to efficiently used the created data for dependent scripts.           
Test Data Module: Test data stored in XML format and this will provide input for the automation test scripts. Each test data is picked based on the Test Environment defined in Application Configuration File.
Object Repository Module: Web elements / objects stored in resource file and this will provide the same as input to the automation test scripts.
Page Object Pattern: By using the page object model we can make non-brittle test code and reduce or eliminate duplicate test code. Beside of that it improves the readability and allows us to create interactive documentation. Last but not least, we can create tests with less keystroke. An implementation of the page object model can be achieved by separating the abstraction of the test object and the test scripts.
Libraries: This is used to load generic and application specific methods, reusable components grouped as a method calls in the page object model.
Logging Module: This module manages method logging along with screenshot handling across the work flow, Log4net can be used for debugging and error handling.
Reporting Module: Custom SpecFlow reports to produce some reports giving information about the features and steps after a test run has finished. The test execution report contains details about the SpecFlow test features, scenarios and steps. The step definition report shows the steps and their statuses, including unused steps and steps with no implementation.
Error Handling: Selenium2 and MS Test have mechanisms to capture the error logs. In case of a failure or crash, the error log stored in the root project folder along with screen shot.
Continuous Integration Environment: Through CI we execute automated tests after each commit in concurrent manner with different environment setup and send feedback with results to developers.

Test Environment Initialization
This module will set all the required parameters for a test run. The central framework engine will read the application configuration file and will set the environment required for the application under test (AUT)
Framework user has the flexibility to update the configuration file to select and execute tests by customizing the following parameters:
  • Base URL of the AUT
  • Type of Browser
  • Remote or Local machine
  • Wait For Element Time Limit
  • Test Environment Setup       

Reporting Engine
The default report of MSTest is comprehensive, but it's not easy to understand at-a-glance; as a result Continuous Integration is integrated with a batch process which converts the MSTest report in a SpecFlow HTML report. After the test run is completed an email alert will be sent to the stake holders and report published on the common web portal.

The Benefits
As a result of the current implementation Excelsoft is now able to ensure more comprehensive test coverage in a shorter period of time, including instant results of post-build basic tests. Allowing team members to easily develop and maintain reusable testing scenarios through the developed BDD layer on top of Selenium and also execute testing tasks with a unified and standardized approach has important positive implications on the overall efficiency and cost effectiveness of the operations. Also, it has made it possible to use convenient scheduled times, such as nightly run tests. As an example of how automation was effective in this case it was possible to reduce the average time for validating an important product feature from 8 hours to 3-4 hours.
  1. Total test cycle reduction from 2 weeks to 8h after automation.
  2. Shorter release cycles with increase from 1 build every 2 weeks to 2 builds per week.
  3. 60% overall cost reduction through the near shore model.
  4. Increased reusability of testing cases.
  5. Low maintenance costs of testing cases and processes.
  6. Possibility of implementing test metrics to guide decision-making.
  7. Introduction of benchmarks to increase software quality.
  8. Cost Reduction of software tools.

Remarks:

Thus we successfully developed a framework that helped us to exceed preset framework/automation objectives and could execute test scripts concurrently, reducing the execution time from 2 Man days to 60 minutes, bringing in 70% productivity improvement using framework on a continuous basis. We believe this approach can be carried across companies to significantly reduce the automation execution cost without compromising on quality. 

Wednesday, August 14, 2013

Software Test Automation Framework Features !

Software Test Automation Framework enable us focus on building robust,  automation solution to solve common problems, such as more frequent product cycles, less preparation time, reduced testing time, more platform choices, more programming language choices and Return on investment (ROI).  It helps in promoting automation and reuse, has broad platform and language support, and provides a common infrastructure across teams.

Automation Framework defines a set of guidelines for all phases of test automation; Test Requirement Analysis, Script Design, Execution, Reporting and maintenance. A framework provides a wrapper around different complex internal architecture layers, which makes it reusable, maintainable and high return on investment. It also enforces a set of standards for implementation.

This framework named as ExcelFramework based on .NET and consists of the following sub-components such as
  • Function Library
  • Common Functions
  • Database Layer (implementation)
  • The Object Repository
  • WebDriver Script
  • Logger
  • BDD Layer
  • Screen Shots
  • Page Object Model
  • Reports
  • DSL

Tuesday, August 13, 2013

Automation Engine Comparison !

Introduction
Automation market is full of tools that support automation for web applications along with the support for mobile automation as well. So, the aim of this post is to collect useful features for each listed above engine and present it in some comparable form. Thus we have:
  • CasperJS
  • AngularJS E2E
  • Jasmine
  • Sahi
  • Ranorex
  • Geb
  • Selenium WebDriver
  • Watin
This list isn't complete as there can be many other similar engines which are simply less popular. All of them have some common set of supported features but there're some restrictions and abilities specific to the actual engine. Key features to be mentioned are:

• Documentation availability
• Auto-complete
• Reporting
• CI Server Support
• Browser Support
• Language Complexity
• Setup Complexity
• Screenshots
• Technology Support (like SPA, Ajax)
                                                    
Each feature has some sub-features which reflect some specific part of functionality. And those little things make each engine different and unique in some cases. So, let's take a closer look at each feature. In order to make some comparison characteristics let's introduce the scale of support quality. Each feature can be values with the grade from 0 to 3 by the following criteria:

Grade
Criteria
0
No support at all
1
Very Less Support
2
Partially Support
3
Full-featured support

Documentation Availability
One of the key factors demonstrating the maturity of the engine is the documentation availability and its completeness. Indeed, when we start using new engine the first thing we usually do is documentation reading where we can take some examples, list of features etc. Actually, the documentation is the part of software product (this is one of the differences between application and software product), so if there's a lack of documentation it indicates that product isn't complete. Additional source of documentation is various online materials we can find in Internet from specialized resources, blogs etc. So, when I estimate the grade of documentation availability I check the following criteria:
  1. Documentation is available in general (it makes grade 1 at once)
  2. Every feature is described and has examples (if it fits it makes grade 2)
  3. There're additional well-grown resources (forums, blogs, user groups) where we can find additional information about the engine
These are external resources which can be found pretty quick (for me it took less than a minute to find them). So, given all the above information we can evaluate the grades for each engine. The table will look like:

Engine
Documentation availability
CasperJS
3
AngularJS E2E
3
Jasmine
3
Sahi
3
Ranorex
3
Geb
3
Selenium
3
Watin
3

Auto-complete
I'd say it's one of the most useful features while writing tests using natural language instructions. You just have to type some key parts of entire phrase and select the most appropriate option. If there's such ability it's definitely great. Unfortunately this feature is quite rare and mostly represented as some IDE plugin for writing tests. Since most of these tools are bounded to the language implementation like in CasperJS you can write your unit tests using Coffee Script & Java Scripts only, on the other side when it comes to language support Selenium & Watin actually wins the race which can be easily integrated to the IDE’s like Eclipse & Visual studio that supports Auto-Complete feature. Selenium’s most useful features have been its support for multiple programming languages, and is support for testing a wide variety of browserswhereas Watin is bounded to only two browsers that are IE & Firefox. Since most of the mentioned engines are bounded to Java scripts only which is a dynamic language and it least supports auto- complete/ intelligence.  So, support table for this feature looks like:

Engine
Auto-complete support
CasperJS
1
AngularJS E2E
1
Jasmine
1
Sahi
2
Ranorex
2
Geb
2
Selenium
3
Watin
3

As it's seen from the table the auto-complete is the weakest feature at the moment for most of the engines.

Reporting
Reporting is one of the most important parts of the testing and automation as well. It's not enough to say that test is passed or failed. Very often we should identify where it fails. It's vital for functional tests where we perform sequence of steps with some interim checks and such test can fail everywhere. And we should be able to say what the actual reason of fail was. We have the built in reports in Sahi, Ranorex, Geb & E2E on the other hand Selenium & Watin can easily integrate with any other reporting extensions like MSTest, Nunit, Junit & TestNG. However when it comes to CasperJS & Jasmine their execution usually happens on the headless browser in Jasmine using HTML Reporter, we can provide reporting for each spec and each suite & CasperJS provides Xunit XML reportSo, support table for this feature looks like:


Engine
    Reporting
CasperJS
2
AngularE2E
3
Jasmine
2
Sahi
3
Renorex
3
Geb
3
Selenium
3
Watin
3

CI Server support
Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. These days almost every tool has the CI server support for tools like CasperJS & AngularE2E we can easily export the results of a test suite to an Xunit XML file, which is compatible with continuous integration tools such as Jenkins. On the other side if talk about other tools like Jasmine, Sahi, Ranorex , Geb, Selenium & watin all these tools can be integrated with the CI server support. We are currently using Selenium WebDriver for our ongoing projects which are already integrated to the CI server (Jenkins) & we get the daily execution report for our automation workflows.


Engine
CI server support
CasperJS
3
AngularJS E2E
3
Jasmine
3
Sahi
3
Ranorex
3
Geb
3
Selenium
3
Watin
3

Browser Support
Browser support now a day is like a challenge for all organizations to perform Cross browser compatibility testing. If we think of this ability we would point to the automation tool that can perform cross browser compatibility testing in an efficient way & surely we would want such tools that can be run over multiple browsers with least effort. CasperJS & Jasmine engines will be run over the headless browser whereas AngularE2E scripts can be run in parallel on multiple browsers like Firefox, Chrome & IE, Geb builds on the WebDriver browser automation library, which means that Geb can work with any browser that WebDriver can, the existing drivers are the ChromeDriver,InternetExplorerDriverFirefoxDriverOperaDriver and HtmlUnitDriver. For more information about each of these, including their relative strengths and weaknesses, please follow the links to the relevant pages. There is also support for mobile testing via the AndroidDriver, OperaMobileDriver and IPhoneDriver. Sahi tool is a record & play web automation tool which also has multiple browser support open source tool. WatiN is developed in C# and aims to bring you an easy way to automate your tests with Internet Explorer and FireFox using .Net.

Engine
Browser Support
CasperJS
2
AngularJS E2E
2
Jasmine
2
Sahi
3
Ranorex
3
Geb
3
Selenium
3
Watin
2

Language Complexity
While choosing an automation tool we always want that our tests can be written in a language which we are familiar with so that least effort shall be required in writing our tests. If the language complexity is high then the effort will be very high for writing our tests. Now, if we talk about CasperJS we can write the tests in Coffee scripts & Java scripts those tests eventually would run over headless browser kit viz. PhantomJS. In AngularE2E & Jasmine these tools are widely used for Java scripts unit testing so it is quite obvious that we can achieve unit testing with the help of Java Scripts only. On the other hand Sahi is a record & play tool which generates the code as soon as we stop it’s recording. If we talk about Geb, the tests will be written with the help of Groovy scripts.  Selenium has its support for multiple programming languages but Watin is bounded to only .net Languages since it is developed in C#. So, the support table for this feature looks like:

Engine
Language Complexity
CasperJS
2
AngularJS E2E
2
Jasmine
2
Sahi
3
Ranorex
3
Geb
2
Selenium
3
Watin
3


Setup Complexity
While starting off with a new tool we always want that less set up configuration should be required in order to run the unit test. If the setup is huge & it is going through lot of complex configurations then it will surely affect the “Time required” factor. If we look at the engines there are many in the list that require external server to be set up on the system whereas there are others which is node independent & platform independent. Selenium WebDriver we can set up with any popular IDE like Eclipse & Visual Studio. Jasmine we can easily set up after downloading its library. CasperJS runs over headless browser webkit viz. PhantomJS so we can easily set up this tool after appending the path environment variable. Sahi tool we can easily set up after integrating it to the browser. Ranorex & Geb also do not require complex set up. Watin tool we can easily integrate it with Visual studio after adding its reference to the project. So, support table for this feature looks like:

Engine
Setup Complexity
CasperJS
3
AngularJS E2E
2
Jasmine
3
Sahi
3
Ranorex
3
Geb
3
Selenium
3
Watin
3












Screenshots
Screenshot is indeed one of the most important part of an automation tool. It gives more clarity to our automation scripts since the automation efforts does not require much human efforts so we will surely go through lot of challenges to know where exactly our tests was failed, in reports we would get the failed or passed log but we would not get a clear idea that why the test was failed what exactly was happening on the UI, was the test failed due to automation code or was it failed due to connection or any application issue so all these questions will be answered by a clean screenshot that will explain the exact behavior running at that time. AngularE2E and Jasmine does not support screen shot feature whereas CasperJS & Sahi has the ability of taking the screenshots. In Ranorex the Report class also supports a method to create screenshots to log images from applications or unexpected dialogs. Every Ranorex test should handle exceptions and generate at least one screenshot from the system's desktop. Geb uses the ScreenshotAndPageSourceReporter implementation that dumps the page source as html and attempts to take a screenshot as a .png file. Selenium & watin engines are well equipped with this ability of taking the screenshots. Watin uses CaptureWebPageToFile function to get the screenshot. So, support table for this feature looks like:

Engine
Setup Complexity
CasperJS
3
AngularJS E2E
0
Jasmine
0
Sahi
3
Ranorex
3
Geb
3
Selenium
3
Watin
3












Technology Support
When it comes to picking an automation engine we would always want that the engine would fully support almost every technical platforms, it’s not a good practice that down the time your application would introduce some new technical stuff which your existing automation tool cannot automate, it would surely add lot of pain to the developers/testers because no one wants to work on different automation tools for a single project. Hence while picking a tool we should first confirm whether the tool can support our existing technical platform or not. We have many new technologies that have been introduced in the market like SPA (using Angular), we cannot confidently take this call that our automation tool will support this platform & can easily automate the things. Support table for this feature looks like:
Engine
SPA & Ajax Compatibility
CasperJS
3
AngularJS E2E
3
Jasmine
3
Sahi
3
Ranorex
3
Geb
3
Selenium
3
Watin
3


Conclusion

If you’re a CasperJS user & you think of automated web testing as a car. CasperJS is the steering wheel & dashboard which interact with the engine. Allowing CasperJS users to use AngularE2E & Jasmine is like providing an additional engine choice, but keeping the steering wheel and dash board the same because the main objective of these three tools are almost same but if we look at the above records among these three Jasmine wins the race. However when it comes to the UI web automation part we believe Selenium & Watin are ahead of the above mentioned tools because of their multiple browser support & their strong reporting makes them the top priority, only flaw in Watin is it can run only on two browser viz. IE & FF & supports only .net languages. Whereas Ranorex is a paid tool & for Geb one has to learn groovy scripting. If we talk about Sahi its interface is very confusing & it is least developed. Hence as per the final conclusion for UI automation the best available tool in the market is Selenium & if an organization wants some of the automated unit tests should be developed by the developer itself then Watin can come in the picture because of its rich API. Hence the overall ratings are:

Engine
Documentation availability
Auto-Complete
Reporting
CI server support
Browser Support
Language Complexity
Setup complexity
Screen shots
Technology
Support
Overall
CasperJS
3
0
2
3
2
2
3
3
3
21
AngularE2E
3
0
3
3
2
2
2
0
3
18
Jasmine
3
0
2
3
2
2
3
0
3
18
Sahi
3
2
3
3
3
3
3
3
3
26
Ranorex
3
2
3
3
3
3
3
3
3
26
Geb
3
2
3
3
3
2
3
3
3
25
Selenium WebDriver
3
3
3
3
3
3
3
3
3
27
Watin
3
3
3
3
3
3
3
3
3
26