Adding the HTML Reporter in Protractor JS !

When we run the tests, we can see that tests are passing or failing in the command console. What if we have hundreds of tests and also want to keep a record of the test results  whenever we run the tests? We need test reporters for that.

Protractor gives the options to add onPrepare function property on its configuration file. This onPrepare function will be called before each tests. Jasmine test framework(which we are using inside protractor) provides the option to add test reporter. We will get the metadata about each test by using these two configurations.

There is a node module available for generating screenshots for each tests. protractor-screenshot-reporter. However this doesn't process the meta data and gives us the result in a readable format.

I couldn't find a node module doing this for protractor. So, I created a module for generating this HTML report, on top of protractor-screenshot reporter, protractor-html-screenshot-reporter. What it does is generate an HTML report with all the details of the test like status, browser used, message, link to screenshot etc. (If you are finding some other modules which are doing the same, please add in the comments. )

Let’s install protractor-html-screenshot-reporter module in our project.

$ npm install protractor-angular-screenshot-reporter --save-dev

We need to add this reporter in the Protractor configuration file.

1. Require the protractor-html-screenshot-reporter module and assign to a variable.

var reporter = new HtmlReporter({
   baseDirectory: '/tmp/screenshots'
});

2. Add this reporter on the onPrepare function.

var HtmlReporter = require('protractor-angular-screenshot-reporter');

exports.config = {
   // your config here 
   onPrepare: function() {
      // Add a screenshot reporter and store screenshots to `/tmp/screenshots`:
      jasmine.getEnv().addReporter(new HtmlReporter({
         baseDirectory: '/tmp/screenshots'
      }));
   }
}

3. Run the test using command line.

$ protractor protractor.conf.js

We can see that the tests are passing. Let’s see if the reports are generated.

Screenshots folder is created.

Inside the screenshots folder, one json and png with a guid filename is available. This is the metadata and screenshot respectively for the single test. There is also a combined.json which combines all the metadata into a single file and a reporter.html file which shows all the test reports.

Opening the reporter.html looks like below.

So, now you have an html reporter ready. But, there is one issue though. Whenever you run the tests, the reporter files are getting overridden. If I want to keep a track of tests, this is not going to help.

Protractor-html-screenshot-reporter provides the option of pathBuilder function property to give your own dynamic paths. We will add this in Protractor configuration file.

I am just using a basic function to give the folder name as the current date and time and browser.

Note that I am using the path module for creating the file path (path is a native node module for handling and transforming file paths. ). Make sure that, you have required the path and assigned to a variable.

E2E Protractor for .NET

Protractor is an end to end test framework for AngularJS applications built on top of WebDriverJS. In this tutorial I’ll guide you through the process of getting it up and running. Protractor can run Jasmine and Mocha tests so feel free to choose the one that suits your needs best.

The .NET port of Protractor, an end to end test framework for Angular applications.

Protractor for .NET is built on top of Selenium WebDriver C# binding.

Get it from NuGet!

PM> Install-Package Protractor

Supports Microsoft .NET Framework 3.5 and higher..

Write Tests!

         

///

/// User Login.

///

private void UserLogin()

{

//Create Webdriver Instance

webDriver = new ChromeDriver();

webDriver.Manage().Timeouts().SetScriptTimeout(TimeSpan.FromSeconds(10));

IWebDriver ngDriver = new NgWebDriver(webDriver);

//Navigate Browser

ngDriver.Navigate().GoToUrl("https://mail.gmail.com");

//Enter User Name                    ngDriver.FindElement(NgBy.Input("user.Email")).SendKeys("dummy.user@gmail.com");

//Enter User Password

ngDriver.FindElement(NgBy.Input("user.Password")).SendKeys("qwertyuio");

//Wait For Element

webDriver.FindElement(By.CssSelector("button[data-qa-id*='login-button']")).Click();

//Click Save Button

webDriver.FindElement(By.CssSelector("span[class*='ico-profile']")).Click();       

}

Changes in Visual Studio 2012 with respect to test features !

Welcome to the first post in this series which will give an overview of the differences in Visual Studio 2012 (VS12) retrospect to Visual Studio 2010 (VS10). In this particular post, we will cover the basic differences between VS2012 and VS2010 Test features, mostly with respect to navigation, running the tests, results and some features that have been deprecated in VS12. Let us now dive into this.

Test List Editor which is very commonly used to run tests in VS10 is deprecated in VS12 and is replaced by Test Explorer. Below screenshots show this difference with a red highlight.

Next question would obviously be, how do I run my tests from Test Explorer and how will this be different from VS2010. The experience of running the tests in VS12 is seemingly different from VS10 for a first time user. When you select Test Explorer in VS12 as shown in the above screen shot (Test – Windows – Test Explorer), it will open the Test Explorer window and you will see ‘Run All’ or ‘Run’ option to execute your tests as compared to ‘Run Checked Tests’ or ‘Debug Checked Tests’ in VS10. Below screen shots will show this difference

In VS12 ‘checkbox’ option is removed which means that if you want to select any particular tests to run, you have to click ‘CTRL + Select’.

Now that we understand the differences on how to run the tests in VS12, let us look at the features that have changed in VS12 while the tests are in progress. Unlike, VS10 where the progress of the test is shown with a distinct ‘In Progress’ icon, this icon is replaced in VS12 with a simple status bar. We have highlighted the differences in the below screenshots

A Case Study on the Building a 21st Century Test Automation Framework!

A Case Study on the Building a 21st Century Test Automation Framework!

Abstract

Re-usable test automation frameworks coupled with open-source tools and technologies is a key solution to shrink test cycle times and related costs. This paper covers an approach for creating a script based web application automation framework around Selenium2 which is robust, flexible, and extensible. Our experimental results on this framework show increased productivity and ROI, reduced learning curve and dependency on skilled resources, eliminated usage of commercial tools and manual configuration. The framework provides ease of script generation & management, page object model, method level logging with screenshots, concurrent and remote execution using continuous integration environment, test data generation and customized reports. Preliminary statistics indicate that the automation framework reduces ~ 40% of total test execution effort per release cycle under a given scenario.

Biography

Frameworks are built at each level of the testing phase/cycle viz. unit, integration, and system testing. Each framework is built on a set of rules which are derived based on need analysis. This paper describes the methodology used in developing a robust web based automation framework.

This paper describes the “Plug and Play” data driven framework architecture in .Net on top of Behavior Driven layer as a best practice this architecture. As per the current design and implementation this framework is scalable because the controller and the execution scripts are designed as individual modules / utilities which give extensibility and ease of maintenance provides the best scalable solution to run tests in parallel.

Architecture and Design Approach

Basic components of an automation framework consist of selenium singleton class, in memory database, object repository, test data, execution engine, reporting and error logging.

Selenium singleton class is the core of the framework which is used to govern the execution work flow and modules listed below-

Application Configuration File: This file will load the required test script execution environment to govern the execution work flow.

BDD Layer: The feature files that are used by SpecFlow to store the acceptance criteria of the features (use cases, user stories) of your application are described in a format that is called Gherkin. Each feature file is associated with its definition file to drive the test scripts written in the page class.

In Memory Database: No physical database needed to avoid the extra burden for installing; all the created data through scripts primarily relies on main memory for computer data storage to efficiently used the created data for dependent scripts.           

Test Data Module: Test data stored in XML format and this will provide input for the automation test scripts. Each test data is picked based on the Test Environment defined in Application Configuration File.

Object Repository Module: Web elements / objects stored in resource file and this will provide the same as input to the automation test scripts.

Page Object Pattern: By using the page object model we can make non-brittle test code and reduce or eliminate duplicate test code. Beside of that it improves the readability and allows us to create interactive documentation. Last but not least, we can create tests with less keystroke. An implementation of the page object model can be achieved by separating the abstraction of the test object and the test scripts.

Libraries: This is used to load generic and application specific methods, reusable components grouped as a method calls in the page object model.

Logging Module: This module manages method logging along with screenshot handling across the work flow, Log4net can be used for debugging and error handling.

Reporting Module: Custom SpecFlow reports to produce some reports giving information about the features and steps after a test run has finished. The test execution report contains details about the SpecFlow test features, scenarios and steps. The step definition report shows the steps and their statuses, including unused steps and steps with no implementation.

Error Handling: Selenium2 and MS Test have mechanisms to capture the error logs. In case of a failure or crash, the error log stored in the root project folder along with screen shot.

Continuous Integration Environment: Through CI we execute automated tests after each commit in concurrent manner with different environment setup and send feedback with results to developers.

Test Environment Initialization

This module will set all the required parameters for a test run. The central framework engine will read the application configuration file and will set the environment required for the application under test (AUT)

Framework user has the flexibility to update the configuration file to select and execute tests by customizing the following parameters:

• Base URL of the AUT
• Type of Browser
• Remote or Local machine
• Wait For Element Time Limit
• Test Environment Setup       

Reporting Engine

The default report of MSTest is comprehensive, but it's not easy to understand at-a-glance; as a result Continuous Integration is integrated with a batch process which converts the MSTest report in a SpecFlow HTML report. After the test run is completed an email alert will be sent to the stake holders and report published on the common web portal.

The Benefits

As a result of the current implementation Excelsoft is now able to ensure more comprehensive test coverage in a shorter period of time, including instant results of post-build basic tests. Allowing team members to easily develop and maintain reusable testing scenarios through the developed BDD layer on top of Selenium and also execute testing tasks with a unified and standardized approach has important positive implications on the overall efficiency and cost effectiveness of the operations. Also, it has made it possible to use convenient scheduled times, such as nightly run tests. As an example of how automation was effective in this case it was possible to reduce the average time for validating an important product feature from 8 hours to 3-4 hours.

1. Total test cycle reduction from 2 weeks to 8h after automation.
2. Shorter release cycles with increase from 1 build every 2 weeks to 2 builds per week.
3. 60% overall cost reduction through the near shore model.
4. Increased reusability of testing cases.
5. Low maintenance costs of testing cases and processes.
6. Possibility of implementing test metrics to guide decision-making.
7. Introduction of benchmarks to increase software quality.
8. Cost Reduction of software tools.

Remarks:

Thus we successfully developed a framework that helped us to exceed preset framework/automation objectives and could execute test scripts concurrently, reducing the execution time from 2 Man days to 60 minutes, bringing in 70% productivity improvement using framework on a continuous basis. We believe this approach can be carried across companies to significantly reduce the automation execution cost without compromising on quality. 

Software Test Automation Framework Features !

Software Test Automation Framework enable us focus on building robust,  automation solution to solve common problems, such as more frequent product cycles, less preparation time, reduced testing time, more platform choices, more programming language choices and Return on investment (ROI).  It helps in promoting automation and reuse, has broad platform and language support, and provides a common infrastructure across teams.

Automation Framework defines a set of guidelines for all phases of test automation; Test Requirement Analysis, Script Design, Execution, Reporting and maintenance. A framework provides a wrapper around different complex internal architecture layers, which makes it reusable, maintainable and high return on investment. It also enforces a set of standards for implementation.

This framework named as ExcelFramework based on .NET and consists of the following sub-components such as –

• Function Library
• Common Functions
• Database Layer (implementation)
• The Object Repository
• WebDriver Script
• Logger
• BDD Layer
• Screen Shots
• Page Object Model
• Reports
• DSL