A Case Study on the Building a 21st Century Test Automation Framework!

A Case Study on the Building a 21st Century Test Automation Framework!

Abstract

Re-usable test automation frameworks coupled with open-source tools and technologies is a key solution to shrink test cycle times and related costs. This paper covers an approach for creating a script based web application automation framework around Selenium2 which is robust, flexible, and extensible. Our experimental results on this framework show increased productivity and ROI, reduced learning curve and dependency on skilled resources, eliminated usage of commercial tools and manual configuration. The framework provides ease of script generation & management, page object model, method level logging with screenshots, concurrent and remote execution using continuous integration environment, test data generation and customized reports. Preliminary statistics indicate that the automation framework reduces ~ 40% of total test execution effort per release cycle under a given scenario.

Biography

Frameworks are built at each level of the testing phase/cycle viz. unit, integration, and system testing. Each framework is built on a set of rules which are derived based on need analysis. This paper describes the methodology used in developing a robust web based automation framework.

This paper describes the “Plug and Play” data driven framework architecture in .Net on top of Behavior Driven layer as a best practice this architecture. As per the current design and implementation this framework is scalable because the controller and the execution scripts are designed as individual modules / utilities which give extensibility and ease of maintenance provides the best scalable solution to run tests in parallel.

Architecture and Design Approach

Basic components of an automation framework consist of selenium singleton class, in memory database, object repository, test data, execution engine, reporting and error logging.

Selenium singleton class is the core of the framework which is used to govern the execution work flow and modules listed below-

Application Configuration File: This file will load the required test script execution environment to govern the execution work flow.

BDD Layer: The feature files that are used by SpecFlow to store the acceptance criteria of the features (use cases, user stories) of your application are described in a format that is called Gherkin. Each feature file is associated with its definition file to drive the test scripts written in the page class.

In Memory Database: No physical database needed to avoid the extra burden for installing; all the created data through scripts primarily relies on main memory for computer data storage to efficiently used the created data for dependent scripts.           

Test Data Module: Test data stored in XML format and this will provide input for the automation test scripts. Each test data is picked based on the Test Environment defined in Application Configuration File.

Object Repository Module: Web elements / objects stored in resource file and this will provide the same as input to the automation test scripts.

Page Object Pattern: By using the page object model we can make non-brittle test code and reduce or eliminate duplicate test code. Beside of that it improves the readability and allows us to create interactive documentation. Last but not least, we can create tests with less keystroke. An implementation of the page object model can be achieved by separating the abstraction of the test object and the test scripts.

Libraries: This is used to load generic and application specific methods, reusable components grouped as a method calls in the page object model.

Logging Module: This module manages method logging along with screenshot handling across the work flow, Log4net can be used for debugging and error handling.

Reporting Module: Custom SpecFlow reports to produce some reports giving information about the features and steps after a test run has finished. The test execution report contains details about the SpecFlow test features, scenarios and steps. The step definition report shows the steps and their statuses, including unused steps and steps with no implementation.

Error Handling: Selenium2 and MS Test have mechanisms to capture the error logs. In case of a failure or crash, the error log stored in the root project folder along with screen shot.

Continuous Integration Environment: Through CI we execute automated tests after each commit in concurrent manner with different environment setup and send feedback with results to developers.

Test Environment Initialization

This module will set all the required parameters for a test run. The central framework engine will read the application configuration file and will set the environment required for the application under test (AUT)

Framework user has the flexibility to update the configuration file to select and execute tests by customizing the following parameters:

• Base URL of the AUT
• Type of Browser
• Remote or Local machine
• Wait For Element Time Limit
• Test Environment Setup       

Reporting Engine

The default report of MSTest is comprehensive, but it's not easy to understand at-a-glance; as a result Continuous Integration is integrated with a batch process which converts the MSTest report in a SpecFlow HTML report. After the test run is completed an email alert will be sent to the stake holders and report published on the common web portal.

The Benefits

As a result of the current implementation Excelsoft is now able to ensure more comprehensive test coverage in a shorter period of time, including instant results of post-build basic tests. Allowing team members to easily develop and maintain reusable testing scenarios through the developed BDD layer on top of Selenium and also execute testing tasks with a unified and standardized approach has important positive implications on the overall efficiency and cost effectiveness of the operations. Also, it has made it possible to use convenient scheduled times, such as nightly run tests. As an example of how automation was effective in this case it was possible to reduce the average time for validating an important product feature from 8 hours to 3-4 hours.

1. Total test cycle reduction from 2 weeks to 8h after automation.
2. Shorter release cycles with increase from 1 build every 2 weeks to 2 builds per week.
3. 60% overall cost reduction through the near shore model.
4. Increased reusability of testing cases.
5. Low maintenance costs of testing cases and processes.
6. Possibility of implementing test metrics to guide decision-making.
7. Introduction of benchmarks to increase software quality.
8. Cost Reduction of software tools.

Remarks:

Thus we successfully developed a framework that helped us to exceed preset framework/automation objectives and could execute test scripts concurrently, reducing the execution time from 2 Man days to 60 minutes, bringing in 70% productivity improvement using framework on a continuous basis. We believe this approach can be carried across companies to significantly reduce the automation execution cost without compromising on quality.