A Case
Study on the Building a 21st Century Test Automation Framework!
Abstract
Re-usable test automation
frameworks coupled with open-source tools and technologies is a key solution to
shrink test cycle times and related costs. This paper covers an approach for
creating a script based web application automation framework around Selenium2
which is robust, flexible, and extensible. Our experimental results on this
framework show increased productivity and ROI, reduced learning curve and
dependency on skilled resources, eliminated usage of commercial tools and
manual configuration. The framework provides ease of script generation &
management, page object model, method level logging with screenshots, concurrent
and remote execution using continuous integration environment, test data
generation and customized reports. Preliminary statistics indicate that the
automation framework reduces ~ 40% of total test execution effort per release
cycle under a given scenario.
Biography
Frameworks are built at
each level of the testing phase/cycle viz. unit, integration, and system
testing. Each framework is built on a set of rules which are derived based on
need analysis. This paper describes the methodology used in developing a robust
web based automation framework.
This paper describes the
“Plug and Play” data driven framework architecture in .Net on top of Behavior
Driven layer as a best practice this architecture. As per the current design
and implementation this framework is scalable because the controller and the
execution scripts are designed as individual modules / utilities which give
extensibility and ease of maintenance provides the best scalable solution to
run tests in parallel.
Architecture
and Design Approach
Basic components of an
automation framework consist of selenium singleton class, in memory database,
object repository, test data, execution engine, reporting and error logging.
Selenium singleton class is
the core of the framework which is used to govern the execution work flow and
modules listed below-
Application
Configuration File: This file will load the required test
script execution environment to govern the execution work flow.
BDD
Layer: The feature files that are used by SpecFlow to store the acceptance criteria of the features (use
cases, user stories) of your application are described in a format that is
called Gherkin. Each feature file is associated with its definition file to
drive the test scripts written in the page class.
In
Memory Database: No physical database needed to avoid the
extra burden for installing; all the created data through scripts primarily
relies on main memory for computer data storage to efficiently used the created
data for dependent scripts.
Test
Data Module: Test data stored in XML format and this
will provide input for the automation test scripts. Each test data is picked
based on the Test Environment defined in Application Configuration
File.
Object
Repository Module: Web elements / objects stored in resource
file and this will provide the same as input to the automation test scripts.
Page
Object Pattern: By using the page object model we can make
non-brittle test code and reduce or eliminate duplicate test code. Beside of
that it improves the readability and allows us to create interactive
documentation. Last but not least, we can create tests with less keystroke. An
implementation of the page object model can be achieved by separating the
abstraction of the test object and the test scripts.
Libraries:
This is used to load generic and application specific methods, reusable
components grouped as a method calls in the page object model.
Logging
Module: This module manages method logging along with
screenshot handling across the work flow, Log4net can be used for debugging and
error handling.
Reporting
Module: Custom SpecFlow reports to produce some reports giving
information about the features and steps after a test run has finished. The
test execution report contains details about the SpecFlow test features,
scenarios and steps. The step definition report shows the steps and their statuses,
including unused steps and steps with no implementation.
Error
Handling: Selenium2 and MS Test have mechanisms to capture the
error logs. In case of a failure or crash, the error log stored in the root
project folder along with screen shot.
Continuous
Integration Environment: Through CI we execute automated tests
after each commit in concurrent manner with different environment setup and
send feedback with results to developers.
Test
Environment Initialization
This module will set all
the required parameters for a test run. The central framework engine will read
the application configuration file and will set the environment required for
the application under test (AUT)
Framework user has the
flexibility to update the configuration file to select and execute tests by
customizing the following parameters:
- Base
URL of the AUT
- Type
of Browser
- Remote
or Local machine
- Wait
For Element Time Limit
- Test
Environment Setup
Reporting
Engine
The default report of
MSTest is comprehensive, but it's not easy to understand at-a-glance; as a
result Continuous Integration is integrated with a batch process which converts
the MSTest report in a SpecFlow HTML report. After the test run is completed an
email alert will be sent to the stake holders and report published on the
common web portal.
The
Benefits
As a result of the current
implementation Excelsoft is now able to ensure more comprehensive test coverage
in a shorter period of time, including instant results of post-build basic
tests. Allowing team members to easily develop and maintain reusable testing
scenarios through the developed BDD layer on top of Selenium and also execute
testing tasks with a unified and standardized approach has important positive
implications on the overall efficiency and cost effectiveness of the
operations. Also, it has made it possible to use convenient scheduled times, such
as nightly run tests. As an example of how automation was effective in this
case it was possible to reduce the average time for validating an important
product feature from 8 hours to 3-4 hours.
- Total
test cycle reduction from 2 weeks to 8h after automation.
- Shorter
release cycles with increase from 1 build every 2 weeks to 2 builds per week.
- 60%
overall cost reduction through the near shore model.
- Increased
reusability of testing cases.
- Low
maintenance costs of testing cases and processes.
- Possibility
of implementing test metrics to guide decision-making.
- Introduction
of benchmarks to increase software quality.
- Cost
Reduction of software tools.
Remarks:
Thus we successfully
developed a framework that helped us to exceed preset framework/automation objectives
and could execute test scripts concurrently, reducing the execution time from 2
Man days to 60 minutes, bringing in 70% productivity improvement using framework
on a continuous basis. We believe this approach can be carried across companies
to significantly reduce the automation execution cost without compromising on quality.
