Calculate Software Automation ROI

Software automation ROI" refers to the Return on Investment (ROI) associated with implementing software automation in a business or organizational context. ROI is a measure used to evaluate the financial benefits of an investment relative to its cost. In the case of software automation, ROI assesses the value gained from automating specific processes or tasks compared to the expenses incurred in implementing and maintaining the automation.

Here are the following calculation parameters: (used google sheets)
 

Considering 2 years(12 months) of ROI projection:

Data Calculation parameters: 

• Number of test-cases can be automated: (B5 -(B5*B6)/100) 
• Automation of the test-cases needs: =(($B$7*$B$11)+B9)/B12
• Time-to-market acceleration: =(B10*B7*B8)/8

Statistics calculation parameters: 

Saved manual testing time (accumulated), hrs = 

• =$B$10*F21*$B$8*0
• =H21+($B$10*F22*$B$8) ...

Automation costs (accumulated), hrs =

• =D21
• =D22+I21 ...

Autotests run and support cost, hrs= 

• 0.00
• =$B$13 ...

Test cases automated = 

• =IF((E21/$B$11)*B12 >= $B$7, $B$7, (E21/$B$11)*B12)
• =IF(((E22/$B$11)*$B$12) + $F21 >= $B$7, $B$7, (($E22/$B$11)*$B$12) + F21) ...

Autotests development cost, hrs using standard working hrs/month =

• =IF($B$14 <= $B$18 * $B$12, $B$14, $B$18*$B$12)
• =IF(SUM($E$21:E21)+$B$18 >= $B$14, MAX(0, $B$14 - SUM($E$21:E21)), $B$18*$B$12) ...

Automation costs, hrs =

• =E21+G21
• =E22+G22 ...

Saved manual testing time, hrs = 

• =$B$10*F21*$B$8*0
• =$B$10*F22*$B$8 ...

ROI, hrs = 

• =H21-I21
• =H22-I22 ...

Total saved in $ (- ve means in loss) = 

• =(B45*B17)

Short Continuous Deployments To Production !

One of my product teams has been successful in deploying to production once every 4 weeks. In the past, our release plan would comprise of an Iterative and incremental development sprint zero, one or three working sprints, one or two alpha or beta sprints, and followed by a hardening sprint. 

We were essentially releasing features to end-users once in every six 8 or more. Our product features are critical to the business. 

Our product team migrated to a shorter release cycle thereby developing and deploying for each sprint and changed to following into 1 month cycle,

Benefits of Small Releases

As the iterative development process is mastered, the door opens for the team to deliver smaller and more frequent releases to the end-users, and that is one of the prime drivers of agile. The ability to release software more frequently has two primary business benefits: increased responsiveness to the customer amidst changing marketing conditions and reduced risk to the enterprise.

1. Frequent deployments to end-user provide a competitive advantage.
2. Product requirements could change significantly if deployed to production once every few months. By launching frequently, you could gather data from end-users earlier and apply adjustments accordingly.
3. Keep your team energized and motivated.
4. Helps in establishing best practices.
5. Exposes risks and challenges with design, architecture, and solutions earlier in the process.

Upgrade data migration validation !

Purpose:  To make sure  data consistency in upgraded environment.

Thoughts : Dump data base before and after upgrade and check data diff for these 2 data bases.

Available Tools:

1. dbForge Data Compare for SQL Server
2. DBDiff 
3. Pgdatadiff tool 

Environment Setup:

1. dbForge Data Compare for SQL Server

Steps 1: Dump sql file before / after upgrade

docker exec -t vaidio pg_dump --format=c  -U ainvr> dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql

docker exec -t your-db-container pg_dump --format=c -U ainvr > dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql

Steps 2: Create Empty Database

createdb -U postgres -W ainvr;
createdb -U postgres -W ainvr2;

Steps3 : Restore data to created empty database

psql -h localhost -p 5433 -U postgres -f  D:\dump_18-10-2021_13_23_57_before.sql ainvr
psql -h localhost -p 5433 -U postgres -f  D:\dump_18-10-2021_14_33_09_after.sql ainvr2

2. DBDiff Tool                               

DBDiff is an automated database schema and data diff tool.

○     It compares two databases, local or remote, and produces a migration file of the differences automatically.

○     Command-Line API

Source Code: https://dbdiff.github.io/DBDiff/

Comparison result

Executed Command:

dbdiff postgres://postgres:@localhost:5433/ainvr postgres://postgres:@localhost:5433/ainvr2

Result:

-- DROP TABLE "public"."scene_object_2021_07_26";

-- DROP TABLE "public"."scene_object_2021_07_27";

-- DROP TABLE "public"."scene_object";

-- DROP TABLE "public"."scene_object_2021_07_28";

3. Pgdatadiff Tool

1. Firstly, compares the row count in both tables.
2. If the row count is the same, it instructs postgres to create MD5 sums from data.

The MD5 sums are based on the data being cast to varchar. 

If you have data types that don't cast to varchar properly then the behavior probably not reliable. 

Source Code: https://github.com/dmarkey/pgdatadiff

Comparison result

Executed command:

pgdatadiff --firstdb=postgres://postgres:@localhost:5433/ainvr --seconddb=postgres://postgres:@localhost:5433/ainvr2

Future

• Integrate with Jenkins through ssh set up environment
• Execute db diff scripts from Jenkins

Recommdation

•  Pgdatadiff Tool

Firstly it compares the row count in both tables.

If the row count is the same, it instructs postgres to create MD5 sums of "chunks" of the table in both DBs and compares them. This way no data is actually read directly by pgdatadiff, it also means that pgdatadiff is relatively fast but is puts a moderate amount of pressure on the DB as it calculates the MD5 sums of large amounts of data. 

If There Isn’t Enough Time To Test !

What to do when there is not enough time for a critical bug fix test just a few days before release?

Partway through your test cycle, everything going smooth and all tasks accomplished at a high success rate but suddenly just before few days to release,

Your manage or alpha tester and internal ad hoc tester found a critical issue and without that fix, the release could not make it.

Set up a meeting, Impact Analysis,

Actors: Project Manger/lead, QA manager/lead, Development manager/lead

It involves analyzing the impact of changes made in features/modules in the application. 

• Analyzing modules with the help of impact analysis documents. It will find the risks associated with any kind of changes in a module/product due to a fixed defect.
• It also helps to implement a prototype for the developers and testers to experience the effects in the system.

Conclusion:

1. Find out the critical fix area.

- Point out the responsibility of the fixed function(s) and associations of same fixed function(s) with other functions within the same module i.e. green area. 

2. Find out the impacted areas/modules due to the critical fix.

- Point out the how many function(s) are affected due to green area fixed function(s) in orange and purple area and what are the effected function(s) responsibilities.

If area/module(s) are equally impacted;

Maybe both impacted modules are sharing the same fixed function.

Test Plan Approach:

Green Module: Smoke Test or End-To-End Test.

Orange or Purple Module: Sanity Test

Here we can customized out Sanity Test:

1. Sanity Test = Regression Test Cases for only effecting function.

2. Sanity Test = Whole system Regression Test - Whole system Negative Regression Test Cases. 

3. Sanity Test = Regression Test Cases for only effecting function - Negative Regression Test Cases for only effecting function.

If area/module(s) are partially impacted;

Maybe partially impacted modules are sharing the return value or function parameter(s) from the highly impacted module due to fixed-function in green module.

Test Plan Approach:

Green Module: Smoke Test or End-To-End Test.

Orange Module : Sanity Test

Purple Module: Smoke Test or End-To-End Test or Ad-Hoc Test.

Here we can customized out Sanity Test as before:

1. Sanity Test = Regression Test Cases for only effecting function.

2. Sanity Test = Whole system Regression Test - Whole system Negative Regression Test Cases. 

3. Sanity Test = Regression Test Cases for only effecting function - Negative Regression Test Cases for only effecting function.

Conclusion:

Impact Analysis is a mandatory phase and crucial point where we can derive our test plan in a limited amount of time to minimize our release risk.

Test plan selection could be changed or customized with the same approach.

Task prioritization, how?

Many real-life decision-making situations have several conflicting criteria that need to be considered at the same time. 

It may be hard to choose, but if my team is great at prioritizing tasks, I can deal with a lot of missed tasks, some distractions, and too many meetings.

The better managers in this world know how to get the best from their teams. They can analyze their team’s strengths and weaknesses, and delegate tasks to team members who have the capabilities to carry them out. 

A Prioritization Matrix is a decision-making method that allows selecting the most appropriate alternative after evaluating multiple conflicting criteria.

This is what we had implemented in our organization and within my Team.

In other way,

Urgent Vs. Important?

Urgency of a task is largely governed by deadlines. Belongs To DO.

Importance of a task is decided by the impact or significant change it can have on your life or an organization.

Urgency is driven by external factors like deadlines.

Importance is more of an introspective(thoughtful) exercise

What you consider important can differ from what others consider as important.

What we got call as: Eisenhower Matrix,

 In case everything is urgent and important? 

Hear comes: "Prioritization"

The action or process of deciding the relative importance or urgency of a thing or things.

Set Priority Methods,

Ultimately prioritize your tasks will depend on the nature of your job and your personal work style, but there are a few common task prioritization methods that might work for you.

1. Eisenhower Decision Matrix

• You sort tasks only based on urgency and importance but it’s not tells the order.

2. Eat the Frog 
3. ABCDE Method
4. Chunking

Eat the frog,

When you have two frogs to eat, eat the ugliest one first or

“If it’s your job to eat a frog, it’s best to do it first thing in the morning.”

Get the biggest, ugliest task out of the way as soon as you can, and the rest will come easily. 

ABCDE method

Separate tasks with similar priorities using the ABCDE method.

Differentiating between tasks that feel like they’re on the same level of importance

Here’s how it works,

Go through your list and give every task a letter from A to E (A being the highest priority).

For every task that has an A, give it a number which dictates the order you’ll do it in.

Repeat until all tasks have letters and numbers.

How to set priority?

Here’s where Brian Tracy’s ABCDE method works wonders. Instead of keeping all tasks on a single level of priority, this method offers two or more levels for each task.

Combined Eat The Frog and ABCDE Method,

The ABCDE method works well when combined with Eat the Frog.

Eat The Frog says “Get the biggest, ugliest task first”

ABCDE says “Assign each task a letter value according to its level of importance”

If you start your day off with your A and B tasks, then you can spend the rest of your day on tasks with lower importance, like C, D, and E.

How To set level of importance:

Ugliest task

Dependent task

Lets, we have following task’s and complete in 10 days sprint – 

According To Eisenhower Matrix all are “DO” tasks i.e. important and urgent,

• Essential Testing
• Defect Verification
• Regression Testing
• Execute API Auto 
• Creating Test Cases for new features
• Upgrade Testing
• Installation Testing
• Device App. Testing

Live workshop: Eat The Frog,

Live workshop: ABCDE Method,

Live workshop: Combined Method,

Live workshop: Final Combined Method,

Conclusion:

- Use Eat the Frog Method

• Based on weighted and complexity.

- Use ABCDE Method 

• Always put the dependent task’s on priority.
• Assign each task a letter value.

- Combine both to set your priority.

When you clearly prioritize your work, you can increase productivity, better manage your time, and feel confident that you’ll hit your deadlines—every time.