Uncommon (but essential) Metrics for Measuring Software Test Automation ROI
By analyzing common metrics such as the number of tests runs, test failure rates, and the time spent executing tests, stakeholders can identify areas for improvement and optimize the test automation process. While these are important to monitor, they are not designed to drive the right behavior of the overall software development organization. Quality is a team sport where all players must be on the field and attacking the same goal to make it work.

Uncommon metrics and why should they be measured
Collecting metrics for IT projects can be a challenging process for companies due to a variety of reasons. One of the main challenges is obtaining accurate data and measuring progress consistently, especially when dealing with complex and dynamic projects. Lack of stakeholder buy-in and insufficient resources for tracking and analyzing metrics can also hinder the collection process. Furthermore, companies may struggle to effectively communicate and use the metrics they collect to drive decision-making and improve project outcomes.
TRUST: The lack of trust in software testing automation.
This is due to concerns about the accuracy of the results generated by automated testing, which can lead to redundant testing being performed. As a result, the business team may be duplicating testing efforts to compensate for this lack of trust in automation. Overall, there is a need for increased confidence in automated testing to reduce redundancy and streamline the testing process.
SCOPE: Issues of not having a clear scope for test automation.
Several issues arise due to the absence of a clear scope, including the lack of hygiene in test automation assets, infrequent reviews of the assets, and no clear understanding of automation efficiency and associated costs. The whole team should understand the need for a more structured and systematic approach to test automation, with clear guidelines for maintaining and evaluating automation assets.
AUTONOMY: The elimination of the need for human interaction to run automated tests.
In most companies, test automation still requires manual intervention to run, which can slow down the testing process and introduce errors. Likewise, a poor or non-implementation of DevOps is lacking, which limits the effectiveness of automation. Investing in test automation also requires investments in automating as much of the testing process as possible including DevOps practices to streamline the process and improve efficiency.
MAINTENANCE: A large effort dedicated to test automation maintenance.
Often, the majority of the team’s time is spent on modifying and updating existing automated tests, which can be a time-consuming and resource-intensive process. There is no proper tracking of changes from a business workflow level, which can make it difficult to assess the impacts of changes on existing automated tests and lead to inefficiencies and errors in the testing process. For a more effective, streamlined and systematic approach to test automation maintenance, it is crucial to have clear guidelines for tracking and assessing changes and their impacts on existing automated tests.
READINESS: Challenges related to manual dependencies and pre-and-post-conditions in test automation.
The deployment process is complex and infrequent, and not fully automated, which can lead to inefficiencies and errors in the testing process. The test environment setup and updates are usually not automated, which can further slow down the testing process. Likewise, test data creation and updates are (totally or partially) manually performed, which leads to inconsistencies and inaccuracies in the testing process. It is essential to have a more streamlined and automated approach to test automation, with automated deployment, environment setup, and test data creation and updates, to improve the efficiency and accuracy of the automation testing process.
VALUE: Not considering all direct and indirect costs when calculating the ROI of software testing automation.
The costs of test automation are often not fully considered, such as the costs of tools, labor, environments, and expertise, as well as the value of early defect detection, including defect prevention and coverage increase. This can lead to inaccurate ROI calculations and underestimation of the true value of test automation. However, only the costs of manual labor and defect leakage are usually considered, which can further limit the accuracy of ROI calculations. It is critical to consider all relevant costs and benefits when calculating the ROI of test automation to obtain a more accurate assessment of its value.
Download our Automation Metrics guide for more information about the metrics to measure the six themes above, their definitions and scorecard. They include:
Trust Factor
Organizational Readiness
Automated Automation Percentage
Test Automation Maintenance Ratio
Scope Clarity Index
True Automation Cost to Value Trend
Toni is the test automation practice leader and head of emerging technology and testing at QA Consultants. Focus on continuous quality within the DevOps practices implementation, development of new approaches and technologies to support QE for Deterministic and Cognitive Systems.

Toni Jardini
Director, Quality Engineering and Emerging Technologies Practices