Home White Paper White Paper: Maximizing the ROI of Software Testing

White Paper: Maximizing the ROI of Software Testing

Every firm depends on the deployment of new software applications and products to compete better. Testing software is an expensive proposition, making up 40% of some development projects. Maximizing the Return on Investment (ROI) from this vital activity is on every executive’s agenda, especially since the price of failure is reduced competitiveness, or worst case, their jobs. One powerful way to generate positive ROI and limit risk is to minimize and preempt costs and preempt project delays.

20 Years of Delivering Testing Solutions

Twenty years of delivering testing solutions for Fortune 500 organizations has taught us a thing or two about improving software development and testing efficiencies. One key observation is that companies and managers typically underestimate the “true” cost of testing, especially indirect costs and opportunity losses (i.e. the value surrendered by performing another activity). Higher testing costs are correlated with lower project ROI and operational performance. Secondly, maximizing ROI is also a function of time to value; the faster a project is completed, the higher the financial returns and market performance.

The goal of this white paper is to bring more visibility to the murky world of understanding and estimating total testing costs. Regardless of the test strategy and design, business and technical project owners should ensure they have financial and operational cost transparency and predictability. Complete visibility will allow managers to make better test strategy decisions, ensure projects are properly estimated and better anticipate risks so they can be preempted.

The following analysis will highlight some important cost considerations around common testing approaches and provide executive teams with some multi-industry best practices for managing and reducing these costs. Please note, this paper is not meant to be a definitive and complete analysis of all costs and inter-department allocations. Actual results will vary by company and project.

Challenges in Determining True Cost

Determining the true cost of testing a new application or product is not easy when firms do not have good cost visibility into their cost structures, and given technical uncertainties and changing business needs. In particular, companies that outsource their testing, either by using independent contractors or by engaging generalist IT services firms via offshore delivery, are particularly challenged. Merely giving an outsourcer a budget and having a contract is no guarantee of financial compliance. In addition, offshore providers come with a host of other cost variables such as currency swings, project management and energy costs that can play havoc with ROI and delivery.

The Impact of Test Strategy on Cost

Large organizations usually perform software testing in one of three ways: 1) ‘Do it yourself’ testing by maintaining an in-house team and tools; 2) Outsource parts or most of the testing activity to independent contractors, while retaining a modest testing infrastructure and; 3) Outsource most testing activity to a third party, usually a generalist IT services firm. This provider will typically undertake the testing and software development in India or in another emerging economy. Each approach has different cost dynamics and implications, depending on the project and the firm’s cost structure. Also, many of the same indirect costs like lower productivity or delays are found in each approach; they just arise under different conditions.

Do it yourself

A DIY strategy has some advantages around cost certainty. Companies can usually tally up their direct labour expenses as well as estimate overhead with reasonable accuracy. What is less apparent are the extra costs that come with performing testing in-house. These hidden, indirect or frequently underestimated labour charges and missed opportunities include:

  1. Turnover – Managers often don’t account for the cost of high turnover. Up to 20% of your project’s labour expense could be tied up in recruiting costs, employee administration, and hassle. These costs can be significant as it is not uncommon for many firms to experience up to a 50% annual turnover within their testing team. Exiting testers also take with them valuable knowledge which can compromise the project and long-term testing capabilities. Minimizing turnover, particularly in dynamic Agile environments, should always be a corporate priority.
  2. Reduced team productivity – Testing productivity is constrained when the team does not have the requisite skills or numbers to effectively perform the testing task. Lower productivity is directly connected to higher costs (ie.e. you need more testers), reduced testing performance (from not leveraging best practices) and more project delays. In particular, skills misalignments are common when companies’ look to adopt new technologies or their IT infrastructures have become exceedingly complex. Fixing this situation is difficult because of the dynamic nature of testing needs, poor requirements definition and relatively static nature of internal resources.
  3. Underutilized testers and tools – Resource utilization is a major driver of cost. Keeping a dedicated roster of testers and the tools makes financial sense when they are 100% utilized. However, in many companies, the testing group often remains idle as they wait for new code or projects to emerge. Low utilization drives up the actual cost of testing and leads to a misallocation of scarce capital. A lack of regular planning plus rapidly changing business needs will compromise efforts to better link resource supply to testing demands.
  4. Unnecessary delays and costs – Common organizational dynamics can surreptitiously drive up cost. For example, some testers to keep their jobs may stretch out the project deliverables thereby driving up cost and time to market. In other cases, weak controls or siloed structures can lead to rogue or wasteful hiring practices and tools purchases even though other parts of the organization have under-utilized resources.
  5. Opportunity cost – CEOs need always to ask whether they need substantial testing capabilities. In this time of challenging economic conditions and competing for business priorities, is retaining (and regularly enhancing) a testing group (as a fixed cost) a good way to maximize a company’s return on capital and assets and focus on core competencies? Enterprises that can’t maintain high-performance testing capabilities and fully utilize them should consider outsourcing testing to a ‘best in class’ provider.

Outsourcing via local contractors

This testing strategy is a hybrid of many approaches. Organizations retain a core group of testers and management but utilize independent contractors for large initiatives or to handle surge requirements. Although this is a more flexible and variable cost model, there are important expenses that are unanticipated or often go unmeasured.

  1. Higher wage and infrastructure rates – Independent contractors are generally more expensive on a per hour basis than internal employees. They also need real estate, desks and computers, which add to the expenses.
  2. Recruiting – Up to 25% of your annual labour expense could be tied up in recruiting costs, employee administration, and hassle. It is not uncommon for many firms to experience 40% annual turnover among contractors.
  3. Lost productivity – Contractors, especially those with specialized or scare skills, do not materialize out of thin air. Recruiting takes time and significant skill. This latency takes a toll in terms of lost productivity, project delays, and lower quality from not leveraging best practices. Boosting productivity can be a challenge due to the dynamic nature of testing needs, cultural barriers around collaboration etc. as well as the static nature of internal resourcing and tools.
  4. Lost knowledge – Contractors address project needs but do little to build organizational capability or add best practices – unless they are properly incentivized and the information is captured. In the long run, this lost knowledge adds cost and hampers performance.

Complete outsourcing to a global IT services firm

Generalist IT services companies like IBM, Wipro, TCS, Accenture and CGI will often assume end-to-end responsibility for a client’s testing and software development. In most cases, the major of the work is performed offshore in places like India, China or Russia. These offshored projects tend to be complex, risky and large. As a result, managers need a clear appreciation of the true, long-term cost. Expenses to watch out for include:

  1. High relationship costs – Offshored IT projects tend to generate their own complexity, in turn requiring considerable staffing on both sides of the arrangement to manage issues, ensure compliance etc. Furthermore, companies regularly underestimate the travel, communication and administrative costs of dealing with a provider thousands of miles away. In some cases, these expenses can make up 20% of the project’s total cost. The more complex or error-prone the initiative, the higher will be the relationship and project management costs.
  2. The cost of poor quality – The possibility of getting poor quality software increases with the size of the physical, the linguistic and cultural divide between the user, developer and tester. Not only does poor code quality lead to higher testing costs and rework but it also leads to project delays (i.e. time is money) and higher relationship and costs.
  3. Diminishing labour returns – India’s outsourcing model is based on providing inexpensive resources to compensate for low labour productivity tester, minimal innovation & automation and high levels of turnover. When projects don’t go smoothly, Indians will often throw more resources or replace existing testers. This turnover (which approaches 80% in some places) adds costs and complexity, not to mention potentially compromising quality.
  4. Conflict of interest – As part of their arrangements, Indian IT services firms (with the client acquiescing) will usually develop and test their own code. Having one provider write and test their programs creates opportunities and incentives to shortchange the testing plan if the development portion is running over budget or if the provider is in a hurry to get paid. The result could be poorer code quality and contract violations. One simple way to avoid conflicts (and secure best in class capabilities) is to use different vendors for the development and testing tasks.
  5. External variables – Many external factors, beyond the control of the client or offshore provider, can significantly drive up costs. Some things, like currency fluctuations, can be hedged. Other factors like energy costs and regulatory fees cannot be passed along if not mitigated. Executives can preempt these costs by choosing a local outsourcer like QA Consultants.
  6. Lost knowledge – These days, operational and product knowledge gleaned through testing is too important to be left at the discretion of the outsourcer. Clients must ensure they are effectively overseeing activities and changes to capture important operational knowledge and costing information. This oversight is extremely difficult when your outsourcer is halfway around the world in India and running an opaque testing process.

Conclusion

Cost expansion is a major cause of software projects running amok and failing to generate a solid ROI. This need not be the case. Managers can go a long of way to improving their project’s financial performance by increasing their visibility into all direct and indirect costs, targeting value enhancement versus cost minimization and better preempting and mitigating likely problems. The following eight steps comes from our best practice consulting work with some sophisticated Fortune 500 clients:

  1. Understand from an inventory perspective what resources, skills, tools and capabilities you already have so you staff properly, eliminate wasteful purchasing (especially testing tools) and forestall unexpected developments.
  2. Engage your finance and accounting group early in the test planning process to get a more fulsome view of costs and risks, especially when it comes to offshore providers.
  3. Identify if and where your IT provider is conflicted (i.e. they write and test their own code) and change the deal, or split the development and testing roles.
  4. Minimize and control the transaction costs (like recruiting, relationship management), where possible, and look to hedge external risks like currency volatility.
  5. Limit the number of testers and providers you work with to make sure your industry, product and organizational learning is retained and passed along.
  6. When it comes to people, recognize there is a link between wages and experience, skills and performance. Focus more on value than on cost.
  7. At specific project milestones including sign off, perform a project debrief to understand what worked and what didn’t.
  8. Explore Canadian-based outsourcers like the Test Factory™ that can deliver high quality, lower total delivered costs (e.g., through lower relationship and transaction costs) and reduced project and business risks.