Skip to content

White Paper: Defining the Right Testing Strategy

When it comes to leveraging new applications and getting more from their existing IT assets, businesses face a cornucopia of opportunities – and risks. Companies want to get to market or value quicker with new products and applications. At the same time, cost containment is de rigueur. Risk mitigation – ranging from preserving operational continuity to ensuring data security and protecting the brand image – is now table stakes everywhere.

Serious Technical and Cost Implications

These demands have serious technical and cost implications for the IT group, such as the growing complexity of software applications and enterprise infrastructures; the steady rise in the cost of IT and; the need to quickly integrate new technologies like mobile and cloud computing in order to maintain competitiveness.

As was shown by the Japanese manufacturing sector in the 1970s, the surest way to improve financial, business and brand performance are to make a product and operational quality the number one corporate goal. For organizations in 2013, maximizing performance while minimizing cost has a lot to do with improving the quality of the software code in applications and new products. To do this, CIOs need a well-crafted testing strategy and plan.

This is easier said than done. Testing is not viewed as sexy in many companies. Its consideration and planning is often an afterthought, at best seen as a necessary evil. Even when testing is valued it often gets short shrift – especially when the project is already late – when it comes to the time allotted and resources since it comes late in the software development cycle. Ultimately, both these results impact technical and business performance.

Testing Matters

Testing matters, both from a financial and technical perspective. Up to 40% of the cost of software development is made up of testing activities and rework. Poor software code will significantly increase business risk and compromise the brand image. Yet, too many firms are flying blind without having a formal testing strategy. Many of these organizations have wasted millions of dollars and hours on ineffective and wasteful testing. In addition, even those firms with formal strategies tend not to update them to reflect new economic realities and market conditions.

Testing is too important and expensive to be handled in an ad hoc, non-strategic manner. The first step is to develop a new or update the existing test strategy. This should be done in the initial phase of the software development lifecycle. This effort should also involve internal IT and business consultation and alignment. When developing the strategy, CIOs will need to consider the following two core elements:

  1. What is the test strategy and plan?
  2. Where is the strategy to be executed: in-house, offshore or local?

What is the best strategy & plan?

Firms often don’t have time to test everything all the time. They have to make choices based on available time, market imperative, budget and available resources. Obviously, the greater the impact the application or product has on bottom line performance or legal risk the greater the testing attention it should get.

Before the strategy and plan are developed, test managers should sit down with their software development and business owner counterparts to ensure everyone is working off a clear set of product and technical requirements. Having requirements that are unclear or missing key information will compromise code quality, and further down the road, expand and complicate the testing effort.

A test strategy defined early in the software development cycle should include the following three steps:

  1. Define the objectives of each test stage and the testing techniques to be employed. Objectives will guide the level of testing effort, approach and resources needed and help management understand the overall effectiveness of the effort. Managers should also guarantee these objectives are consistent with overall corporate objectives.
  2. Analyze the risks, cost of code defects and revenue/productivity gains so management can make informed decisions about what, when and where to test. Some projects are so important that they require a comprehensive testing effort while others could use risk-based methodologies that focus only on the most mission-critical parts of the code.
  3. Understand available resources including internal staff and the testing tools required. Regardless of how testing is executed, managers need to understand the scale and scope of their project as well as their organizational readiness to manage the initiative.

Any testing strategy should be accompanied by a plan with sufficient detail to effectively manage the effort. This plan should include details on:

  • Test items
  • Features to be tested / not tested
  • Software risk issues
  • Item pass/fail criteria
  • Suspension criteria and resumption requirements
  • Security & privacy considerations
  • Milestones
  • Test deliverables
  • Resourcing needs- People, Infrastructure, Tools
  • Roles & responsibilities (all stakeholders)
  • Risks and contingencies
  • Approvals

A few areas in the strategy development phase require special consideration: Firstly, CIOs need to look at quality assurance as an end-to-end activity requiring executive-level accountability, SLA-based project ownership, and process standardization. Moreover, the right metrics need to be defined and standardized across all projects for easier tracking and analysis. Having a good testing strategy and plan is only a first step; organizational compliance is crucial to improve quality assurance. To ensure compliance, testing managers should consider utilizing proven quality processes like Lean or Six Sigma that streamline activities and reduce waste. Finally, testing is an activity that lends itself well to internal learning and continuous improvement. Managers should regularly debrief on their projects to ensure key learnings are captured and inculcated for later initiatives.

Where to execute the test strategy?

Increasingly, testing is being recognized as a strategic activity that is vital for organizational performance and product quality – as well as maximizing IT efficiencies. Ultimately, the testing location should be driven by the market and IT objectives defined early on as well as the perceived risks/opportunities and resourcing.

In the past, testing options were pretty clear-cut. Up until 2000, the default choice for many large companies was to perform all testing activities with internal resources. From the late 1990s to 2010, offshoring the software development cycle to India became popular as firms looked to reduce their testing expenditures and focus on core competencies. Starting around 2008, new market, economic and technical realities prompted CIOs to reconsider where they test various projects. These realities including rising Indian wage rates (in combination with moderating North American wages), the hassle of doing business halfway around the world and the emergence of innovative Canadian-based testing centres of excellence like the Test Factory™. Like many things in life, one size (i.e. testing location) rarely fits all. Nowadays, prudent CIOs recognize that there are many options on what to test, and where.

A more nuanced quality assurance approach has evolved whereby firms can now rely on a combination of offshore, in-house and locally outsourced testing operations. This approach seeks to leverage the best value from each model while minimizing risk. For example:

In-house

Testing within the organization is ideal for small, rush projects and where the company wants to preserve some testing competencies. However, maintaining an internal team could be expensive and be difficult to maintain at a world-class performance level.

Offshore

Using an offshore provider is the right choice for very large testing projects or where the client wants to outsource the entire software development cycle. At the same time, it is a challenge to secure the desired cost savings given the issues with testing in emerging countries. Moreover, it is not always smart having testing undertaken far away from key stakeholders (e.g., users, customers) and there are usually nagging doubts around security and privacy.

Outsourced But Onshore

Local (i.e. Canadian-based) testing operations can provide the best of both worlds: the Lowest total cost of testing, high quality and responsiveness coupled with low risk. However, local operations may have issues scaling up for projects requiring several hundred testers plus there are not many provider options at this time.

Related to the question around testing location is the potential role played by automation. These tools can deliver long-term cost savings and higher quality but are not suitable for every project and do require a high initial investment in scripts and tools. Finance and IT managers should perform a cost/benefit analysis before jumping in.

The final testing location mix should be based on a comprehensive cost, vendor and performance analysis. Prudent IT managers should experiment with different locations to find the optimal testing and organizational model.