Much has been written on the best way to perform software testing. Surprisingly, little thought leadership has been written on choosing the right testing vendor. For many testing projects, the choice of provider can be the difference between financial and technical success or failure. This white paper will provide some best practices on vendor selection so that organizations can make better, more informed decisions around their testing activities.
Choosing the Ideal Testing Partner
Choosing the ideal testing partner is both a simple and a complicated exercise. Naturally, the testing strategy, vendor reputation and quoted cost will go a long way in defining a short list and influencing the selection – as it should be. However, firms should not stop there. Other factors are also important, and should be considered in due course. Moreover, considering a wide range of selection factors is also critical because projects will vary and clients & vendors mesh differently.
Prudent CIOs, CTOs and CFOs should consider the following six questions to ensure they choose the best provider for their project and company.
1. Do they have sufficient critical mass?
When it comes to software quality assurance, a vendor’s critical mass – their testing focus, size, pedigree and scalability – should be a major consideration:
Focus – Many IT services firms profess to be expert testers but are really software developers or contractors looking for an opportunistic sale. In general, the tighter the vendor’s focus on testing, the more experience and skill they will possess.
Size – Having sufficient size enables the provider to invest in a robust and secure infrastructure, develop automation as well as to train its staff. Larger firms also tend to be better at capturing a wide range of industry knowledge as well as tracking client information and delivering service excellence.
Pedigree – Also known as brand image, pedigree is the level of trust, relevance and credibility in a company, as seen by the market. Pedigree is a function of the testers client roster, thought leadership, time in business, etc. Having a strong pedigree also helps attract the right staff and partnerships, and is a sign of financial stability.
Scalability – Clients need to know whether their vendor can quickly deploy enough testers and tools to finish or accelerate a project without compromising quality, price or service. In particular, ‘On-Demand’ scalability is helpful when the project has uncertain requirements and tight time lines. An On-demand capability will also add value by making testing a variable, as opposed to a fixed (e.g. headcount, infrastructure), cost.
There is no rule of thumb on what should be the minimum size for a testing partner. However, fully-capable vendors tend to employ at least 100 dedicated testers; have a well-honed local operation and; have a referenceable list of clients in your industry.
2. How do they protect your intellectual property, customer data and brand assets?
These days, safeguarding critical information is table stakes. Clients should insist that their providers have robust security measures that meet or exceed their own infrastructure. This is especially important with cutting-edge technologies like mobile and cloud computing that may carry serious security and enterprise risks.
3. What value-add, such as innovation or though leadership, do they bring?
Firms who make decisions based solely on price, delivery or technical support are ignoring a lot of potential business value. This value could include access to industry best practices, available complementary services like project tracking or consulting, and pre-built automation.
Furthermore, it is important to understand the tester’s commitment to continuous improvement. With long term projects, testers will gain experience effects that can lead to lower client costs, more streamlined processes and more creative problem solving.
4. How deep is their industry experience?
To maximize project ROI and payback, utilizing a testing team with industry experience is almost as important as technical proficiency. The more complex the industry (eg. insurance, banking and healthcare), the more the testing group will need familiarity or experience with the key industry issues like regulatory changes and heterogeneous IT infrastructures. However, it may be difficult to know the level of industry expertise. Clients should explore whether their providers have low worker turnover, run new employee on-boarding programs and have ongoing industry training.
5. Is the vendor, delivery or client driven?
A tester’s business model goes a long way in determining their service offering, the organizational fit with the client and the ultimate value provided. Delivery-driven testers usually provide their services through a single model to minimize cost. Offshore testers and recruiting agencies are examples of this model. These vendors are often a good solution for simple, clearly-defined and low-risk projects. However, clients using delivery-focused testers face a key question: Is a ‘one size fits all’ model the best way to satisfy project requirements, business needs, and integrate seamlessly with the clients workflows, teams and culture?
On the other hand, client-driven testers look to build their delivery model back from the client need. These partners can design the optimal testing configuration for each project and deploy a bespoke service and relationship model that meshes with the clients organization. This model is ideal when the testing needs are complex or change quickly; there are many internal and external stakeholders and where organizational fit is key.
6. Are the any hidden conflicts that could compromise testing integrity?
To ensure superior code quality and avoid conflicts around priorities, testing and development should be separate functions undertaken by different groups. Having proper vendor objectivity and process governance ensures the testing effort is correctly designed, implemented and resourced.
Beyond price and delivery, there is much to consider within a vendor selection process. Each of the above questions, if not properly considered, can result in poor project outcomes, increased cost, and reduced application and product performance. As such, managers should regularly perform provider assessments – both for new and existing vendors – to ensure they are receiving sufficient value and benefiting from the latest advances in testing innovation and business model improvements.