QA Consultants’ performance testing runs full steam ahead with transit client

QA Consultants was tasked with establishing an independent, full-featured performance testing and engineering service for Canada’s largest transit farecard platform. As a result, QA Consultants delivered performance testing strategy, tools, automated performance scripting and execution, plus defect management with experienced onshore Canadian resources.

Highlights of this engagement include:

• Performance testing strategy
• Integrated performance test project planning
• Script creation & maintenance
• Multi-cycle perf test execution
• Results analysis & reporting
• Tools selection, setup, maintenance
• Knowledge transfer & client team training
• Overall project & resource management
• On-demand resource availability
• Laboratory performance testing environment
• Local onsite delivery team
• Local nearshore/offsite team

Metrolinx sought a partner with proven, world-class testing expertise to assist in the transformation of their performance testing with a focus on their Presto system. The successful system was experiencing significant growth including ongoing releases. QA Consultants provided Metrolinx with a performance testing service with the following capabilities:

Project summary and background and significance

Metrolinx sought a partner with proven, world-class testing expertise to assist in the transformation of their performance testing with a focus on their Presto system.

QA Consultants provided Metrolinx with a performance testing service with the following capabilities:

• Stream 1 – Define the overall program and project approach using a performance roadmap to set the strategy and common goals of the program
  • A roadmap that defines people/skills/roles, process, and tools for a vibrant performance CoE
  • QA Consultants is responsible for directing and managing the delivery and outcomes, deliverables, and milestones of projects selected by the Ministry
  • Working both independently and with other programs/project stakeholders; QA Consultants is responsible for the creation and maintenance of the Project Plans (MS Project) and change management processes
  • QA Consultants is responsible for resource staffing, task and deliverable management, issue & risk management, and overall schedule
  • Working with the client and within the delivery teams, QA Consultants project managers are responsible for the work breakdown structure (task and deliverable scheduling) of the project during the development, implementation, and operations phases
  • QA Consultants is responsible for setting a budget and ensuring that projects are delivered within the budget and schedule
  • QA Consultants is responsible for project management reporting to senior MTO stakeholders on an ongoing basis and at scheduled points in the life cycle
  • QA Consultants project managers are required to prepare project plans, progress charts, and other items to assist in analyzing or displaying problems

 

• Stream 2 – Managed performance tools implementation with local HP tools specialist
  • HP performance centre SaaS
  • Maximum project delivery flexibility
  • Maximum license flexibility (pay as you go)
  • Experience integrating with third parties and offsite/data centre implementations

 

• Stream 3 – Fast, flexible script conversion service using onshore Micro Focus (HPE) Load Runner-skilled resources
  • Maximize time to delivery
  • Minimize cost
  • Share application and process knowledge across the team for future project requirements

 

• Stream 4 – Ongoing performance architect and delivery management for Presto projects
  • Best in class performance testing resources at all levels with relevant experience
  • Implement a mature performance-as-a-service capability based on the roadmap and project requirements

 

• Stream 5 – Ongoing program and release support
  • Onsite skills, off-site delivery capabilities to maximize coverage and budgets

Why now?

As an organization with significantly complex and public-facing applications, the client understood that end-to-end performance testing of their multi-vendor platform was critical to their business and image as the platform and usage grew. The client turned to QA Consultants to ensure the successful provision of these services to:

• Manage the process to review and set up new performance testing projects and initiatives (e.g. onboarding, estimating, project sizing, etc.)
• Maintain and update reusable performance assets, standards, and processes/templates
• Maintain, configure/customize performance tools, code libraries, etc. to support new in-scope applications
• Identify future performance/load testing opportunities (releases, projects) using performance test scorecard
• Review performance/load testing processes, standards, frameworks, etc. required to be followed
• Provide performance/load testing requirements for new systems or major overhauls to existing systems
• Analyze the current state of the application and infrastructure, design, develop and execute automated performance tests to validate
• Compile realistic test data to simulate increased load/volume
• Support stress testing with proper simulation
• Provide detailed reports and metrics regarding performance outcomes to see where the system or program is both performing well and performing poorly
• Assess each application for its capability to handle the given number of concurrent users, transaction volumes with the available infrastructure
• Assess system infrastructure adequacy/scalability and certify the current application’s ability to perform under peak concurrent user volumes over a mutually agreed sustained duration of time for the identified transaction mix
• Execute batch performance at varied volumes to validate the throughput compliance of the platform over a mutually agreed sustained duration of time

 

What resources were devoted to this project?

QA Consultants utilized an expert team of performance testing process consultants, technical architects, tools specialists, and performance test automation engineers throughout the project. At its peak, the team included eight FTEs during the strategy and roadmap phase. Eventually, the project stabilized to six FTEs during the steady state test development phase.

Project aims/goals

• Manage the process to review and set up new performance testing projects and initiatives (e.g. onboarding, estimating, project sizing, etc.)

• Maintain and update reusable performance assets, standards, and processes/templates
• Maintain, configure/customize performance tools, code libraries, etc. to support new in-scope applications
• Identify future performance/load testing opportunities (releases, projects) using performance test scorecard
• Review performance/load testing processes, standards, frameworks, etc., required to be followed
• Provide performance/load testing requirements for new systems or major overhauls to existing systems
• Analyze the current state of the application and infrastructure, design, develop and execute automated performance tests to validate
• Compile realistic test data to simulate increased load/volume
• Support stress testing with proper simulation
• Provide detailed reports and metrics regarding performance outcomes to see where the system or program is both performing well and performing poorly
• Assess each application for its capability to handle the given number of concurrent users, transaction volumes with the available infrastructure
• Assess system infrastructure adequacy/scalability and certify the current application’s ability to perform under peak concurrent user volumes over a mutually agreed sustained duration of time for the identified transaction mix
• Execute batch performance at varied volumes to validate the throughput compliance of the platform over a mutually agreed sustained duration of time

Methodology/work undertaken

• Waterfall SDLC-based testing methodology was utilized based on QA Consultants’ performance testing methodology
• Performance test automation activities
  • Organizational performance test strategy and test planning
  • Performance test delivery management
  • Performance test case development using Micro Focus Load Runner
  • Performance test data development
  • Test environment management
  • Performance test execution and defect capture
  • Defect management and resolution
  • Performance test closure reporting
  • Executive program quality presentations

Challenges/obstacles overcome

The client’s IT environment had multiple vendor partners with competing priorities and scope of responsibilities. Working as the client’s advocate, QA Consultants successfully created an end-to-end performance testing strategy, clearly defined roles and responsibilities, and a metrics framework to support objective reporting. QA Consultants was able to obtain support and buy-in from various teams such as the client’s system integrator, infrastructure partner, networking and telecom provider, and internal teams.

Results and Evaluation

Metrolinx valued QA Consultants’ ability to provide independent performance testing services with local, and on-site resources. Furthermore, the client appreciated how QA Consultants’ service goals aligned with their objectives:

• A cost-effective, robust performance testing program to support fast delivery speeds and quality demands
• A partner to help Presto mature their performance testing and engineering processes with skilled SMEs and best-in-class performance test methodologies, approaches, etc.
• Demonstrate vendor Independence while supporting and collaborating with current vendors

After three years, QA Consultants is still engaged with this client to run their performance testing service. This type of partnership demonstrated successful delivery of automated performance testing skills, tools, and thought leadership. With over twenty-five years of experience, and a dedicated bench of testers (TestFactory™), QA Consultants delivered solid results.