QA Consultants was tasked with establishing an independent, full-featured performance testing and engineering service for Canada’s largest transit farecard platform. As a result, QAC delivered performance testing strategy, tools, automated performance scripting and execution, plus defect management with experienced onshore Canadian resources.
Highlights of this engagement include:
- Performance testing strategy
- Integrated performance test project planning
- Script creation & maintenance
- Multi-cycle perf test execution
- Results analysis & reporting
- Tools selection, setup, maintenance
- Knowledge transfer & client team training
- Overall project & resource management
- On-demand resource availability
- Laboratory performance testing environment
- Local onsite delivery team
- Local nearshore/offsite team
Metrolinx sought a partner with proven, world-class testing expertise to assist in the transformation of their performance testing with a focus on their Presto system. The successful system was experiencing significant growth including ongoing releases. QAC provided Metrolinx with a performance testing service with the following capabilities:
[su_table responsive=”no” alternate=”yes” fixed=”no” class=”Striped tables”]
| Performance Service Component | In-Scope for QAC? |
| Multi-vendor environment | ✔ |
| N-tier performance testing | ✔ |
| Complex integrated environments | ✔ |
| Web interface perf testing | ✔ |
| SOA/integration bus/service bus testing | ✔ |
| Data definition, setup, maintenance | ✔ |
| Legacy application components | ✔ |
| ERP performance testing | ✔ |
| .Net applications in scope | ✔ |
| Batch performance testing | ✔ |
[/su_table]
Project summary and background and significance
Metrolinx sought a partner with proven, world-class testing expertise to assist in the transformation of their performance testing with a focus on their Presto system.
QAC provided Metrolinx with a performance testing service with the following capabilities:
- Stream 1 – Define the overall program and project approach using a performance roadmap to set the strategy and common goals of the program
- A roadmap that defines people/skills/roles, process, and tools for a vibrant performance CoE
- QAC is responsible for directing and managing the delivery and outcomes, deliverables, and milestones of projects selected by the Ministry
- Working both independently and with other programs/project stakeholders; QAC is responsible for the creation and maintenance of the Project Plans (MS Project) and change management processes
- QAC is responsible for resource staffing, task and deliverable management, issue & risk management, and overall schedule
- Working with the client and within the delivery teams, QAC project managers are responsible for the work breakdown structure (task and deliverable scheduling) of the project during the development, implementation, and operations phases
- QAC is responsible for setting a budget and ensuring that projects are delivered within the budget and schedule
- QAC is responsible for project management reporting to senior MTO stakeholders on an ongoing basis and at scheduled points in the life cycle
- QAC project managers are required to prepare project plans, progress charts, and other items to assist in analyzing or displaying problems
- Stream 2 – Managed performance tools implementation with local HP tools specialist
- HP performance centre SaaS
- Maximum project delivery flexibility
- Maximum license flexibility (pay as you go)
- Experience integrating with third parties and offsite/data centre implementations
- Stream 3 – Fast, flexible script conversion service using onshore Micro Focus (HPE) Load Runner-skilled resources
- Maximize time to delivery
- Minimize cost
- Share application and process knowledge across the team for future project requirements
- Stream 4 – Ongoing performance architect and delivery management for Presto projects
- Best in class performance testing resources at all levels with relevant experience
- Implement a mature performance-as-a-service capability based on the roadmap and project requirements
- Stream 5 – Ongoing program and release support
- Onsite skills, off-site delivery capabilities to maximize coverage and budgets
Why now?
As an organization with significantly complex and public-facing applications, the client understood that end-to-end performance testing of their multi-vendor platform was critical to their business and image as the platform and usage grew. The client turned to QAC to ensure the successful provision of these services to:
- Manage the process to review and set up new performance testing projects and initiatives (e.g. onboarding, estimating, project sizing, etc.)
- Maintain and update reusable performance assets, standards, and processes/templates
- Maintain, configure/customize performance tools, code libraries, etc. to support new in-scope applications
- Identify future performance/load testing opportunities (releases, projects) using performance test scorecard
- Review performance/load testing processes, standards, frameworks, etc. required to be followed
- Provide performance/load testing requirements for new systems or major overhauls to existing systems
- Analyze the current state of the application and infrastructure, design, develop and execute automated performance tests to validate
- Compile realistic test data to simulate increased load/volume
- Support stress testing with proper simulation
- Provide detailed reports and metrics regarding performance outcomes to see where the system or program is both performing well and performing poorly
- Assess each application for its capability to handle the given number of concurrent users, transaction volumes with the available infrastructure
- Assess system infrastructure adequacy/scalability and certify the current application’s ability to perform under peak concurrent user volumes over a mutually agreed sustained duration of time for the identified transaction mix
- Execute batch performance at varied volumes to validate the throughput compliance of the platform over a mutually agreed sustained duration of time
What resources were devoted to this project?
QA Consultants utilized an expert team of performance testing process consultants, technical architects, tools specialists, and performance test automation engineers throughout the project. At its peak, the team included eight FTEs during the strategy and roadmap phase. Eventually, the project stabilized to six FTEs during the steady state test development phase.
Project aims/goals
- Manage the process to review and set up new performance testing projects and initiatives (e.g. onboarding, estimating, project sizing, etc.)
- Maintain and update reusable performance assets, standards, and processes/templates
- Maintain, configure/customize performance tools, code libraries, etc. to support new in-scope applications
- Identify future performance/load testing opportunities (releases, projects) using performance test scorecard
- Review performance/load testing processes, standards, frameworks, etc., required to be followed
- Provide performance/load testing requirements for new systems or major overhauls to existing systems
- Analyze the current state of the application and infrastructure, design, develop and execute automated performance tests to validate
- Compile realistic test data to simulate increased load/volume
- Support stress testing with proper simulation
- Provide detailed reports and metrics regarding performance outcomes to see where the system or program is both performing well and performing poorly
- Assess each application for its capability to handle the given number of concurrent users, transaction volumes with the available infrastructure
- Assess system infrastructure adequacy/scalability and certify the current application’s ability to perform under peak concurrent user volumes over a mutually agreed sustained duration of time for the identified transaction mix
- Execute batch performance at varied volumes to validate the throughput compliance of the platform over a mutually agreed sustained duration of time
Methodology/work undertaken
- Waterfall SDLC-based testing methodology was utilized based on QA Consultants’ performance testing methodology
- Performance test automation activities
- Organizational performance test strategy and test planning
- Performance test delivery management
- Performance test case development using Micro Focus Load Runner
- Performance test data development
- Test environment management
- Performance test execution and defect capture
- Defect management and resolution
- Performance test closure reporting
- Executive program quality presentations
Challenges/obstacles overcome
The client’s IT environment had multiple vendor partners with competing priorities and scope of responsibilities. Working as the client’s advocate, QAC successfully created an end-to-end performance testing strategy, clearly defined roles and responsibilities, and a metrics framework to support objective reporting. QAC was able to obtain support and buy-in from various teams such as the client’s system integrator, infrastructure partner, networking and telecom provider, and internal teams.
Results and Evaluation
Metrolinx valued QAC’s ability to provide independent performance testing services with local, and on-site resources. Furthermore, the client appreciated how QAC’s service goals aligned with their objectives:
- A cost-effective, robust performance testing program to support fast delivery speeds and quality demands
- A partner to help Presto mature their performance testing and engineering processes with skilled SMEs and best-in-class performance test methodologies, approaches, etc.
- Demonstrate vendor Independence while supporting and collaborating with current vendors
After three years, QAC is still engaged with this client to run their performance testing service. This type of partnership demonstrated successful delivery of automated performance testing skills, tools, and thought leadership. With over twenty-five years of experience, and a dedicated bench of testers (TestFactory™), QA Consultants delivered solid results.