DZone listed QAOps as one of the Top 6 Software Testing Trends to Look Out For in 2020.
This, of course, begs the question: what is QAOps? Why is it important? However, an explanation of QAOps cannot take place without first explaining an important related topic – DevOps.
Some Background: DevOps Explained
Before there was DevOps most discourse centred around CI/CD which refers to Continuous Integration, Continuous Delivery, Continuous Testing, and/or Continuous Deployment with respect to the organization of technology, process, and skills toward software delivery.
This CI/CD forms the very foundation of what we call the DevOps pipeline and its resultant approach.
Continuous Integration (CI) is the first phase of the pipeline and involves:
- Integrating code and content changes from multiple sources rapidly into a single main build of the application; and then,
- Executing various automated testing to verify the functionality of that build. The execution of such can collectively be referred to as Continuous Testing (CT).
Continuous Delivery (CD) is the natural follow on process of continuous integration. It involves the packaging of an application with all its deliverable artifacts together for delivery to end-users.
Automated building tools (or ‘DevOps tooling’) are employed to generate these packages. Ideally, at this stage, the packaged application should be ready to deploy to users at will.
Continuous Deployment is the third and final phase of this pipeline and is dependent upon the successful execution of both the prior CI and CD phases in that order. This phase involves the automatic launching and distribution of the packaged and acceptance-tested application to the targeted end-users.
DevOps, in turn, sits atop, and is dependent upon, the CI/CD approach. It goes some steps further however and incorporates cultural philosophies, practices, and tooling. When these items all work in concert as designed, they facilitate significant increases in an organization’s ability to:
- Deliver applications and services to clients much faster than pre-DevOps.
- Implement continual and iterative improvements to deliverable applications and products at a rate much faster than before DevOps.
- Better overall service and customer satisfaction through faster turnaround of apps and services.
- More competitiveness through quicker Time-to-Market.
- Overall high performance in the delivery of product and/or service.
DevOps Culture and Philosophy are an important aspect (possibly the most important) and, thus, key to the successful transition to this approach. It entails the following:
- The elimination of barriers between the disparate entities of development and operations to form and optimize both the productivity and reliability of both activities.
- Frequent and beneficial communication across Development, Operations, and QA teams to increase efficiencies and output.
- Ownership and shared responsibility expressed by all team members for the entire development process and infrastructure lifecycle.
- This mindset is well suited for Agile software development and management models which can be used to great effect within the DevOps approach.
QAOps Explained
QAOps can be explained in a few main concepts:
- It is the operations of running and orchestrating QA across the CI/CD pipeline (all the way from development start to deployment into production environments).
- It prescribes the QA process, rather than being a set of isolated activities performed here and there, to become an integral part of the DevOps process. This specific concept is also known as Continuous Quality (CQ) (this topic will be discussed in further detail in a future article).
- It looks to erode the functional and tribal barriers between QA, Development, and Operations Teams. QA practitioners are meant to work closely with development, operations, and anyone else within the CI/CD pipeline.
How to Implement QAOps
QAOps, in its current incarnation, can be successfully executed through the holistic adoption of Test Automation, Test Parallelization, Test Scalability, and what we can refer to as Holistic Connectivity & Collaboration. Let us take a quick look at each of these.
Automation
Automation is a central tenet of QAOps. It involves using technology and tools to mitigate the otherwise manual efforts expended upon repeatable testing and test-supporting activities that are characterized by:
- Automated build\deploy cycles involving-
- Environment allocation
- Code analysis, unit testing, & deployment
- Functional testing
- Performance testing
- Security analysis
This results in freeing up QA practitioners so their time can better be spent on more important activities such as:
- Test design, analysis, and optimization
- Defect analysis and processing
- Participation in the continual improvement of product and process
Test automation also accelerates the testing feedback loop which fulfills the cadence requirements of an effective QAOps setup. This can be accomplished via the following:
- The automation of repeatable testing (like Regression). Regression tests are usually large in number and are to be executed repeatedly, especially before major releases. Automating these tests removes the burden from the manual testers but also empowers the project to execute far more often than if performed manually.
- Tests for New Functionality can be automated once they have been identified as suitable for automation. Ideally, this will occur in the current sprint and be incorporated into the ‘living’ regression in time for the next sprint.
- The automation of Smoke and/or Sanity Tests. These tests can be subsets of regression or separately formed tests altogether. They are utilized to provide very quick feedback to all in the QAOps pipeline as to the viability of a particular build or release. Such quick feedback will facilitate quick action and turnaround.
- The automation of Unit Testing. Unit testing should typically be done by the Development team as the first check that their created code components (or units) behave as intended. A suite of automated unit tests should be executed whenever a new build is created.
- Automated testing is the perfect vehicle for efficient testing across Multiple Browser & O/S Combinations. Vast efficiencies, time, and cost savings are evident by employing this method versus trying to do this manually.
Test Parallelization
Parallel testing involves executing instances of the same tests, different tests, or any combination thereof across a multitude of target browsers and/or operating systems.
For example, rather than executing the automated regression suite first on O/S#1 and then O/S#2 immediately afterward, Parallel testing would, instead, prescribe the concurrent execution of the automated regression suite on both O/S. Another example could see the breaking up of the regression suite into smaller, disparate, test suites which could be, once again, executed at the same time. This would drastically reduce overall total testing time, making parallel testing an ideal component of QAOps.
Facilitating such parallelism in terms of target test environments, however, might require greater resourcing in terms of hardware and infrastructure. An alternate and cost-effective solution that is well suited to the QAOps methods could be cloud-based testing environments that could be accessed as required.
Scalability Testing
Scalability testing is the testing that is done against the capacity of a network, system, or process to adapt to expected and sudden changes. Applications must handle large increasing changes in data volume, simultaneous users, and other workloads. Scalability tests can ensure that the applications are ready for such situations prior to going live.
Scalability tests are designed to be scalable with the CI/CD pipeline and the results of such will allow the QA engineers to find and fix the performance issues of the application. Some of the key measures (or attributes) of importance – but not limited to – in this testing are response time, throughput, CPU usage, memory usage, and network usage.
Holistic Connectivity & Collaboration
The idea of improved collaboration, from a QA point of view, is made possible and realized through the following components:
- The usage of test automation tools – for Unit, Regression, API, Performance, and other types of testing – is noted given the automation-centric ideas behind QAOps.
- The usage of QA lifecycle and supportive tools are vital to enable critical activities like test design, planning, execution, and defect processing. Using tools that are not designed specifically for testing artifacts and tasks will lead to inefficiencies and a sub-optimal pipeline if any pipeline at all.
- Once these necessary QA lifecycle and supportive tools are in place. it is important to integrate the communications between them and the other Application lifecycle tools to realize a Tooling Ecosystem. This integrated Ecosystem entails the following points:
- Seamless transfer of important project information across tools to ensure real-time availability for the intended target audience. An example of this would be the ability to update a User Stories’ Acceptance Criteria (which is authored in Gherkin and may reside in a tool like JIRA) and then have this information automatically update the automated tests related to that Story (that may reside as Selenium Code or other open-source or commercial automation tool). This reduces onerous manual activities.
- Ease of linkage with regards to artifacts. For example, linking a test case to its corresponding Story, Requirement, Test Case runs, and Defects. This facilitates a greater breadth and depth of resultant metrics and reports. An obvious example of this is the ability to run a test coverage report that displays how much of the given requirements (or Stories) are covered by the existing test suite. The explicit linkage between the test and the requirement provides the necessary basis.
- The enhanced employment of metrics to aid decision making. Such can entail (but is not limited to):
- Applying risk and priority levels at the story level to drive the level of testing needed for that story and its features.
- Setting test coverage targets and then actively measuring against those targets at key intervals.
- Adjusting test coverage as per changing functionality and its associated risk level.
- All QA tasks and artifacts must be planned, executed, and measured like other project tasks. For example, all tasks and tests spawned from a User Story or Requirement should be processed like any other artifact in the project within the Tooling Ecosystem. QA tasks and activities should appear in task/asset lists or Boards as pieces of work that have a status, an estimate, an assignment, and any related info. The main idea is that all project tasks and artifacts should appear in one place and be given appropriate, if not equal, consideration in terms of importance.
- Collaboration: QA practitioners must fully participate in and contribute to a project just like Development and Operations. QA must work closely with other groups to ensure optimal execution of the pipeline. These activities can include:
- Participation in planning, daily stand-ups, retrospectives, and other relevant project activities.
- Cross training (if required) in some Development and Operations skills (and likewise for Dev and Ops).
- Cross tasking (when possible) – allowing QA to undertake some Development and/or Operations tasks (and likewise for Dev and Ops).
How will QAOps evolve?
The natural progression of QAOps evolution is likely to be enhanced by the following factors:
Predictive Analytics is something that could be facilitated by the collection of data that is inherent when an integrated and effective Tooling Ecosystem is in place. The metrics generated therein could form baselines in terms of testing or project performance precedents across different type of measures such as:
- Deployment Frequency
- Post-Production Incidents
- Defect Escape Rate
- Defect & Error Rates
- And more…
These baselined or precedent measures could be used to guide other projects of like size and content both within, across, and beyond an organization.
Machine Learning, a technique of Artificial Intelligence (AI), is considered the natural next step of Predictive Analytics. The incorporation of AI would potentially enhance the analytics in place. Machine learning would be able to make assumptions, test, and learn autonomously. As a result, it could provide even deeper insight into the data at hand.
Concluding Notes
In short, QAOps is an evolution of DevOps by providing greater insight and focus on optimizing QA. It espouses Test- Automation, Parallelization, Scalability, and Holistic Connectivity & Collaboration all in the interest of improving the quality of both product and process across the pipeline. In its further evolution, QAOps will no doubt involve Predictive Analytics and Artificial Intelligence. In future articles, I will be discussing these subjects and more.
Rob Virdee
Director, QA and DEV/OPS Transformation
Email: [email protected]