Redefining QA: Parallel System Verification

Abstract:

This document stands as a primer on the use of System Oriented Programming as a QA Model for the creation of QA solutions that depart from the traditional approach of User Stories and Test Cases driving individual tests and scenarios. Often, the “long pole in the tent” in traditional software testing is the duration it takes to properly set up test data and establish a scenario to be tested. Often, subtle but important changes to the code base and application under test are not able to be tested until later in the customer or workflow lifecycle, often transpiring over logical date changes, triggering backend jobs, etc. This is an intensive and repeated process that while often automated, still increases the “Time to Test” measure from the moment code and environments are delivered until the specific test cases can be executed.

System Oriented Programming, which is a modern approach to software development focused on performance and simplicity and has multiple industry uses beyond QA, uniquely provides an alternative use for the creation of parallel test systems.
This approach, currently in use in Texas for energy markets allows for rapid validation of system changes without the need for arduous and complex test cycles and scenarios, but rather, a completed end to end validation of core system processes with mathematical proof points of system correctness.

As a company that invests in exploring the future of Software Quality Assurance, this is a solution that still requires more modeling and examples. However, given the early success and the applicability to our mission, we feel it is worth considering in large transactional systems testing and it is our obligation continue to explore it.

Authors:

Michael Cation

Cloud Sliver
 

Brian Bernknopf

Managing Director, U.S. Ops

Brian Bernknopf is the Managing Director for U.S. Operations at QA Consultants.
Mr. Bernknopf currently serves as the Managing Director of QA Consultants USA, Inc. (QAC). He is responsible for the daily operations and management of the US business, ongoing sales and business development, staff and project management, and solution creation. QAC has been one of North America’s fasted growing independent software testing services firms over the last 20 years and Mr. Bernknopf is responsible for their US expansion. Mr. Bernknopf is also a board member to the Dallas Per Scholas program that has the mission of breaking the cycle of poverty through IT Job and Workforce Training. Mr. Bernknopf is a proven industry leader with a deep specialization in IT Consulting and Advisory, Software Quality Assurance, and IT Systems Integration. He has spent his 20-year career in management consulting and while having worked in multiple industry verticals and with a variety of software technologies, has always been focused on software quality management and governance. His responsibilities have included sales, delivery, and management of IT solutions and engagements focused on: QA Governance, Test Automation, Performance, Mobile QA, ERP (SAP), Package Applications (ERP\BPM\Etc.), COE Development, and complex QA solutions. Prior to joining QAC, Mr. Bernknopf was a Senior Vice President for KMS and a Vice President at Virtusa, where he led a 2000 person QA practice. Before Virtusa, Mr. Bernknopf was a Principal at Capgemini and was Director of their Insurance QA Practice. In addition to QA, Brian has experience in large program and project management, alliance management, and marketing. Mr. Bernknopf has over 20 years in the Quality Assurance and Quality Management industry and is a frequent speaker at Quality Management events. His early patent applications for test automation built the foundation for his QA career and a pursuit of QA Solutions

Expand to see all text about Brian Bernknopf

Collapse text

November 05, 2020

100% Test Coverage
With System Oriented Programming
A New Testing Paradigm

When QA Consultants begins a system QA assessment, we are often looking for the right balance between the cost of quality and speed to market. Of course, doing that without a sacrifice to quality beyond the allowable risk levels of a particular system. When applicable, if it is a high transactional system or high-volume processing product, we are also evaluating if a newly emerging technique in QA, “Parallel System Verification”, is applicable. This process allows us to mathematically ‘prove’ that a system is performing as expected. Not by running hypothetical test scenarios, but by running real data, in high volume, with real expected results.

The higher the volume, the better the math.

In short, if two systems with the same data input produce the same results, there is a high probability they are performing correctly.

Certainly, the system under test must be 100% separated from the parallel test system. And further, that parallel test system must also be validated against a core data set to be accurate.

Using new technologies such as System Oriented Programming allows us to rapidly stand up a parallel system to deliver 100%, continuous test coverage as a viable alternative.

We are currently seeing that continuous testing with 100% test coverage has delivered significant benefits. Among these is the ability to essentially eliminate transactional production errors. Another benefit is a dramatic cost reduction in ongoing QA costs while increasing the surface area for testing critical new app features.

The Power of Parallel Execution: Continuous Parallel Testing and Coverage

One of the primary advantages of using System Oriented Programming in the development of enterprise software is its ability to rapidly process significant amounts of data. Our software partners use this feature to run entire days, months, or years of input data through the system under test, as a parallel system. Here, two systems, with different codebases, are performing what should be identical tasks. The independent QA as a Service vendor, QA Consultants, measures every transaction from both systems to determine, line by line if they are identical.

This is a two-step process that first validates that the parallel application accurately matches the existing legacy system. Through this process, multiple years of data are run and reconciled for correctness.

Once the parallel System Oriented Programming application is validated, the second step is to make the suggested software changes to both the legacy system and the parallel System Oriented Programming system. Then the data sets are re-run looking for expected and unexpected outcomes. The outcome differences are analyzed and confirmed for defects.

Our experience has shown a 100% accuracy rate for the System Oriented Programming parallel system in identifying production defects when there is a mismatch between the two systems.

When calculations from two independent systems, each with different codebases, reach the same conclusion, the probability that the final product – i.e. a utility bill — is correct, approaches 100%.

The volume of data allows for the mathematical certainty of system sameness.

Eliminate Testing Through Validation: Testing a System of Record as New Features are Added

As enhancements to the system of record become available, the System Oriented Programming approach allows the QA team to build them into a version of the parallel system.

Because the System Oriented Programming technology enables complex applications to be modified, configured and delivered rapidly, it is practical to construct parallel systems for each set of feature enhancements. With this approach, every line item is reconciled for every calculation in real-time. The system of record can be checked against an established parallel system and reconciled against one or more additional parallel systems to ensure that all of them agree.

This process is continuous. The parallel systems can be validated 24/7. Across years of historical data.

When an anomaly appears, there is immediate feedback. As each new system is validated and then moved over to the role of “primary parallel system”, it is possible to introduce significant updates into a production system and test them at the level of every line item, on every bill or other output, for every customer for 100% test coverage of the output of the updated system.

Not 100% hypothetical coverage, 100% historical coverage. Complicate “edge” scenarios or complex processes are all accounted for. There is ZERO QA time spent on the creation of test data.

Further, the comparison of expected results to actual results also takes place in a Systems Oriented Programming comparison solution allowing for rapid validation.

Since Systems Oriented Programming applications are more efficient than traditional applications, the hardware requirements are minimal. This makes it economical to run multiple systems in parallel without breaking the program budget. It also makes it possible to do extensive retrospective testing – for example, running three years of transactions through each of the parallel systems on a regular basis to find anomalies.

It also opens the possibilities up for fraud detection and hypothetical scenario modeling.

Summary

System Oriented Programming approaches applied for enterprise applications have been in production for seven years. They have been used for 100% continuous testing with great success.

We have seized this technology and created a new way to test and validate transactional oriented applications on an ongoing\continuous basis to deliver a higher level of quality than what has been previously possible and at a speed that allows for rapid system deployments with low risk.

QA Consultants believes that this unique approach to quality, validation, and velocity is a significant mindset change for IT organizations, and we further believe it is ground-breaking. Its current use means that we think there would be some relevance in considering this approach as new systems come online or legacy systems go through refresh and modernization.

There are many options for ETL validation as well as traditional system maintenance and upgrades.