Skip to content

Redefining QA: Parallel System Verification

Abstract:

This document stands as a primer on the use of System Oriented Programming as a QA Model for the creation of QA solutions that depart from the traditional approach of User Stories and Test Cases driving individual tests and scenarios. Often, the “long pole in the tent” in traditional software testing is the duration it takes to properly set up test data and establish a scenario to be tested. Often, subtle but important changes to the code base and application under test are not able to be tested until later in the customer or workflow lifecycle, often transpiring over logical date changes, triggering backend jobs, etc. This is an intensive and repeated process that while often automated, still increases the “Time to Test” measure from the moment code and environments are delivered until the specific test cases can be executed.

System Oriented Programming, which is a modern approach to software development focused on performance and simplicity and has multiple industry uses beyond QA, uniquely provides an alternative use for the creation of parallel test systems.
This approach, currently in use in Texas for energy markets allows for rapid validation of system changes without the need for arduous and complex test cycles and scenarios, but rather, a completed end to end validation of core system processes with mathematical proof points of system correctness.

As a company that invests in exploring the future of Software Quality Assurance, this is a solution that still requires more modeling and examples. However, given the early success and the applicability to our mission, we feel it is worth considering in large transactional systems testing and it is our obligation continue to explore it.

Authors:

 
[row] [col columns=”2″ breakpoint=”md”]
[/col] [col columns=”4″ breakpoint=”md”]

Michael Cation

Cloud Sliver
 
LinkedIn Michael Cation
[/col] [col columns=”2″ breakpoint=”md”]
[/col] [col columns=”4″ breakpoint=”md”]

Brian Bernknopf

Managing Director, U.S. Ops

LinkedIn Brian Bernknopf
[/col] [/row]  
November 05, 2020


100% Test Coverage
With System Oriented Programming
A New Testing Paradigm

When QA Consultants begins a system QA assessment, we are often looking for the right balance between the cost of quality and speed to market. Of course, doing that without a sacrifice to quality beyond the allowable risk levels of a particular system. When applicable, if it is a high transactional system or high-volume processing product, we are also evaluating if a newly emerging technique in QA, “Parallel System Verification”, is applicable. This process allows us to mathematically ‘prove’ that a system is performing as expected. Not by running hypothetical test scenarios, but by running real data, in high volume, with real expected results.

The higher the volume, the better the math.

In short, if two systems with the same data input produce the same results, there is a high probability they are performing correctly.

Certainly, the system under test must be 100% separated from the parallel test system. And further, that parallel test system must also be validated against a core data set to be accurate.

Using new technologies such as System Oriented Programming allows us to rapidly stand up a parallel system to deliver 100%, continuous test coverage as a viable alternative.

We are currently seeing that continuous testing with 100% test coverage has delivered significant benefits. Among these is the ability to essentially eliminate transactional production errors. Another benefit is a dramatic cost reduction in ongoing QA costs while increasing the surface area for testing critical new app features.
[row] [col columns=”1″ breakpoint=”md”] [/col] [col columns=”11″ breakpoint=”md”]

The Power of Parallel Execution: Continuous Parallel Testing and Coverage

One of the primary advantages of using System Oriented Programming in the development of enterprise software is its ability to rapidly process significant amounts of data. Our software partners use this feature to run entire days, months, or years of input data through the system under test, as a parallel system. Here, two systems, with different codebases, are performing what should be identical tasks. The independent QA as a Service vendor, QA Consultants, measures every transaction from both systems to determine, line by line if they are identical.

This is a two-step process that first validates that the parallel application accurately matches the existing legacy system. Through this process, multiple years of data are run and reconciled for correctness.

Once the parallel System Oriented Programming application is validated, the second step is to make the suggested software changes to both the legacy system and the parallel System Oriented Programming system. Then the data sets are re-run looking for expected and unexpected outcomes. The outcome differences are analyzed and confirmed for defects.

Our experience has shown a 100% accuracy rate for the System Oriented Programming parallel system in identifying production defects when there is a mismatch between the two systems.

When calculations from two independent systems, each with different codebases, reach the same conclusion, the probability that the final product – i.e. a utility bill — is correct, approaches 100%.

The volume of data allows for the mathematical certainty of system sameness.

Eliminate Testing Through Validation: Testing a System of Record as New Features are Added

As enhancements to the system of record become available, the System Oriented Programming approach allows the QA team to build them into a version of the parallel system.

Because the System Oriented Programming technology enables complex applications to be modified, configured and delivered rapidly, it is practical to construct parallel systems for each set of feature enhancements. With this approach, every line item is reconciled for every calculation in real-time. The system of record can be checked against an established parallel system and reconciled against one or more additional parallel systems to ensure that all of them agree.

This process is continuous. The parallel systems can be validated 24/7. Across years of historical data.

When an anomaly appears, there is immediate feedback. As each new system is validated and then moved over to the role of “primary parallel system”, it is possible to introduce significant updates into a production system and test them at the level of every line item, on every bill or other output, for every customer for 100% test coverage of the output of the updated system.

Not 100% hypothetical coverage, 100% historical coverage. Complicate “edge” scenarios or complex processes are all accounted for. There is ZERO QA time spent on the creation of test data.

Further, the comparison of expected results to actual results also takes place in a Systems Oriented Programming comparison solution allowing for rapid validation.

Since Systems Oriented Programming applications are more efficient than traditional applications, the hardware requirements are minimal. This makes it economical to run multiple systems in parallel without breaking the program budget. It also makes it possible to do extensive retrospective testing – for example, running three years of transactions through each of the parallel systems on a regular basis to find anomalies.

It also opens the possibilities up for fraud detection and hypothetical scenario modeling.
[/col] [/row]

Summary

System Oriented Programming approaches applied for enterprise applications have been in production for seven years. They have been used for 100% continuous testing with great success.

We have seized this technology and created a new way to test and validate transactional oriented applications on an ongoing\continuous basis to deliver a higher level of quality than what has been previously possible and at a speed that allows for rapid system deployments with low risk.

QA Consultants believes that this unique approach to quality, validation, and velocity is a significant mindset change for IT organizations, and we further believe it is ground-breaking. Its current use means that we think there would be some relevance in considering this approach as new systems come online or legacy systems go through refresh and modernization.

There are many options for ETL validation as well as traditional system maintenance and upgrades.