QAC Automotive and Robotics (QAaR) Quality Assurance
FOCUSED ON THE FUTURE
Welcome to the fifth edition of the QAaR newsletter. In this edition, we’ll focus on sharing more information about model-based testing and why they are needed for automotive system testing. We will be presenting how QAC developed different ways to abstract real-world scenarios and develop them through models that can be analyzed by algorithms and create test cases from it.
Keeping you informed
Our automotive and robotics quality assurance workstreams
Software testing is evolving, and model-based testing is an integral piece of modern test automation. Model-based testing is a testing approach where test cases are automatically generated from models. Those models are the expected behaviour of the system under test (SUT) and can be used to represent the overall testing strategy. A model is an abstraction of system requirements. From the QA perspective, we can leverage to ensure large scalable test scenarios. A graph is used to represent different artefacts of the system as a node which can have different states, and edges represent the relationships between different permutations of the system are the generated and evaluated based on the relationships and rules. The model can be executed as a finite state machine or as a constraint logic machine.
Algorithms developed by QAC scan meta-models and identifies each component, its respective states, all relationships with other components, business rules for each component and activators, and with all that information, automatically generate test cases with different percentage of coverage. Model-based testing can be used for different domains and some of the industry examples on how QAC has leveraged model-based testing are presented in the following sections.
Autonomous robot vehicle model representation
Mesh Model with Static and Dynamic Obstacles
The robot vehicle needs to keep capturing what’s going on around it and taking decisions constantly until it reaches the final goal without colliding or causing any damage. Add all this complexity to specific requirements and rules that the robot needs to follow, and you can realize how hard it would be to manually think and design test cases for robot systems. QAC team was able to create a model that represents and maps all those situations, and an algorithm that scans all nodes, representing different entities and the interaction and rules between them. Uncountable test cases are generated, covering different permutations in different stages and repetition, to simulate and verify how the robot vehicle approaches and behaves.
Connected vehicle model representation
Automotive relative components
In this MBT approach, a list of total possible combinations of all component’s states (which could be physical or virtual components) is generated, considering the set of rules (conditional statements) based on the system’s requirements and integrated into the model. Using the developed system model with associated rules, a set of test cases is generated through a graph traversal technique (Depth First Search).
This model represents a very small subset of the actual full automotive model mapping. In this example, it shows how “gear” impacts other components such as rear camera, rear sensor and multimedia volume. A simple gear action might generate hundreds of test cases. Can you imagine the number of test cases in a full automotive system? Hundreds of thousands!
Insurance claim model representation
Applying similar approach that QAC has been using for robot and automotive vehicles, we are able to create models for other domains such as Insurance, by abstracting business rules and representing them in a graph-based model, which nodes represent entities that are part of a specific business flow.
In this example, you can notice a small part of how insurance premium is calculated. The possible combinations depending on the different entries in a policy quote and the different scenarios can be quickly generated and prioritized by risk factors. With a risk-driven test execution, issues related to the most important components in the system can be identified. Whenever the requirements change, all test cases will be automatically updated, saving a lot of hours of rework for redesigning and executing tests.
Some advantages of model testing:
- Easy test case/suite maintenance
- High test coverage
- Reduction in test costs
- Early defect detection
- Increase in defect count
- Time savings
Prioritization of test cases and smart testing
Having an effective prioritization mechanism in place for the test suite execution to detect faults early is a challenging task. It’s due to the limited number of resources in terms of time and computation power. Considering the system requirements, defining appropriate criteria for prioritization of test cases is the main cause of uncertainty. For automotive and robot vehicle domains, QAC has developed a scoring method for prioritization of test suite. In the context of connected and autonomous vehicles, safety and security risks are determined to be two critical dimensions for optimal test case prioritization. Utilizing the functional safety of road vehicles (ISO 26262), a scoring method was developed by performing hazard analysis and risk assessment (HARA), followed by evaluating various risk parameters (Severity, Exposure, and Controllability). For different domains, business risk and severity of potential defects are considered.
Prioritization of automotive tests using ISO 26262, 13849, & IEC 62061
Our test case prioritization process follows industry standards for both road and robotics standards to ensure our customers are focused on critical systems and can communicate to their clients that they are aware and following best testing practices.
Re-prioritization during execution by Machine Learning algorithms
QAC has also developed algorithms that analyze the results of the tests during their execution to learn which components or business rules present less or more defects, and based on that information, reprioritized the remaining test cases to focus on items with many, or no defects.