QAC Automotive and Robotics (QAaR) Quality Assurance
This monthly newsletter will focus on QAC’s activities regarding Automotive and Robotics Quality Assurance Services.
FOCUSED ON THE FUTURE
Leading the way
Welcome to the fifth edition of the QAaR newsletter. In this edition, we’ll focus on sharing more information about model-based testing and why they are needed for automotive system testing. We will be presenting how QAC developed different ways to abstract real-world scenarios and develop them through models that can be analyzed by algorithms and create test cases from it.
Welcome to our 5th edition of our Automotive and Robotics Quality Assurance Newsletter
Keeping you informed
Our automotive and robotics quality assurance workstreams
Research and Grant Projects
Grant budgets allow for research and development of new technologies that position QAC to become a world leader in quality assurance services.
AQS (Automotive Quality Services)
Testing and Quality Assurance services exclusively developed for Automotive and Autonomous Road vehicles.
RQS (Robotic Quality Services)
Testing and Quality Assurance services exclusively developed for Robotics and Autonomous small vehicles.
Safety and Cybersecurity
Focus on developing services that adhere to ISO’s compliance verification testing automation Cybersecurity, and Connectivity standards.
Model-Based Testing
Software testing is evolving, and model-based testing is an integral piece of modern test automation. Model-based testing is a testing approach where test cases are automatically generated from models. Those models are the expected behaviour of the system under test (SUT) and can be used to represent the overall testing strategy. A model is an abstraction of system requirements. From the QA perspective, we can leverage to ensure large scalable test scenarios. A graph is used to represent different artefacts of the system as a node which can have different states, and edges represent the relationships between different permutations of the system are the generated and evaluated based on the relationships and rules. The model can be executed as a finite state machine or as a constraint logic machine.
Algorithms developed by QAC scan meta-models and identifies each component, its respective states, all relationships with other components, business rules for each component and activators, and with all that information, automatically generate test cases with different percentage of coverage. Model-based testing can be used for different domains and some of the industry examples on how QAC has leveraged model-based testing are presented in the following sections.
Autonomous robot vehicle model representation
Mesh Model with Static and Dynamic Obstacles
When you think about scenarios and different possible situations that an autonomous robot vehicle can be inserted in the real world, it might be hard to list them out, once we need to consider, among so many other things, people and crowd gathering and dispersing, different moving objects and vehicles, furniture, equipment, walls, entrances, transparent and reflective obstacles, terrain conditions, holes, inclinations, weather conditions, such as sun, light, rain, snow, fog, wind, etc.
The robot vehicle needs to keep capturing what’s going on around it and taking decisions constantly until it reaches the final goal without colliding or causing any damage. Add all this complexity to specific requirements and rules that the robot needs to follow, and you can realize how hard it would be to manually think and design test cases for robot systems. QAC team was able to create a model that represents and maps all those situations, and an algorithm that scans all nodes, representing different entities and the interaction and rules between them. Uncountable test cases are generated, covering different permutations in different stages and repetition, to simulate and verify how the robot vehicle approaches and behaves.
Example of how Autonomous Robot Vehicle can be represented in a model:
Connected vehicle model representation
Automotive relative components
Example of how automotive components can be represented in a model:
Nowadays, a vehicle is not only a set of mechanical devices and equipment but basically dozens of software and ECU’s (Electronic Control Unit) under wheels that interact with each other all the time. Every time specific components are enabled, by the driver or passengers, through a mechanic action, or even through an ECU, a set of other components behave accordingly as an answer for that trigger.
In this MBT approach, a list of total possible combinations of all component’s states (which could be physical or virtual components) is generated, considering the set of rules (conditional statements) based on the system’s requirements and integrated into the model. Using the developed system model with associated rules, a set of test cases is generated through a graph traversal technique (Depth First Search).
This model represents a very small subset of the actual full automotive model mapping. In this example, it shows how “gear” impacts other components such as rear camera, rear sensor and multimedia volume. A simple gear action might generate hundreds of test cases. Can you imagine the number of test cases in a full automotive system? Hundreds of thousands!
Insurance claim model representation
Using a similar idea applied in robot and automotive vehicles, other domains, such as Insurance can also take advantage of Model-Based testing. QAC has developed a way to abstract insurance business rules and represent it in a model that can have test cases automatically.
Applying similar approach that QAC has been using for robot and automotive vehicles, we are able to create models for other domains such as Insurance, by abstracting business rules and representing them in a graph-based model, which nodes represent entities that are part of a specific business flow.
In this example, you can notice a small part of how insurance premium is calculated. The possible combinations depending on the different entries in a policy quote and the different scenarios can be quickly generated and prioritized by risk factors. With a risk-driven test execution, issues related to the most important components in the system can be identified. Whenever the requirements change, all test cases will be automatically updated, saving a lot of hours of rework for redesigning and executing tests.
Some advantages of model testing:
- Easy test case/suite maintenance
- High test coverage
- Reduction in test costs
- Early defect detection
- Increase in defect count
- Time savings
Example of how an insurance premium calculation can be represented in a model:
Prioritization of test cases and smart testing
Having an effective prioritization mechanism in place for the test suite execution to detect faults early is a challenging task. It’s due to the limited number of resources in terms of time and computation power. Considering the system requirements, defining appropriate criteria for prioritization of test cases is the main cause of uncertainty. For automotive and robot vehicle domains, QAC has developed a scoring method for prioritization of test suite. In the context of connected and autonomous vehicles, safety and security risks are determined to be two critical dimensions for optimal test case prioritization. Utilizing the functional safety of road vehicles (ISO 26262), a scoring method was developed by performing hazard analysis and risk assessment (HARA), followed by evaluating various risk parameters (Severity, Exposure, and Controllability). For different domains, business risk and severity of potential defects are considered.
Prioritization of automotive tests using ISO 26262, 13849, & IEC 62061
Our test case prioritization process follows industry standards for both road and robotics standards to ensure our customers are focused on critical systems and can communicate to their clients that they are aware and following best testing practices.
Re-prioritization during execution by Machine Learning algorithms
QAC has also developed algorithms that analyze the results of the tests during their execution to learn which components or business rules present less or more defects, and based on that information, reprioritized the remaining test cases to focus on items with many, or no defects.
STAY TUNED
Coming next month
In our next editions, we will present research that the QAaR team has completed that highlights the exposures, and knowledge sharing of Connected and Autonomous vehicle cybersecurity threats and opportunities to use advanced QA techniques.
Our partners:
Recent thought leadership
[qac-carousel id=”20158″]