Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments
Testing with simulation environments helps to identify critical failing scenarios for self-driving cars (SDCs). Simulation-based tests are safer than in-field operational tests and allow detecting software defects before deployment. However, these tests are very expensive and are too many to be run frequently within limited time constraints.
In this paper, we investigate test case prioritization techniques to increase the ability to detect SDC regression faults with virtual tests earlier. Our approach, called SDC-Prioritizer, prioritizes virtual tests for SDCs according to static features of the roads we designed to be used within the driving scenarios. These features can be collected without running the tests, which means that they do not require past execution results. We introduce two evolutionary approaches to prioritize the test cases using diversity metrics (black-box heuristics) computed on these static features. These two approaches, called SO-SDC-Prioritizer and MO-SDC-Prioritizer, use single-objective and multi-objective genetic algorithms, respectively, to find trade-offs between executing the less expensive tests and the most diverse test cases earlier.
Our empirical study conducted in the SDC domain shows that MO-SDC-Prioritizer significantly (p-value <= 0.1e−10) improves the ability to detect safety-critical failures at the same level of execution time compared to baselines: random and greedy-based test case orderings. Besides, our study indicates that multi-objective meta-heuristics outperform single-objective approaches when prioritizing simulation-based tests for SDCs.
MO-SDC-Prioritizer prioritizes test cases with a large improvement in fault detection while its overhead (up to 0.45% of the test execution cost) is negligible.
Fri 19 MayDisplayed time zone: Hobart change
15:45 - 17:15
Cyber-physical systems testingSEIP - Software Engineering in Practice / Technical Track / Journal-First Papers at Meeting Room 106
Chair(s): Shahar Maoz Tel Aviv University
|Data-driven Mutation Analysis for Cyber-Physical Systems|
Enrico Viganò University of Luxembourg, Oscar Cornejo SnT Centre, University of Luxembourg, Fabrizio Pastore University of Luxembourg, Lionel Briand University of Luxembourg; University of OttawaLink to publication Pre-print
|Finding Causally Different Tests for an Industrial Control System|
Chris Poskitt Singapore Management University, Yuqi Chen ShanghaiTech University, China, Jun Sun Singapore Management University, Yu Jiang Tsinghua UniversityPre-print File Attached
|Doppelganger Test Generation for Revealing Bugs in Autonomous Driving Software|
Yuqi Huai University of California, Irvine, Yuntianyi Chen University of California, Irvine, Sumaya Almanee University of California, Irvine, Tuan Ngo VNU University of Engineering and Technology, Xiang Liao University of California, Irvine, Ziwen Wan University of California, Irvine, Qi Alfred Chen University of California, Irvine, Joshua Garcia University of California, IrvinePre-print
|Generating Realistic and Diverse Tests for LiDAR-Based Perception Systems|
Garrett Christian University of Virginia, Trey Woodlief University of Virginia, Sebastian Elbaum University of VirginiaPre-print
|Automated Test Case Generation for Safety-Critical Software in Scade|
SEIP - Software Engineering in Practice
Elson Kurian University of Milano Bicocca, Pietro Braione University of Milano-Bicocca, Daniela Briola University of Milano Bicocca, Dario D'Avino , Matteo Modonato , Giovanni Denaro University of Milano-Bicocca, Italy
|Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments|
Christian Birchler Zurich University of Applied Sciences, Sajad Khatiri USI-Lugnao & Zurich University of Applied Sciences, Pouria Derakhshanfar JetBrains Research, Sebastiano Panichella Zurich University of Applied Sciences, Annibale Panichella Delft University of Technology
|Parameter Coverage for Testing of Autonomous Driving Systems Under Uncertainty|
Thomas Laurent JSPS@National Institute of Informatics, Japan, Stefan Klikovits Johannes Kepler University, Linz, Paolo Arcaini National Institute of Informatics , Fuyuki Ishikawa National Institute of Informatics, Anthony Ventresque Trinity College Dublin & Lero, IrelandLink to publication DOI