Mutation Testing in Evolving Systems: Studying the relevance of mutants to code evolution
Context: When software evolves, opportunities for introducing faults appear. Therefore, it is important to test the evolved program behaviors during each evolution cycle. However, while software evolves, its complexity is also evolving, introducing challenges to the testing process. To deal with this issue, testing techniques should be adapted to target the effect of the program changes instead of the entire program functionality. To this end, commit-aware mutation testing, a powerful testing technique, has been proposed. Unfortunately, commit-aware mutation testing is challenging due to the complex program semantics involved. Hence, it is pertinent to understand the characteristics, predictability, and potential of the technique.
Objective: We conduct an exploratory study to investigate the properties of commit-relevant mutants, i.e., the test elements of commit-aware mutation testing, by proposing a general definition and an experimental approach to identify them. We thus, aim at investigating the prevalence, location, and comparative advantages of commit-aware mutation testing over time (i.e., the program evolution). We also investigate the predictive power of several commit-related features in identifying and selecting commit-relevant mutants to understand the essential properties for its best-effort application case.
Method: Our commit-relevant definition relies on the notion of observational slicing, approximated by higher-order mutation. Specifically, our approach utilizes the impact of mutants, the effects of one mutant on another in capturing and analyzing the implicit interactions between the changed and unchanged code parts. The study analyses millions of mutants (over 10 million), 288 commits, and five (5) different open-source software projects involving over 68,213 CPU days of computation and sets a ground truth where we perform our analysis.
Results: Our analysis shows that commit-relevant mutants are located mainly outside of program commit change (81%), suggesting a limitation in previous work. We also note that the effective selection of commit-relevant mutants has the potential of reducing the number of mutants by up to 93%. In addition, we demonstrate that commit relevant mutation testing is significantly more effective and efficient than state-of-the-art baselines, i.e., random mutant selection and analysis of only mutants within the program change. In our analysis of the predictive power of mutants and commit-related features (e.g., number of mutants within a change, mutant type, and commit size) in predicting commit-relevant mutants, we found that most proxy features do not reliably predict commit-relevant mutants.
Conclusion: This empirical study highlights the properties of commit-relevant mutants and demonstrates the importance of identifying and selecting commit-relevant mutants when testing evolving software systems.
Mon 14 NovDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
16:00 - 17:30
JF Software TestingJournal First at SRC GLR
Chair(s): Aldeida Aleti Monash University
|Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study|
Tao Ma Simula Research Laboratory, Shaukat Ali Simula Research Laboratory, Tao Yue Simula Research Laboratory
|ARTE: Automated Generation of Realistic Test Inputs for Web APIs|
Juan C. Alonso Universidad de Sevilla, Alberto Martin-Lopez University of Seville, Sergio Segura University of Seville, José María García Universidad de Sevilla, Antonio Ruiz-Cortés University of Seville
|Locating Performance Regression Root Causes in the Field Operations of Web-based Systems: An Experience Report|
Lizhi Liao Concordia University, Jinfu Chen Centre for Software Excellence, Huawei, Canada, Heng Li Polytechnique Montréal, Yi Zeng Concordia University, Weiyi Shang Concordia University, Catalin Sporea ERA Environmental Management Solutions, Andrei Toma ERA Environmental Management Solutions, Sarah Sajedi ERA Environmental Management Solutions
|iBiR: Bug Report driven Fault Injection|
Ahmed Khanfir University of Luxembourg, Anil Koyuncu Sabanci University, Mike Papadakis University of Luxembourg, Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Tegawendé F. Bissyandé SnT, University of Luxembourg, Jacques Klein University of Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg
|Mutation Testing in Evolving Systems: Studying the relevance of mutants to code evolution|
Milos Ojdanic University of Luxembourg, Ezekiel Soremekun SnT, University of Luxembourg, Renzo Degiovanni SnT, University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, LuxembourgLink to publication DOI Pre-print