A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments
Sat 28 Aug 2021 05:00 - 05:10 - Testing—Test Prioritization and Selection Chair(s): Michalis Famelis
This extended abstract proposes the paper entitled “A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments”, authored by Jackson A. Prado Lima and Silvia R. Vergilio for inclusion in the Journal First track of ESEC/FSE 2021. The paper was accepted for publication into the IEEE Transactions on Software Engineering (TSE) journal on May 4th, 2020, adhering to the period criterion established in the call for papers.
In this paper, we introduce a novel Test Case Prioritization (TCP) approach for the Continuous Integration (CI) context, namely COLEMAN (Combinatorial VOlatiLE Multi-Armed BANdit). The main motivation behind the use of MAB is to overcome some limitations of the existing approaches and to take into account some CI particularities. For instance, some approaches take time to execute, which is not suitable due to the CI environment restrictions. The great majority do not consider the dynamic environment of the software life-cycle in which new test cases can be added or removed (discontinued), fact known as test case volatility. We evaluate COLEMAN by carrying out experiments involving open-source real-world software systems, and different time budgets, reward functions, and MAB policies. We compared our approach against to a state-of-the-art approach widely used in the literature based on Reinforcement Learning (RL). A total of 11,210 experiments was performed, and the results were evaluated with both quality indicators and statistical significance tests, following the most recent best practice in the literature. The results show that our approach outperformed the RL approach in most of cases, regarding different metrics and independently of the time budget investigated.
The original journal paper can be found at https://doi.org/10.1109/TSE.2020.2992428, and our supplementary material containing assets such as datasets, results, technical report, and analysis can be found at https://doi.org/10.17605/OSF.IO/WMCBT.
Fri 27 AugDisplayed time zone: Athens change
17:00 - 18:00
Testing—Test Prioritization and SelectionIndustry Papers / Journal First / Research Papers +12h
Chair(s): Wing Lam University of Illinois at Urbana-Champaign, Myra Cohen Iowa State University
|A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments|
Jackson A. Prado Lima Federal University of Paraná, Silvia Regina Vergilio Federal University of ParanáLink to publication DOI Pre-print Media Attached
|Data-Driven Test Selection at Scale|
Sonu Mehta Microsoft Research, Farima Farmahinifarahani University of California at Irvine, Ranjita Bhagwan Microsoft Research, Suraj Guptha Microsoft, Sina Jafari Microsoft, Rahul Kumar Microsoft Research, Vaibhav Saini Microsoft, Anirudh Santhiar MicrosoftDOI
|Automating System Test Case Classification and Prioritization for Use Case-Driven Testing in Product Lines|
Ines Hajri University of Luxembourg, Arda Goknil SINTEF Digital, Fabrizio Pastore University of Luxembourg, Lionel Briand University of Ottawa, Canada / University of Luxembourg, Luxembourg
|Q&A (Testing—Test Prioritization and Selection)|